# W&B Tutorials & Blog
Source: https://docs.wandb.ai/blog





# W&B Courses
Source: https://docs.wandb.ai/courses





# Build AI agents, applications, and models
Source: https://docs.wandb.ai/index

Develop AI models and ship LLM applications with the W&B platform for experiment tracking, evaluation, and observability.

## Weights & Biases

<h3>Develop AI models and ship LLM applications with the W\&B platform for experiment tracking, evaluation, and observability.</h3>

<div>
  <ProductCard title="W&B Models" href="/models">
    Manage AI model development with experiment tracking, fine-tuning, reporting, hyperparameter sweeps, and a model registry for versioning and reproducibility.
  </ProductCard>

  <ProductCard title="W&B Weave" href="/weave">
    Manage AI models in your code with tracing, output evaluation, cost estimates, and a playground to compare large language models (LLMs) and settings.
  </ProductCard>
</div>

## Powered by CoreWeave

<h3>Managed services that run on CoreWeave infrastructure.</h3>

<div>
  <ProductCard title="Serverless Inference" href="/inference">
    Access leading open-source foundation models through an OpenAI-compatible API, with usage tracking and Weave integration for tracing and evaluation.
  </ProductCard>

  <ProductCard title="Serverless Training" href="/serverless-training">
    Now in public preview. Post-train and fine-tune LLMs with Serverless RL and Serverless SFT, managed GPU infrastructure, ART and RULER integration, and auto-scaling for multi-turn agentic tasks.
  </ProductCard>

  <ProductCard title="Serverless Sandboxes" href="/sandboxes">
    Now in private preview by invitation. Run code in isolated compute environments with lifecycle management, secrets handling, file access, and a Python SDK.
  </ProductCard>
</div>


# W&B Models
Source: https://docs.wandb.ai/models

Use W&B Models for experiment tracking, dataset versioning, model management, and collaborative ML development.

W\&B Models is the system of record for ML Practitioners who want to organize their models, boost productivity and collaboration, and deliver production ML at scale.

<img alt="W&B Models architecture diagram" />

With W\&B Models, you can:

* Track and visualize all [ML experiments](/models/track/).
* Optimize and fine-tune models at scale with [hyperparameter sweeps](/models/sweeps/).
* [Maintain a centralized hub of all models](/models/registry/), with a seamless handoff point to devops and deployment
* Configure custom automations that trigger key workflows for [model CI/CD](/models/automations/).

Machine learning practitioners rely on W\&B Models as their ML system of record to track and visualize experiments, manage model versions and lineage, and optimize hyperparameters.


# Get Started with W&B Models
Source: https://docs.wandb.ai/models/models_quickstart

Get started with W&B Models by tracking experiments, logging metrics, and visualizing results in a few lines of code.

Learn when and how to use W\&B to track, share, and manage model artifacts in your machine learning workflows. This page covers logging experiments, generating reports, and accessing logged data using the appropriate W\&B API for each task.

This tutorial uses the following:

* [W\&B Python SDK](/models/ref/python) (`wandb.sdk`): to log and monitor experiments during training.
* [W\&B Public API](/models/ref/python/public-api) (`wandb.apis.public`): to query and analyze logged experiment data.
* [W\&B Reports and Workspaces API](/models/ref/wandb_workspaces) (`wandb.wandb-workspaces`): to create a report to summarize findings.

## Sign up and create an API key

To authenticate your machine with W\&B, you must first generate an API key at [wandb.ai/settings](https://wandb.ai/settings). Copy the API key and store it securely.

## Install and import packages

Install the W\&B library and some other packages you will need for this walkthrough.

```python theme={null}
pip install wandb
```

Import W\&B Python SDK:

```python theme={null}
import wandb
```

Specify the entity of your team in the following code block:

```python theme={null}
TEAM_ENTITY = "<Team_Entity>" # Replace with your team entity
PROJECT = "my-awesome-project"
```

## Train a model

The following code simulates a basic machine learning workflow: training a model, logging metrics, and saving the model as an artifact.

Use the W\&B Python SDK (`wandb.sdk`) to interact with W\&B during training. Log the loss using [`wandb.Run.log()`](/models/ref/python/experiments/run/#method-runlog), then save the trained model as an artifact using [`wandb.Artifact`](/models/ref/python/experiments/artifact) before finally adding the model file using [`Artifact.add_file`](/models/ref/python/experiments/artifact#add_file).

```python theme={null}
import random # For simulating data

def model(training_data: int) -> int:
    """Model simulation for demonstration purposes."""
    return training_data * 2 + random.randint(-1, 1)  

# Simulate weights and noise
weights = random.random() # Initialize random weights
noise = random.random() / 5  # Small random noise to simulate noise

# Hyperparameters and configuration
config = {
    "epochs": 10,  # Number of epochs to train
    "learning_rate": 0.01,  # Learning rate for the optimizer
}

# Use context manager to initialize and close W&B runs
with wandb.init(project=PROJECT, entity=TEAM_ENTITY, config=config) as run:    
    # Simulate training loop
    for epoch in range(config["epochs"]):
        xb = weights + noise  # Simulated input training data
        yb = weights + noise * 2  # Simulated target output (double the input noise)
        
        y_pred = model(xb)  # Model prediction
        loss = (yb - y_pred) ** 2  # Mean Squared Error loss

        print(f"epoch={epoch}, loss={loss}")
        # Log epoch and loss to W&B
        run.log({
            "epoch": epoch,
            "loss": loss,
        })

    # Unique name for the model artifact,
    model_artifact_name = f"model-demo"  

    # Local path to save the simulated model file
    PATH = "model.txt" 

    # Save model locally
    with open(PATH, "w") as f:
        f.write(str(weights)) # Saving model weights to a file

    # Create an artifact object
    # Add locally saved model to artifact object
    artifact = wandb.Artifact(name=model_artifact_name, type="model", description="My trained model")
    artifact.add_file(local_path=PATH)
    artifact.save()
```

The key takeaways from the previous code block are:

* Use `wandb.Run.log()` to log metrics during training.
* Use `wandb.Artifact` to save models (datasets, and so forth) as an artifact to your W\&B project.

Now that you have trained a model and saved it as an artifact, you can publish it to a registry in W\&B. Use [`wandb.Run.use_artifact()`](/models/ref/python/experiments/run/#method-runuse_artifact) to retrieve the artifact from your project and prepare it for publication in the Model registry. `wandb.Run.use_artifact()` serves two key purposes:

* Retrieves the artifact object from your project.
* Marks the artifact as an input to the run, ensuring reproducibility and traceability. See [Create and view lineage map](/models/registry/lineage) for details.

## View the training data in the dashboard

Log in to your account at [https://wandb.ai/login](https://wandb.ai/login)

Under **Projects** you should see `my-awesome-project` (or whatever you used as a project name above). Click this to enter the workspace for your project.

From here, you can see details about every run you've done. In this screenshot, the code was re-run several times, generating a number of runs, each of which is given a randomly-generated name.

<Frame>
  <img alt="W&B project page with multiple runs in a table view, including run names, metrics, and status information" />
</Frame>

## Publish the model to the W\&B Registry

To share the model with others in your organization, publish it to a [collection](/models/registry/create_collection) using `wandb.Run.link_artifact()`. The following code links the artifact to a [registry](/models/registry), making it accessible to your team.

```python theme={null}
# Artifact name specifies the specific artifact version within our team's project
artifact_name = f'{TEAM_ENTITY}/{PROJECT}/{model_artifact_name}:v0'
print("Artifact name: ", artifact_name)

REGISTRY_NAME = "Model" # Name of the registry in W&B
COLLECTION_NAME = "DemoModels"  # Name of the collection in the registry

# Create a target path for our artifact in the registry
target_path = f"wandb-registry-{REGISTRY_NAME}/{COLLECTION_NAME}"
print("Target path: ", target_path)

with wandb.init(entity=TEAM_ENTITY, project=PROJECT) as run:
    model_artifact = run.use_artifact(artifact_or_name=artifact_name, type="model")
    run.link_artifact(artifact=model_artifact, target_path=target_path)
```

After running `wandb.Run.link_artifact()`, the model artifact will be in the `DemoModels` collection in your registry. From there, you can view details such as the version history, [lineage map](/models/registry/lineage), and other [metadata](/models/registry/registry_cards).

For additional information on how to link artifacts to a registry, see [Link artifacts to a registry](/models/registry/link_version).

## Retrieve model artifact from registry for inference

To use a model for inference, use `wandb.Run.use_artifact()` to retrieve the published artifact from the registry. This returns an artifact object that you can then use [`wandb.Artifact.download()`](/models/ref/python/experiments/artifact/#method-artifactdownload) to download the artifact to a local file.

```python theme={null}
REGISTRY_NAME = "Model"  # Name of the registry in W&B
COLLECTION_NAME = "DemoModels"  # Name of the collection in the registry
VERSION = 0 # Version of the artifact to retrieve

model_artifact_name = f"wandb-registry-{REGISTRY_NAME}/{COLLECTION_NAME}:v{VERSION}"
print(f"Model artifact name: {model_artifact_name}")

with wandb.init(entity=TEAM_ENTITY, project=PROJECT) as run:
    registry_model = run.use_artifact(artifact_or_name=model_artifact_name)
    local_model_path = registry_model.download()
```

For more information on how to retrieve artifacts from a registry, see [Download an artifact from a registry](/models/registry/download_use_artifact).

Depending on your machine learning framework, you may need to recreate the model architecture before loading the weights. This is left as an exercise for the reader, as it depends on the specific framework and model you are using.

## Share your finds with a report

<Note>
  W\&B Report and Workspace API is in Public Preview.
</Note>

Create and share a [report](/models/reports) to summarize your work. To create a report programmatically, use the [W\&B Report and Workspace API](/models/ref/wandb_workspaces/reports).

First, install the W\&B Reports API:

```python theme={null}
pip install wandb wandb-workspaces -qqq
```

The following code block creates a report with multiple blocks, including markdown, panel grids, and more. You can customize the report by adding more blocks or changing the content of existing blocks.

The output of the code block prints a link to the URL report created. You can open this link in your browser to view the report.

```python theme={null}
import wandb_workspaces.reports.v2 as wr

experiment_summary = """This is a summary of the experiment conducted to train a simple model using W&B."""
dataset_info = """The dataset used for training consists of synthetic data generated by a simple model."""
model_info = """The model is a simple linear regression model that predicts output based on input data with some noise."""

report = wr.Report(
    project=PROJECT,
    entity=TEAM_ENTITY,
    title="My Awesome Model Training Report",
    description=experiment_summary,
    blocks= [
        wr.TableOfContents(),
        wr.H2("Experiment Summary"),
        wr.MarkdownBlock(text=experiment_summary),
        wr.H2("Dataset Information"),
        wr.MarkdownBlock(text=dataset_info),
        wr.H2("Model Information"),
        wr.MarkdownBlock(text = model_info),
        wr.PanelGrid(
            panels=[
                wr.LinePlot(title="Train Loss", x="Step", y=["loss"], title_x="Step", title_y="Loss")
                ],
            ),  
    ]

)

# Save the report to W&B
report.save()
```

For more information on how to create a report programmatically or how to create a report interactively with the W\&B App, see [Create a report](/models/reports/create-a-report) in the W\&B Docs Developer guide.

## Query the registry

Use the [W\&B Public APIs](/models/ref/python/public-api) to query, analyze, and manage historical data from W\&B. This can be useful for tracking the lineage of artifacts, comparing different versions, and analyzing the performance of models over time.

The following code block demonstrates how to query the Model registry for all artifacts in a specific collection. It retrieves the collection and iterates through its versions, printing out the name and version of each artifact.

```python theme={null}
import wandb

# Initialize wandb API
api = wandb.Api()

# Find all artifact versions that contains the string `model` and 
# has either the tag `text-classification` or an `latest` alias
registry_filters = {
    "name": {"$regex": "model"}
}

# Use logical $or operator to filter artifact versions
version_filters = {
    "$or": [
        {"tag": "text-classification"},
        {"alias": "latest"}
    ]
}

# Returns an iterable of all artifact versions that match the filters
artifacts = api.registries(filter=registry_filters).collections().versions(filter=version_filters)

# Print out the name, collection, aliases, tags, and created_at date of each artifact found
for art in artifacts:
    print(f"artifact name: {art.name}")
    print(f"collection artifact belongs to: { art.collection.name}")
    print(f"artifact aliases: {art.aliases}")
    print(f"tags attached to artifact: {art.tags}")
    print(f"artifact created at: {art.created_at}\n")
```

For more information on querying the registry, see the [Query registry items](/models/registry/search_registry/#query-registry-items-with-mongodb-style-queries).


# W&B Quickstart
Source: https://docs.wandb.ai/models/quickstart

Install W&B and start tracking, visualizing, and managing machine learning experiments in minutes.

Install W\&B to track, visualize, and manage machine learning experiments of any size.

<Note>
  Are you looking for information on W\&B Weave? See the [Weave Python SDK quickstart](/weave/quickstart) or [Weave TypeScript SDK quickstart](/weave/reference/generated_typescript_docs/intro-notebook).
</Note>

## Sign up and create an API key

To authenticate your machine with W\&B, you need an API key.

To create an API key, select the **Personal API key** or **Service Account API key** tab for details.

<Tabs>
  <Tab title="Personal API key">
    To create a personal API key owned by your user ID:

    1. Log in to W\&B, click your user profile icon, then click **User Settings**.
    2. Click **Create new API key**.
    3. Provide a descriptive name for your API key.
    4. Click **Create**.
    5. Copy the displayed API key immediately and store it securely.
  </Tab>

  <Tab title="Service account API key">
    To create an API key owned by a service account:

    1. Navigate to the **Service Accounts** tab in your team or organization settings.
    2. Find the service account in the list.
    3. Click the **action (<Icon icon="ellipsis" />)** menu, then click **Create API key**.
    4. Provide a name for the API key, then click **Create**.
    5. Copy the displayed API key immediately and store it securely.
    6. Click **Done**.

    You can create multiple API keys for a single service account to support different environments or workflows.
  </Tab>
</Tabs>

<Warning>
  The full API key is only shown once at creation time. After you close the dialog, you cannot view the full API key again. Only the key ID (first part of the key) is visible in your settings. If you lose the full API key, you must create a new API key.
</Warning>

For secure storage options, see [Store API keys securely](/platform/app/settings-page/user-settings/#store-and-handle-api-keys-securely).

This quickstart is also available as a Colab notebook:

<div>
  <ColabLink />

  <GitHubLink />
</div>

## Install the `wandb` library and log in

<Tabs>
  <Tab title="Command Line">
    1. Set the `WANDB_API_KEY` [environment variable](/models/track/environment-variables/).

       ```bash theme={null}
       export WANDB_API_KEY=<your_api_key>
       ```

    2. Install the `wandb` library and log in.

       ```shell theme={null}
       pip install wandb
       wandb login
       ```
  </Tab>

  <Tab title="Python">
    ```bash theme={null}
    pip install wandb
    ```

    ```python theme={null}
    import wandb

    wandb.login()
    ```
  </Tab>

  <Tab title="Python notebook">
    ```notebook theme={null}
    !pip install wandb
    import wandb
    wandb.login()
    ```
  </Tab>
</Tabs>

## Initialize a run and track hyperparameters

In your Python script or notebook, initialize a W\&B run object with [`wandb.init()`](/models/ref/python/experiments/run/). Use a dictionary for the `config` parameter
to specify hyperparameter names and values. Within the `with` statement, you can log metrics and other information to W\&B.

```python theme={null}
import wandb

wandb.login()

# Project that the run is recorded to
project = "my-awesome-project"

# Dictionary with hyperparameters
config = {
    'epochs' : 10,
    'lr' : 0.01
}

with wandb.init(project=project, config=config) as run:
    # Training code here
    # Log values to W&B with run.log()
    run.log({"accuracy": 0.9, "loss": 0.1})
```

See the next section for a complete example that simulates a training run and logs accuracy and loss metrics to W\&B.

<Info>
  A [run](/models/runs/) is a core element of W\&B. You use runs to [track metrics](/models/track/), [create logs](/models/track/log/), track artifacts, and more.
</Info>

## Create a machine learning training experiment

This mock training script logs simulated accuracy and loss metrics to W\&B. Copy and paste the following code into a Python script or notebook cell and run it:

```python theme={null}
import wandb
import random

wandb.login()

# Project that the run is recorded to
project = "my-awesome-project"

# Dictionary with hyperparameters
config = {
    'epochs' : 10,
    'lr' : 0.01
}

with wandb.init(project=project, config=config) as run:
    offset = random.random() / 5
    print(f"lr: {config['lr']}")
    
    # Simulate a training run
    for epoch in range(2, config['epochs']):
        acc = 1 - 2**-config['epochs'] - random.random() / config['epochs'] - offset
        loss = 2**-config['epochs'] + random.random() / config['epochs'] + offset
        print(f"epoch={config['epochs']}, accuracy={acc}, loss={loss}")
        run.log({"accuracy": acc, "loss": loss})
```

Visit [wandb.ai/home](https://wandb.ai/home) to view recorded metrics such as accuracy and loss and how they changed during each training step. The following image shows the loss and accuracy tracked from each run. Each run object appears in the **Runs** column with generated names.

<Frame>
  <img alt="Shows loss and accuracy tracked from each run." />
</Frame>

## Next steps

Explore more features of the W\&B ecosystem:

1. Read the [W\&B Integration tutorials](/models/integrations) that combine W\&B with frameworks like PyTorch, libraries like Hugging Face, and services like SageMaker.
2. Organize runs, automate visualizations, summarize findings, and share updates with collaborators using [W\&B Reports](/models/reports).
3. Create [W\&B Artifacts](/models/artifacts) to track datasets, models, dependencies, and results throughout your machine learning pipeline.
4. Automate hyperparameter searches and optimize models with [W\&B Sweeps](/models/sweeps).
5. Analyze runs, visualize model predictions, and share insights on a [central dashboard](/models/tables).
6. Visit [W\&B AI Academy](https://wandb.ai/site/courses/) to learn about LLMs, MLOps, and W\&B Models through hands-on courses.
7. Visit [weave-docs.wandb.ai](/weave) to learn how to track, experiment with, evaluate, deploy, and improve your LLM-based applications using Weave.


# Overview
Source: https://docs.wandb.ai/models/runs

Learn about the basic building block of W&B, Runs.

A *run* is a single unit of computation logged by W\&B. You can think of a W\&B Run as an atomic element of your whole project. In other words, each run is a record of a specific computation, such as training a model and logging the results, hyperparameter sweeps, and so forth.

Common use cases for initializing and logging to a run include:

* Training a model and [recording metrics](/models/ref/python/experiments/run#method-run-log) such as accuracy and loss
* Conducting [hyperparameter tuning](/models/sweeps/) and running new experiments
* Conducting a new machine learning experiment with a different model
* Tracking and saving datasets and models as [W\&B Artifacts](/models/artifacts/)
* [Downloading and using](/models/artifacts/download-and-use-an-artifact/) datasets or models used by other members of your team as W\&B Artifacts

To initialize a W\&B run, call the [`wandb.init()`](/models/ref/python/functions/init) method from the W\&B Python SDK. This starts a new run and returns a `wandb.Run` object that you can use to log metrics, artifacts, and other information to the run. For more information about initializing a run, see [Initialize runs](/models/runs/initialize-run).

Each run object has a [unique identifier known as a *run ID*](/models/runs/run-identifiers#unique-run-identifiers). [You can specify a unique ID](/models/runs/run-identifiers#unique-run-identifiers) or let [W\&B randomly generate one for you](/models/runs/run-identifiers#autogenerated-run-ids). Each run object also has a human-readable, non-unique [run name](/models/runs/run-identifiers#run-name). You can specify a name for your run or let W\&B randomly generate one for you. You can rename a run after initializing it.

W\&B logs your run to a [*project*](/models/track/project-page/). You specify the project when you initialize the run with `wandb.init(project="")`. W\&B creates a new project if the project does not exist. If the project does exist, W\&B logs the run to the project you specified.

<Note>
  If you do not specify a project name, W\&B stores the run in a project called `Uncategorized`.
</Note>

`wandb.init()` returns a `wandb.Run` object that contains properties of the run, such as its ID, name, configuration, and state. Use the run object to log metrics, artifacts, and other information to the run with methods such as `wandb.Run.log()`, `wandb.Run.log_code()`, and `wandb.Run.use_artifact()`.

Each run has a state that describes the current status of the run. See [Run states](/models/runs/run-states) for a full list of possible run states.

[View runs and their properties](/models/runs/view-logged-runs) within the run's project workspace on the W\&B App. You can also programmatically access run properties with the [`wandb.Api.Run`](/models/ref/python/experiments/run) object.

As an example, consider the following code snippet that initializes a W\&B run and logs some metrics to it:

<Note>
  Pass your W\&B entity to the `entity` variable in the code snippets below if you want to follow along. Your entity is your W\&B username or team name. You can find it in the URL of your W\&B App workspace. For example, if your workspace URL is `https://wandb.ai/nico/awesome-project`, then your entity is `nico`.
</Note>

```python theme={null}
import wandb

entity = "nico"  # Replace with your W&B entity
project = "awesome-project"

with wandb.init(entity=entity, project=project) as run:
    run.log({"accuracy": 0.9, "loss": 0.1})
```

The first line imports the W\&B Python SDK. The second line initializes a run in the project `awesome-project` under the entity `nico`. The third line logs the accuracy and loss of the model to that run.

Within the terminal, W\&B returns:

```bash theme={null}
wandb: Syncing run earnest-sunset-1
wandb: ⭐️ View project at https://wandb.ai/nico/awesome-project
wandb: 🚀 View run at https://wandb.ai/nico/awesome-project/runs/1jx1ud12
wandb:                                                                                
wandb: 
wandb: Run history:
wandb: accuracy ▁
wandb:     loss ▁
wandb: 
wandb: Run summary:
wandb: accuracy 0.9
wandb:     loss 0.5
wandb: 
wandb: 🚀 View run earnest-sunset-1 at: https://wandb.ai/nico/awesome-project/runs/1jx1ud12
wandb: ⭐️ View project at: https://wandb.ai/nico/awesome-project
wandb: Synced 6 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)
wandb: Find logs at: ./wandb/run-20241105_111006-1jx1ud12/logs
```

W\&B returns two URLs in the terminal output. The first URL directs you to the [specific run's workspace](/models/runs/view-logged-runs), and the second URL directs you to the [project](/models/track/project-page) page.

<Frame>
  <img alt="Single run workspace" />
</Frame>

Logging a metric at a single point of time might not be that useful. A more realistic example in the case of training discriminative models is to log metrics at regular intervals. For example, consider the following code snippet:

```python theme={null}
import wandb
import random

config = {
    "epochs": 10,
    "learning_rate": 0.01,
}

with wandb.init(project="awesome-project", config=config) as run:
    print(f"lr: {config['learning_rate']}")
      
    # Simulating a training run
    for epoch in range(config['epochs']):
      offset = random.random() / 5
      acc = 1 - 2**-epoch - random.random() / (epoch + 1) - offset
      loss = 2**-epoch + random.random() / (epoch + 1) + offset
      print(f"epoch={epoch}, accuracy={acc}, loss={loss}")
      run.log({"accuracy": acc, "loss": loss})
```

The training script calls `wandb.Run.log()` 10 times. Each time the script calls `wandb.Run.log()`, W\&B logs the accuracy and loss for that epoch.

Within your terminal, you should see output similar to the following:

```bash theme={null}
wandb: Syncing run jolly-haze-4
wandb: ⭐️ View project at https://wandb.ai/nico/awesome-project
wandb: 🚀 View run at https://wandb.ai/nico/awesome-project/runs/pdo5110r
lr: 0.01
epoch=0, accuracy=-0.10070974957523078, loss=1.985328507123956
epoch=1, accuracy=0.2884687745057535, loss=0.7374362314407752
epoch=2, accuracy=0.7347387967382066, loss=0.4402409835486663
epoch=3, accuracy=0.7667969248039795, loss=0.26176963846423457
epoch=4, accuracy=0.7446848791003173, loss=0.24808611724405083
epoch=5, accuracy=0.8035095836268268, loss=0.16169791827329466
epoch=6, accuracy=0.861349032371624, loss=0.03432578493587426
epoch=7, accuracy=0.8794926436276016, loss=0.10331872172219471
epoch=8, accuracy=0.9424839917077272, loss=0.07767793473500445
epoch=9, accuracy=0.9584880427028566, loss=0.10531971149250456
wandb: 🚀 View run jolly-haze-4 at: https://wandb.ai/nico/awesome-project/runs/pdo5110r
wandb: Find logs at: wandb/run-20241105_111816-pdo5110r/logs
```

W\&B captures the simulated training loop within a single run called `jolly-haze-4`. This is because the script calls `wandb.init()` method only once.

Copy and paste the URL that W\&B prints from the previous output into your browser. The URL directs you to the run's workspace in the W\&B App UI. For example, the following image shows the workspace for the run `jolly-haze-4`:

<Frame>
  <img alt="Training run with logged metrics" />
</Frame>


# Send an alert
Source: https://docs.wandb.ai/models/runs/alert

Send alerts, triggered from your Python code, to your Slack or email

<ColabLink />

Create alerts with Slack or email if your run crashes or with a custom trigger. For example, you can create an alert if the gradient of your training loop starts to blow up (reports NaN) or a step in your ML pipeline completes. Alerts apply to all projects where you initialize runs, including both personal and team projects.

And then see W\&B Alerts messages in Slack (or your email):

<Frame>
  <img alt="Slack alert setup" />
</Frame>

<Note>
  W\&B Alerts require you to add `run.alert()` to your code. Without modifying your code, [Automations](/models/automations/) provide another way to notify Slack based on an event in W\&B, such as when an [artifact](/models/artifacts/) artifact version is created or when a [run metric](/models/runs/) meets or changes by a threshold.

  For example, an automation can notify a Slack channel when a new version is created, run an automated testing webhook when the `production` alias is added to an artifact, or start a validation job only when a run's `loss` is within acceptable bounds.

  Read the [Automations overview](/models/automations/) or [create an automation](/models/automations/create-automations/).
</Note>

<Note>
  The following guide only applies to alerts in Multi-tenant Cloud.

  If you're using [W\&B Server](/platform/hosting/) in your Private Cloud or on W\&B Dedicated Cloud, refer to [Configure Slack alerts in W\&B Server](/platform/hosting/monitoring-usage/slack-alerts) to set up Slack alerts.
</Note>

To set up an alert, take these steps, which are detailed in the following sections:

1. Turn on Alerts in your W\&B [User Settings](https://wandb.ai/settings).
2. Add `run.alert()` to your code.
3. Test the configuration.

### 1. Turn on alerts in your W\&B User Settings

In your [User Settings](https://wandb.ai/settings):

* Scroll to the **Alerts** section
* Turn on **Scriptable run alerts** to receive alerts from `run.alert()`
* Use **Connect Slack** to pick a Slack channel to post alerts. We recommend the **Slackbot** channel because it keeps the alerts private.
* **Email** will go to the email address you used when you signed up for W\&B. We recommend setting up a filter in your email so all these alerts go into a folder and don't fill up your inbox.

You will only have to do this the first time you set up W\&B Alerts, or when you'd like to modify how you receive alerts.

<Frame>
  <img alt="Alerts settings in W&B User Settings" />
</Frame>

### 2. Add `run.alert()` to your code

Add `run.alert()` to your code (either in a Notebook or Python script) wherever you'd like it to be triggered

```python theme={null}
import wandb

with wandb.init() as run:
    run.alert(title="High Loss", text="Loss is increasing rapidly")
```

### 3. Test the configuration

Check your Slack or emails for the alert message. If you didn't receive any, make sure you've got emails or Slack turned on for **Scriptable Alerts** in your [User Settings](https://wandb.ai/settings)

## Example

This simple alert sends a warning when accuracy falls below a threshold. In this example, it only sends alerts at least 5 minutes apart.

```python theme={null}
import wandb
from wandb import AlertLevel

with wandb.init() as run:

    if acc < threshold:
        run.alert(
            title="Low accuracy",
            text=f"Accuracy {acc} is below the acceptable threshold {threshold}",
            level=AlertLevel.WARN,
            wait_duration=300,
        )
```

## Tag or mention users

Use the at sign `@` followed by the Slack user ID to tag yourself or your colleagues in either the title or the text of the alert. You can find a Slack user ID from their Slack profile page.

```python theme={null}
run.alert(title="Loss is NaN", text=f"Hey <@U1234ABCD> loss has gone to NaN")
```

## Configure team alerts

Team admins can set up alerts for the team on the team settings page: `wandb.ai/teams/your-team`.

Team alerts apply to everyone on your team. W\&B recommends using the **Slackbot** channel because it keeps alerts private.

## Change Slack channel to send alerts to

To change what channel alerts are sent to, click **Disconnect Slack** and then reconnect. After you reconnect, pick a different Slack channel.


# Semantic run plot legends
Source: https://docs.wandb.ai/models/runs/color-code-runs

Color-code W&B runs based on metrics or config parameters to create visually meaningful chart legends.

Create visually meaningful line plots and plot legends by color-coding your W\&B runs based on metrics or configuration parameters. Identify patterns and trends across experiments by coloring runs according to their performance metrics (highest, lowest, or latest values). W\&B automatically groups your runs into color-coded buckets based on the values of your selected parameter.

To use metric or configuration-based colors for your runs, you need to configure two settings:

### Turn on key-based colors

1. Navigate to your W\&B project.
2. Select the **Workspace** tab from the project sidebar.
3. Click on the **Settings** icon in the top right corner.
4. From the drawer, select **Runs**.
5. In the **Run colors** section, select **Key-based colors**.
6. Configure the following options:
   * From the **Key** dropdown, select the metric you want to use for assigning colors to runs.
   * From the **Y value** dropdown, select the y value you want to use for assigning colors to runs.
   * Set the number of buckets to a value from 2 to 8.

<Note>
  When you use key-based colors, the option to [customize run colors](/models/runs/run-colors) is not available.
</Note>

The following sections describe how to set the metric and y value and as how to customize the buckets used for assigning colors to runs.

### Example: Key-based coloring with loss metric

In this example plot, runs are colored with a gradient where darker colors represent higher loss values and lighter colors represent lower loss values. The Y value is set to `latest` to use the most recent loss value for each run.

<Image alt="W&B workspace showing runs colored based on their loss values using key-based coloring." />

## Set a metric

The metric options in your **Key** dropdown are derived from the key-value pairs [you log to W\&B](/models/runs/color-code-runs/#custom-metrics) and [default metrics](/models/runs/color-code-runs/#default-metrics) defined by W\&B.

### Default metrics

* `Relative Time (Process)`: The relative time of the run, measured in seconds since the start of the run.
* `Relative Time (Wall)`: The relative time of the run, measured in seconds since the start of the run, adjusted for wall clock time.
* `Wall Time`: The wall clock time of the run, measured in seconds since the epoch.
* `Step`: The step number of the run, which is typically used to track the progress of training or evaluation.

### Custom metrics

Color runs and create meaningful plot legends based on custom metrics logged by your training or evaluation scripts. Custom metrics are logged as key-value pairs, where the key is the name of the metric and the value is the metric value.

For example, the following code snippet logs accuracy (`"acc"` key) and loss (`"loss"` key) during a training loop:

```python theme={null}
import wandb
import random

epochs = 10

with wandb.init(project="basic-intro") as run:
  # Block simulates a training loop logging metrics
  offset = random.random() / 5
  for epoch in range(2, epochs):
      acc = 1 - 2 ** -epoch - random.random() / epoch - offset
      loss = 2 ** -epoch + random.random() / epoch + offset

      # Log metrics from your script to W&B
      run.log({"acc": acc, "loss": loss})
```

Within the **Key** dropdown, both `"acc"` and `"loss"` are available options.

## Set a configuration key

The configuration options in your **Key** dropdown are derived from the key-value pairs you pass to the `config` parameter when you initialize a W\&B run. Configuration keys are typically used to log hyperparameters or other settings used in your training or evaluation scripts.

```python theme={null}
import wandb

config = {
  "learning_rate": 0.01,
  "batch_size": 32,
  "optimizer": "adam"
}

with wandb.init(project="basic-intro", config=config) as run:
  # Your training code here
  pass
```

Within the **Key** dropdown, `"learning_rate"`, `"batch_size"`, and `"optimizer"` are available options.

## Set a y value

You can choose from the following options:

* **Latest**: Determine color based on Y value at last logged step for each line.
* **Max**: Color based on highest Y value logged against the metric.
* **Min**: Color based on lowest Y value logged against the metric.

## Customize buckets

Buckets are ranges of values that W\&B uses to categorize runs based on the metric or configuration key you select. Buckets are evenly distributed across the range of values for the specified metric or configuration key and each bucket is assigned a unique color. Runs that fall within that bucket's range are displayed in that color.

Consider the following:

<Frame>
  <img alt="Color coded runs" />
</Frame>

* **Key** is set to `"Accuracy"` (abbreviated as `"acc"`).
* **Y value** is set to `"Max"`

With this configuration, W\&B colors each run based on their accuracy values. The colors vary from a light yellow color to a deep color. Lighter colors represent lower accuracy values, while deeper colors represent higher accuracy values.

Six buckets are defined for the metric, with each bucket representing a range of accuracy values. Within the **Buckets** section, the following range of buckets are defined:

* Bucket 1: (Min - 0.7629)
* Bucket 2: (0.7629 - 0.7824)
* Bucket 3: (0.7824 - 0.8019)
* Bucket 4: (0.8019 - 0.8214)
* Bucket 5: (0.8214 - 0.8409)
* Bucket 6: (0.8409 - Max)

In the line plot below, the run with the highest accuracy (0.8232) is colored in a deep purple (Bucket 5), while the run with the lowest accuracy (0.7684) is colored in a light orange (Bucket 2). The other runs are colored based on their accuracy values, with the color gradient indicating their relative performance.

<Frame>
  <img alt="Color coded runs plot" />
</Frame>


# Pin and compare runs
Source: https://docs.wandb.ai/models/runs/compare-runs

Learn how to use pinned and baseline runs to keep track of important runs and efficiently evaluate model experiments.

Use the W\&B App to organize, identify, and compare important runs, such as top performers, production models, failed experiments, and reference runs. To accomplish this, you can organize and compare runs with:

* **Pinned runs**: Pin runs from any project in your workspace to keep them visible at the top of the runs list. You can pin up to 20 runs in a workspace, including runs from other projects.
* **Baseline run**: Specify a baseline run as your reference point for comparisons. The baseline run is always visible in the workspace and at the top of the runs list. In the runs table, summary metric deltas show how each run compares to the baseline. In line plots, the baseline appears with visually distinct styling to help with comparison.

<img alt="Line plot with baseline and pinned runs" />

These features are particularly useful for:

* Comparing new experiments against your production model.
* [Comparing runs across projects](/models/runs/compare-runs#compare-runs-across-projects).
* [Creating a baseline for your experiments to evaluate how new runs perform against it.](/models/runs/compare-runs#manage-and-compare-a-baseline-run)
* Tracking multiple candidate models during experimentation.
* Evaluating whether new runs improve on your best results.

See [Limitations](#limitations).

## Pin runs

Pin runs to keep them easily accessible at the top of your workspace. To hide a pinned or baseline run, click the <i /> icon. To show a hidden run, click the <i /> icon. Pinned runs appear at the top of the run selector with a circular pin icon, separated from other runs by a visual divider.

To pin a run:

1. Navigate to your workspace.
2. In the run selector or runs table, find the run you want to pin.
3. Click the **action (<Icon icon="ellipsis" />)** menu, then select **Pin run**.

You can pin up to 20 runs in a workspace. If you have a baseline run, you can pin up to 19 runs because the baseline is implicitly pinned.

<Frame>
  <img alt="Runs table with pinned runs" />
</Frame>

To unpin a run, click the pin icon, or follow the procedure to pin the run, but select **Unpin run** instead.

Runs that you pin to a project only impact your [personal or saved workspace view](/models/track/workspaces#saved-workspace-views), including [runs pinned from another project](/models/runs/compare-runs#compare-runs-across-projects).

## Compare runs across projects

Compare runs from different projects by selecting runs from another project and [pinning](/models/runs/compare-runs#pin-runs) them to your current workspace.

1. Navigate to your project's workspace.
2. Click on the **Select runs from another project** button (small square box with rounded corners and a diagonal arrow) at the top of your workspace's run selector or the runs table.

<Frame>
  <img alt="Runs table with pinned runs from another project" />
</Frame>

3. In the modal, select a project from the **Source project** dropdown.
4. Search the source project’s runs to find the run you want to compare. You can search by run name or unique ID.
5. Select the checkbox next to each run you want to compare, then click **Pin runs**.

View runs that you pin from another project in the run selector or runs table. Pinned runs from other projects have an open circle icon next to their name. To see which project a pinned run comes from, hover over the run name.

Artifacts, logs, and other details for pinned runs link back to the original run. W\&B does not import this data into the current project or workspace. To view the details for a run pinned from another project, click the run. W\&B opens the original run details in a new browser tab.

To compare runs from different projects, pin the runs you want to compare, then use [line plots](/models/app/features/panels/line-plot) to visually compare them.

## Manage and compare a baseline run

You can designate one run as the baseline for the workspace to use it as a reference point for evaluating other runs in your workspace.

In the runs selector and runs table, the baseline run appears at the top alongside pinned runs, and has a bookmark icon instead of a pin.

In line plots, lines for the baseline run appear bolder than other lines. When hovering over the plot or legend, the baseline run's line is dashed.

<Frame>
  <img alt="Demo of comparing another run with the baseline" />
</Frame>

### Set a baseline run

To set a baseline run:

1. Navigate to your workspace.
2. In the run selector or runs table, find the run you want to use as your baseline.
3. Click the **action (<Icon icon="ellipsis" />)** menu, then select **Set as baseline**.

The baseline run appears at the top of the run selector, separated from other runs by a visual divider. The baseline run has a bookmark icon instead of a circle.

<Frame>
  <img alt="Runs table with a baseline run and pinned runs" />
</Frame>

### Change the baseline run

Only one run can be the baseline at a time. To change which run is your baseline:

1. Navigate to your workspace.
2. In the run selector or runs table, find the run you want to use as your new baseline.
3. Click the **action (<Icon icon="ellipsis" />)** menu, then select **Replace baseline**. <Note>If the menu item is inactive, ensure that you have at least one pinning slot available. If necessary, unpin a pinned run by clicking the circular pin icon next to a pinned run.</Note>
4. The new run becomes the baseline, and the previous baseline is automatically pinned so you can find it easily. Optionally, unpin it by clicking its pin icon.

### Remove the baseline designation

To remove the baseline designation:

1. Navigate to your workspace.
2. In the run selector or runs table, find the current baseline run.
3. Click the **action (<Icon icon="ellipsis" />)** menu, then select **Remove baseline**. <Note>If the menu item is inactive, ensure that you have at least one pinning slot available. If necessary, unpin a pinned run by clicking the circular pin icon next to a pinned run.</Note>
4. The previous baseline is automatically pinned so you can find it easily. Optionally, unpin it by clicking its pin icon.

### Compare runs to the baseline

The baseline run is always visible in line plots for metrics the run has logged. In line plots, lines for the baseline run appear bolder than other lines.

* Hover over a part of the plot to display a tooltip with values for all visible runs, including the baseline run and pinned runs.
  <Frame>
    <img alt="Demo showing details for all visible runs at a given point" />
  </Frame>
* Hover over the baseline run's legend label to display the line prominently. It appears as a heavy dashed line. Lines for other visible runs appear with reduced saturation.
  <Frame>
    <img alt="Demo showing details for the baseline run" />
  </Frame>
* Hover over another run's legend label to display that run's line prominently and compare it with the baseline, which appears as a heavy dashed line. Lines for other visible runs appear with reduced saturation.
  <Frame>
    <img alt="Demo of comparing another run with the baseline" />
  </Frame>

### Summary metric deltas

When a run is set as the baseline, by default every other run that logs the same summary metric as the baseline run shows the delta (amount of change) of that metric from the baseline. The delta appears to the right of the metric's value in the run's row in the runs table.

By default, the delta is shown with dark gray text on a dark gray background. To turn on semantic coloring for quick visual reference, you can set the **Metric directionality** for a column. With directionality set:

* If the other run **outperforms** (is directionally better than) the baseline, the delta is shown in dark red text with a light red background.
* If the other run **underperforms** (is directionally better than) the baseline, the delta is shown in dark teal text with a light teal background.

To set the directionality for a metric:

1. In the runs table, hover over the column heading for the metric.
2. Click the **action (<Icon icon="ellipsis" />)** menu that appears.
3. Set **Metric directionality** to **Higher values are best** or **Lower values are best**.

The following screenshot shows how the runs `nanochat-train-base` and `nanochat-train-mid` compare with the baseline run `nanochat-train`. Delta metrics are shown for `TOTAL_TRAINING_TIME`, `TRAIN/DT`, AND `TRAIN/GRAD_NORM`.

<img alt="Screenshot comparing summary metric deltas from the baseline run" />

## Hide summary metric deltas in a workspace

By default, a workspace with a baseline run always displays summary metric deltas. To hide them for a workspace:

1. In the workspace, click **Settings**.
2. In the drawer that appears, click **Runs**.
3. In the **Baseline** tab, toggle **Show value deltas in the runs table**.
4. Close the workspace settings drawer.

## Use cases

This section describes some scenarios where pinned and baseline runs can help guide your experiments.

* **Track production models**: Ensure that new models meet your quality bar before deployment.
  1. Set your production model as the baseline.
  2. Compare all experiments against your deployed model to identify candidates that outperform production.
* **Compare hyperparameter experiments**: Evaluate hyperparameter sweeps or manual experiments against your best-known configuration.
  1. Set your best known configuration as the baseline.
  2. Pin promising candidates as you discover them.
  3. Use the line plots to visually compare runs against the baseline.
  4. Continue to update the baseline as you find better configurations.

## Example workflow

This section illustrates how pinned and baseline runs can help you to compare runs.

1. Run this example code, which simulates a hyperparameter-tuning scenario with a series of runs. Replace placeholders surrounded with angle brackets (`<>`) with your own values.

   ```python theme={null}
   import wandb
   import random
   import math

   def train_model(learning_rate, batch_size, run_name, tags=None):
       """Simulate training a model with given hyperparameters."""
       config = {
           "learning_rate": learning_rate,
           "batch_size": batch_size,
           "optimizer": "adam",
           "architecture": "resnet50"
       }
       
       with wandb.init(
         # Replace with your team and project name
           project="hyperparameter-tuning",
           entity="<team>",
           name=run_name,
           config=config,
           tags=tags or []
       ) as run:
           # Simulate training loop
           for epoch in range(50):
               # Simulated metrics
               accuracy = 0.6 + 0.3 * (1 - math.exp(-learning_rate * epoch / 10))
               loss = 1.0 * math.exp(-learning_rate * epoch / 10)
               
               run.log({
                   "epoch": epoch,
                   "accuracy": accuracy,
                   "loss": loss
               })

   # Create baseline run with standard configuration
   train_model(
       learning_rate=0.001,
       batch_size=64,
       run_name="baseline-config",
       tags=["baseline", "production"]
   )

   # Experiment with different learning rates
   train_model(
       learning_rate=0.003,
       batch_size=64,
       run_name="lr-experiment-0.003",
       tags=["experiment"]
   )

   train_model(
       learning_rate=0.0001,
       batch_size=64,
       run_name="lr-experiment-0.0001",
       tags=["experiment"]
   )
   ```

   After running this code, your workspace has three runs.
2. Set `baseline-config` as your baseline run.
3. Pin `baseline-config` to keep it visible.
4. Compare the experiment runs against the baseline.

* In the runs table, review the summary metric deltas next to each run's values to compare the run to the baseline.
* In line plots, compare the performance of one or more runs to the baseline, which is always visible.

5. Pin promising experiments for further investigation. In this example, after 50 epochs, `lr-experiment-0.003` has the highest accuracy (`~0.64`) and the lowest loss (`~0.86`).

## Limitations

The following features are not yet supported for pinned and baseline runs:

* **Grouping**: When [viewing runs](/models/runs#view-logged-runs) in the run selector or runs table, if runs are grouped by a column, pinned and baseline runs are not visually distinct from other runs.
* **Reports**: In a run set in a [W\&B Report](/models/reports), pinned and baseline runs are not visually distinct from other runs.
* **Workspace view only**: The baseline does not appear when viewing a single run's workspace.
* **Line plots only**: Baseline comparison is available only for line plots, and is not yet available for other panels such as bar plots or media panels.


# View runs in a project
Source: https://docs.wandb.ai/models/runs/customize-run-display

Details about customizing how runs are displayed in your project's runs table

View all runs logged to your W\&B project in the **Runs** tab of your project sidebar. Within the Runs tab is the *Runs table*. The Runs table shows details about all of your runs in a project. Use the Runs table to compare runs, sort runs by specific columns, and organize runs into groups.

The following image shows the Runs table for a project named `deep-drive`:

<Frame>
  <img alt="Runs table" />
</Frame>

## Manage columns

The following sections describe how to customize the Runs table.

### Add columns

Add columns in the Runs table to customize which properties associated with your project are visible.

To add a columns in the Runs table:

1. In the project sidebar, select the **Runs** tab.
2. Above the list of runs, click the **Columns** (six horizontal dashes) button.
3. Select the name of a property within the **Hidden** section of the modal.
4. Drag columns to change their order.
5. Click **Close**.

### Remove columns

Remove columns in the Runs table to customize which properties associated with your project are visible.

To remove columns in the Runs table:

1. In the project sidebar, select the **Runs** tab.
2. Above the list of runs, click the **Columns** (six horizontal dashes) button.
3. Select the name of a property within the **Visible & Pinned** section of the modal.
4. Click **Close**.

### Move columns

To move columns in the Runs table:

1. In the project sidebar, select the **Runs** tab.
2. Drag a column to the left or right.

### Pin columns

Pinned columns are shown on the left-hand side. Unpinned columns are shown on the right-hand side of the Runs tab. If you pin a column in the **Runs** tab, it is also pinned in the **Workspace** tab. Similarly, if you pin a column in the **Workspace** tab, it is also pinned in the **Runs** tab.

To pin a column:

1. In the project sidebar, navigate to the **Runs** tab.
2. Click the **Columns** (six horizontal dashes) button.
3. Within the **Visible & Pinned** section of the modal, click on the pin icon next to the column name you want to pin.

To unpin a column:

1. In the project sidebar, navigate to the **Runs** tab.
2. Either hover over the column name, then click its **action (<Icon icon="ellipsis-vertical" />)** menu, or click the **Columns** (six horizontal dashes) button and click on the pin icon next to the column name you want to unpin.
3. Click **Unpin column**.

<Note>
  W\&B persists columns you pin or unpin in the Runs table in the Runs selector of the Workspace tab.
</Note>

### Hide columns

To hide columns in the Runs table:

1. In the project sidebar, select the **Runs** tab.
2. Hover over the column name, click the **action (<Icon icon="ellipsis-vertical" />)** menu that appears.
3. Click **Hide column**.

To view all columns that are currently hidden, click **Columns**.

## Sort runs by column

Sort runs by any visible column in the Runs table. This is particularly useful if you want to view the best (or worst) recorded value.

To sort the list of runs by any visible column:

1. Hover over the column name, then click its **action (<Icon icon="ellipsis-vertical" />)** menu.
2. Optinally hover over **Show latest**. From the dropdown, select **latest**, **min**, or **max**.
3. Click **Sort ascending** or **Sort descending**.

The following animation shows sorting runs by the maximum value of a logged metric:

<Frame>
  <img alt="Sort by min/max values" />
</Frame>

<Note>
  W\&B persists the sort order you select in the Runs table in the Runs selector of the Workspace tab.
</Note>

## Export runs table to CSV

Export the table of all your runs, hyperparameters, and summary metrics to a CSV with the download button.

1. In the project sidebar, select the **Runs** tab.
2. Above the list of runs, click the **Download** (downward arrow) button.


# Delete runs
Source: https://docs.wandb.ai/models/runs/delete-runs

Delete runs from a W&B project using the W&B App or the Public API, and learn how deleted run data is removed from storage.

## Delete runs

Delete runs from a project with the W\&B App or the Python API.

<Tabs>
  <Tab title="W&B App">
    1. Navigate to the project that contains the runs you want to delete.
    2. Select the **Runs** tab.
    3. Select the checkbox next to the runs you want to delete.
    4. Choose the **Delete** button (trash can icon) above the table.
    5. From the drawer that appears, choose **Delete**.

    For projects that contain a large number of runs, you can use either the search bar to filter runs you want to delete using Regex or the filter button to filter runs based on their status, tags, or other properties.
  </Tab>

  <Tab title="Python">
    You can delete runs programmatically with [`Run.delete()`](/models/ref/python/public-api/run#method-run-delete). Set `delete_artifacts=True` if you also want to remove artifacts associated with the run.

    ```python theme={null}
    import wandb

    api = wandb.Api()
    runs = api.runs("<entity>/<project>")
    for run in runs:
        if run.state == "finished":  # Replace with your own condition
            run.delete(delete_artifacts=False)
    ```

    For the full method signature and behavior, see the [`Run.delete` reference](/models/ref/python/public-api/run#method-run-delete).

    To remove individual files attached to a run, like logged media:

    1. Obtain the relevant file handles with [`Run.files()`](/models/ref/python/public-api/run#method-run-files).
    2. Use [`File.delete()`](/models/ref/python/public-api/file#method-file-delete) to delete individual files.
  </Tab>
</Tabs>

A run ID cannot be reused, even after the run is deleted. Instead, the run will fail with an error.

<Warning>
  When you delete a run and choose to delete associated artifacts, the artifacts are permanently removed and can't be recovered, even if the run is restored later. This includes artifacts linked to the Registry.
</Warning>

## Run deletion flowchart

The following diagram illustrates the complete run deletion process, including the handling of associated artifacts and Registry links:

```mermaid theme={null}
graph TB
    Start([User Initiates<br/>Run Deletion]) --> RunSelect[Select Runs<br/>to Delete]
    RunSelect --> DeletePrompt{Delete Associated<br/>Artifacts?}
    DeletePrompt -->|No| DeleteRunOnly[Delete Run Only<br/><br/>- Run metadata removed<br/>- Artifacts remain available<br/>- Can still access artifacts]
    DeletePrompt -->|Yes| CheckArtifacts[Check for<br/>Associated Artifacts]
    CheckArtifacts --> HasRegistry{Artifacts Linked to<br/>Model Registry?}
    HasRegistry -->|Yes| RegistryWarning[⚠️ Warning<br/><br/>Registry models will be deleted<br/>Production aliases affected]
    HasRegistry -->|No| DirectDelete
    RegistryWarning --> ConfirmRegistry{Confirm Registry<br/>Model Deletion?}
    ConfirmRegistry -->|No| DeleteRunOnly
    ConfirmRegistry -->|Yes| DirectDelete[Delete Run + Artifacts<br/><br/>- Run metadata removed<br/>- Artifacts permanently deleted<br/>- Registry links removed<br/>- Cannot be recovered]
    DeleteRunOnly --> PartialEnd([Run Deleted<br/>Artifacts Preserved])
    DirectDelete --> FullEnd([Run + Artifacts<br/>Permanently Deleted])
    style Start fill:#e1f5fe,stroke:#333,stroke-width:2px,color:#000
    style DeletePrompt fill:#fff3e0,stroke:#333,stroke-width:2px,color:#000
    style RegistryWarning fill:#ffecb3,stroke:#333,stroke-width:2px,color:#000
    style DirectDelete fill:#ffebee,stroke:#333,stroke-width:2px,color:#000
    style DeleteRunOnly fill:#e8f5e9,stroke:#333,stroke-width:2px,color:#000
    style PartialEnd fill:#c8e6c9,stroke:#333,stroke-width:2px,color:#000
    style FullEnd fill:#ffcdd2,stroke:#333,stroke-width:2px,color:#000
```

## When deleted run data is removed from storage

On [W\&B Dedicated Cloud](/platform/hosting/hosting-options/dedicated-cloud) and [W\&B Self-Managed](/platform/hosting/hosting-options/self-managed), the `GORILLA_DATA_RETENTION_PERIOD` environment variable controls how long **deleted run data** is retained before it can be permanently removed from object storage. **Artifacts are not removed by this setting**; they follow the artifact deletion and garbage collection flow described in [Delete an artifact](/models/artifacts/delete-artifacts).

Setting or changing `GORILLA_DATA_RETENTION_PERIOD` is irreversible for data past the retention window. Back up your database and bucket before enabling or tightening retention. See [Configure environment variables](/platform/hosting/env-vars) for the reference table and warnings.

Even after run or file deletion and retention processing, **bucket usage can lag** while background jobs catch up. W\&B does not guarantee immediate reclamation of object storage. For a full overview of artifacts versus run data, timing expectations, and optional operator actions, see [Manage bucket storage and costs](/platform/hosting/managing-bucket-storage).

<Note>
  If deletions do not appear as expected in the W\&B App when using the Public API, upgrade the W\&B Python SDK to a current release and retry.
</Note>


# Filter runs
Source: https://docs.wandb.ai/models/runs/filter-runs

Learn how to filter runs in the Runs table using the expression editor.

Filter runs based on their name, state, [tags](#filter-runs-with-tags), or other properties with the expression editor in the Runs table.

When you add a filter, you first choose a field (for example, tags, timestamp, or entity). Each field has an underlying type, such as text, time, or ID.

The list of operators you see (for example, is, in, ≥, within last) depends on this type. After you choose a field, the UI only shows operators that are valid for that field's type.

## Common operators by type

| Filter type | Example fields      | Common operators       | Example usage                      |
| ----------- | ------------------- | ---------------------- | ---------------------------------- |
| Tags        | `tags`              | is, is not, in, not in | `tags is "baseline"`               |
| Time        | `created timestamp` | ≤, ≥, within last      | `created timestamp` ≥ `01/16/2026` |
| String      | `state`             | =, ≠, IN, NOT IN       | `state = "finished"`               |

<Note>
  The above table shows only a subset of available fields and operators. The expression editor shows all available fields and operators.
</Note>

## Create a filter expression

1. Navigate to the **Runs** tab from the project sidebar.
2. Select the **Filter** button, which looks like a funnel, above the runs table.
3. From left to right, select a column name, a logical operator, and a filter value to create a filter expression. The filter is applied as soon as you select a filter value.
4. Optionally select **New Filter** to apply additional AND or OR conditions.
5. Optionally select **New group** to group filters together with parentheses. This allows you to create complex filter expressions such as [A AND (B OR C)](#example-combine-filters-with-and-and-or-conditions).
6. Close the filter expression editor by clicking the **x** icon in the top right corner.

The following image filters runs based on loss values less than or equal to `1`:

<Frame>
  <img alt="Incorrect predictions filter" />
</Frame>

The following sections show some examples of how to filter runs in the Runs table.

### Example: Filter runs with tags

Filter runs based on their tags:

1. Click on the **Runs** tab from the project sidebar.
2. Select the **Filter** button, which looks like a funnel, above the runs table.
3. From left to right, select `"Tags"` from the dropdown menu select a logic operator.
4. Select is, is not, in, or not in from the second dropdown menu.
5. Enter the tag name you want to filter by from the third dropdown menu.

<Frame>
  <img alt="Filter runs by tags" />
</Frame>

### Example: Combine filters with AND and OR conditions

This section describes how to create a complex filter expression like `A AND (B OR C)`. In this example, A is `loss` ≤ `1`, B is `State` = `finished`, and C is `State` = `crashed`.

1. Click on the **Runs** tab from the project sidebar.
2. Select the **Filter** button, which looks like a funnel, above the runs table.
3. From left to right, select `loss` from the first dropdown menu, select ≤ from the second dropdown menu, and enter `1` in the third dropdown menu. This creates filter A.
4. Click **New Filter** and select **New group** to create a new group for filters B and C.
5. For filter B, select `State` from the first dropdown menu, select `=` from the second dropdown menu, and select `finished` from the third dropdown menu.
6. Click **New Filter** to create filter C, and select `State` from the first dropdown menu, select `=` from the second dropdown menu, and select `crashed` from the third dropdown menu.

The following image demonstrates how the resulting filter expression appears in the filter expression editor:

<Frame>
  <img alt="Combine filters with AND and OR conditions" />
</Frame>

## Default filters

By default, W\&B provides the following filters:

* **Show only my works**: Shows only runs created by the current user.
* **Hide crashed runs**: Hides runs with the `crashed` state.

Default filters appear as toggles below the **New filter** button in the filter expression editor.

## Remove a filter

To remove a filter from the Runs table:

1. Click on the **Filter** button, which looks like a funnel, above the runs table.
2. Select the `x` icon next to the filter you want to remove.


# Fork a run
Source: https://docs.wandb.ai/models/runs/forking

Explore different parameters or models from a specific point in an experiment without impacting the original run.

<Note>
  The ability to fork a run is in active development. It is in preview for Multi-tenant Cloud and Dedicated Cloud, and not yet available in Self-Managed.
</Note>

You can explore different hyperparameters or models from a specific point in an experiment without impacting the original run. To do this, fork from an existing W\&B run.

When you fork from a run, W\&B creates a new run using the source run’s [unique ID](/models/runs/run-identifiers#unique-run-identifiers) and a specified step. Summary metrics from the source run are copied to the forked run. The forked run shares all history and files from the source run up to the specified step.

After the fork step, you can log new data to the forked run independently of the original run.

View a [live demo](https://wandb.ai/wandb/test-fork-run/workspace?nw=nwuserjuliarose) of a forked run produced by the code below.

<Note>
  * Forking a run requires [`wandb`](https://pypi.org/project/wandb/) SDK version >= 0.16.5
  * Forking a run requires monotonically increasing steps. You cannot fork from a run that uses non-monotonic steps defined with [`define_metric()`](/models/ref/python/experiments/run#define_metric). Non-monotonic steps break the chronological order of run history and system metrics.
</Note>

Specify the source run's unique `run ID` and the `step` you want to start the forked run from as arguments to `fork_from` in [`wandb.init()`](/models/ref/python/functions/init).

## Fork from a previously logged run

The following code snippet shows how to fork from a run that was previously logged to W\&B.

First, obtain the run ID of the run you want to fork from. Next, specify the run ID and the step you want to fork from as arguments to `fork_from` in [`wandb.init()`](/models/ref/python/functions/init).

Copy and paste the following code into a Python script or notebook cell. Replace `<source-run-id>`, `<project>`, and `<entity>` with your own values:

```python theme={null}
import wandb

# The unique ID of the source run to fork from
source_run_id = "<source-run-id>"

# Specify the step to fork from
fork_step = 200

# Fork the run
with wandb.init(
    project="<project>",
    entity="<entity>",
    fork_from=f"{source_run_id}?_step={fork_step}",
) as forked_run:
    pass
```

## Fork from a run in the same script

The following code snippet shows how to create a run and fork from that run within the same script. This might occur if you want to fork from a run that you just created without having to look up the run ID in the W\&B App.

First, initialize a run and log some data. Next, use the original run object's `id` property to obtain the run ID of that run. Finally, initialize a new run and pass the original run's ID and the step you want to fork from as arguments to `fork_from` in [`wandb.init()`](/models/ref/python/functions/init).

```python theme={null}
import wandb

# Initialize a run
with wandb.init(
    project="<project>",
    entity="<entity>"
) as original_run:
    # ...training logic goes here ...
    pass

# Specify the step to fork from
fork_step = int("<num>")

# Use the original run's ID and specify the step to fork from
with wandb.init(
    project="<project>",
    entity="<entity>",
    fork_from=f"{original_run.id}?_step={fork_step}",
) as forked_run:
    # ...training logic goes here ...
    pass
```

<Tip>
  Use the `original_run.id` property to obtain the unique run ID of the original run.
</Tip>

### Example script

For example, the following code example shows how to first fork a run and then how to log metrics to the forked run starting from a training step of 200.

Copy and paste the following code into a Python script or notebook cell. Replace `<project>` and `<entity>` with your own values.

```python theme={null}
import wandb
import math

# Initialize the first run and log some metrics
with wandb.init(
    project="<project>",
    entity="<entity>"
) as run1:
    for i in range(300):
        run1.log({"metric": i})

# Fork from the first run at a specific step and log the
# metric starting from step 200
with wandb.init(
    project="<project>", 
    entity="<entity>", 
    fork_from=f"{run1.id}?_step=200"
) as run2:
    # Continue logging in the new run
    # For the first few steps, log the metric as is from run1
    # After step 250, start logging the spikey pattern
    for i in range(200, 300):
        if i < 250:
            # Continue logging from run1 without spikes
            metric_value = i
        else:
            # Introduce the spikey behavior starting from step 250
            metric_value = i + (2 * math.sin(i / 3.0))  # Apply a subtle spikey pattern

        # Log both metrics in a single call to ensure they're
        # logged at the same step
        run2.log({
            "metric": metric_value,
            "additional_metric": i * 1.1
        })
```

<Note>
  **Rewind and forking compatibility**

  Forking complements a [`rewind`](/models/runs/rewind/) by providing more flexibility in managing and experimenting with your runs.

  When you fork from a run, W\&B creates a new branch off a run at a specific point to try different parameters or models.

  When you rewind a run, W\&B lets you correct or modify the run history itself.
</Note>


# Organize runs
Source: https://docs.wandb.ai/models/runs/grouping

Organize your runs into groups and other properties.

Organize your runs into *groups*. A group is a collection of runs that share a common purpose, such as training runs for a specific model or evaluation runs for a specific dataset.

You can also organize runs by other properties such as *job type*. [Job types](/models/runs/grouping#organize-runs-by-job-type) indicate the function of a run, such as `preprocessing`, `training`, or `evaluation`.

## Organize runs into groups

You can add runs to a group programmatically using the W\&B Python SDK or interactively in the W\&B App.

<Note>
  W\&B stores group names as a run [`wandb.Run.group`](/models/ref/python/experiments/run#property-run-group) property.
</Note>

<Tabs>
  <Tab title="Python SDK">
    Programmatically add one or more runs to a group with the W\&B Python SDK. Pass the name of your group as an argument to the `group` parameter when you initialize a run with `wandb.init(group="")`. You can use group names to organize and filter runs in the W\&B App.

    The following example creates three groups named `A`, `B`, and `C`. Each group contains three runs.

    ```python theme={null}
    import wandb

    entity = "<entity>"
    project = "<project>"

    for group in ["A", "B", "C"]:
        for i in range(3):
            with wandb.init(entity=entity, project=project, group=group, name=f"{group}_run_{i}") as run:
                # Simulate some training
                for step in range(100):
                    run.log({
                        "acc": 0.5 + (step / 100) * 0.3 + (i * 0.05),
                        "loss": 1.0 - (step / 100) * 0.5
                    })
    ```

    In the project's workspace, you can view runs organized by group. The following image illustrates organizing the runs table by group name. Three groups named `A`, `B`, and `C` appear in the runs table, each containing three runs.

    <Frame>
      <img alt="Runs table grouped by group name" />
    </Frame>
  </Tab>

  <Tab title="W&B App">
    1. Navigate to your W\&B project.
    2. Select the **Runs** tab from the project sidebar.
    3. Above the list of runs, click the **Group** button.
    4. Click the checkbox next to one or more runs you want to group.
    5. Select **Move to group**.
    6. In the drawer, select an existing group or create a new group.
  </Tab>
</Tabs>

### View groups

View runs organized by group in the W\&B App:

1. In your project sidebar, select the **Runs** tab.
2. Above the list of runs, click the **Group** button.
3. From the dropdown, select a **Group**.

### Move runs between groups

Move runs from one group to another group:

1. Navigate to your W\&B project.
2. Select the **Runs** tab from the project sidebar.
3. Select one or more runs by clicking their checkboxes.
4. Above the table, click **Move to group**.
5. Within the drawer, select the target group or create a new group.
6. Click **Move**.

### Remove runs from a group

1. Navigate to your W\&B project.
2. Select the **Runs** tab from the project sidebar.
3. Above the list of runs, click the **Group** button.
4. From the dropdown, select the **X** next to the name of the group you want to remove.

### Delete a group

To delete a group, remove all runs from it. This automatically deletes the group.

## Organize runs by job type

Organize runs by their *job type*. A job type indicates the function of a run, such as `preprocessing`, `training`, or `evaluation`.

<Note>
  View a run's job type by accessing the run's [`wandb.Run.job_type`](/models/ref/python/experiments/run#property-run-job-type) property.
</Note>

Add a job type to a run by passing the `job_type` parameter to `wandb.init(job_type="")`. For example, the following code snippet creates runs with job types of either `training` or `evaluation`:

```python theme={null}
import wandb

entity = "<entity>"
project = "<project>"

for job_type in ["training", "evaluation"]:
    for i in range(2):
        with wandb.init(entity=entity, project=project, job_type=job_type, name=f"{job_type}_run_{i}") as run:
            # Simulate some process
            for step in range(50):
                run.log({
                    "metric1": 0.2 + (step / 50) * 0.4 + (i * 0.03),
                    "metric2": 0.8 - (step / 50) * 0.3
                })

```

The following image shows runs organized by job type:

<Frame>
  <img alt="Ungrouped runs table" />
</Frame>

### View runs organized by job type

View runs organized by group in the W\&B App:

1. In your project sidebar, select the **Runs** tab.
2. Above the list of runs, click the **Group** button.
3. From the dropdown, select **Job Type**.


# Visualize CoreWeave infrastructure alerts
Source: https://docs.wandb.ai/models/runs/infrastructure-alerts

View CoreWeave infrastructure alerts such as GPU failures and thermal violations on your W&B experiment run plots.

Observe infrastructure alerts such as GPU failures, thermal violations, and more during machine learning experiments you log to W\&B. When you run on a supported [CoreWeave Kubernetes Service (CKS)](https://docs.coreweave.com/products/cks) cluster, enable this integration, and satisfy the prerequisites on this page, [CoreWeave Mission Control](https://www.coreweave.com/mission-control) can monitor your compute infrastructure during a [W\&B run](/models/runs).

<Note>
  This feature is in Preview. Contact your W\&B representative for access.
</Note>

## Prerequisites

The following must be true for this integration to work end-to-end.

| Prerequisite                                      | Details                                                                                                                                                                                                                                                                                             |
| ------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **CoreWeave platform**                            | Available only on [CoreWeave Kubernetes Service (CKS)](https://docs.coreweave.com/products/cks) clusters. Not available on CoreWeave bare metal clusters or CoreWeave Classic. Training jobs running through [SUNK](https://docs.coreweave.com/products/sunk) on CKS also satisfy this requirement. |
| **W\&B Python SDK**                               | For training jobs, use the `wandb` package version `0.20.1` or later when you log a run.                                                                                                                                                                                                            |
| **W\&B Server (Dedicated Cloud or Self-Managed)** | If using a W\&B Dedicated Cloud or W\&B Self-Managed deployment, use W\&B Server version `0.73.0` or later. Set the `SERVER_FLAG_ENABLE_CORE_WEAVE_OBSERVABILITY` environment variable on the W\&B app pod so the server can accept CoreWeave observability data.                                   |

If an error occurs, CoreWeave sends that information to W\&B. W\&B populates infrastructure information onto your run's plots in your project's workspace. CoreWeave attempts to automatically resolve some issues, and W\&B surfaces that information in the run's page.


# Initialize runs
Source: https://docs.wandb.ai/models/runs/initialize-run

Initialize W&B runs with wandb.init() to start tracking experiments, including handling concurrent runs in one process.

Initialize a W\&B Run with [`wandb.init()`](/models/ref/python/functions/init).

By default, W\&B assumes each Python process has only one active run at a time when you call `wandb.init()`. If you call `wandb.init()` again, W\&B will either return the same run or finish the old run before starting a new one. How W\&B handles multiple calls to `wandb.init()` in the same process depends on the environment (notebook vs. non-notebook) and the `reinit` configuration.

To manage multiple active runs in the same process, see [Multiple runs in one process](/models/runs/initialize-run#multiple-runs-in-one-process).

<Note>
  W\&B recommends using a `with` block when calling `wandb.init()`. This ensures that W\&B properly finalizes the run and uploads all data when the block ends.
</Note>

## Single run per process

The following example code snippet shows how to import the W\&B Python SDK and initialize a run.

```python title="basic.py" theme={null}
import wandb

with wandb.init(entity="nico", project="awesome-project") as run:
    # Your training logic here
```

The code snippet produces the following output:

```bash theme={null}
🚀 View run exalted-darkness-6 at: 
https://wandb.ai/nico/awesome-project/runs/pgbn9y21
Find logs at: wandb/run-20241106_090747-pgbn9y21/logs
```

The output shows that W\&B logs the run `exalted-darkness-6` to the project `awesome-project` under the entity `nico`. `pgbn9y21` is the unique run ID that W\&B generates for this run.

## Multiple runs in one process

Manage multiple runs in a single Python process. This is useful for workflows where you want to keep a primary process active while creating short-lived secondary processes for sub-tasks. Some use cases include:

* Keeping a single “primary” run active throughout a script while spinning up short-lived “secondary” runs for evaluations or sub-tasks.
* Orchestrating sub-experiments in a single file.
* Logging from one “main” process to several runs that represent different tasks or time periods.

By default, W\&B assumes each Python process has only one active run at a time when you call `wandb.init()`. If you call `wandb.init()` again, W\&B will either return the same run or finish the old run before starting a new one, depending on the configuration.

The content in this guide explains how to use `reinit` to modify the `wandb.init()` behavior to enable multiple runs in a single Python process.

<Note>
  **Requirements**

  To manage multiple runs in a single Python process, you must have W\&B Python SDK version `v0.19.10` or newer.
</Note>

### `reinit` options

Use the `reinit` parameter to configure how W\&B handles multiple calls to `wandb.init()`. The following table describes valid arguments and their effects:

|                   | Description                                                                                                                                                                                                                                                                                                                            | Creates a run? | Example use case                                                                                                                                |
| ----------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------- | ----------------------------------------------------------------------------------------------------------------------------------------------- |
| `create_new`      | Create a new run with `wandb.init()` without finishing existing, active runs. W\&B does not automatically switch the global `wandb.Run` to new runs. You must hold onto each run object yourself. See the [multiple runs in one process example](/models/runs/initialize-run/#example-multiple-runs-in-one-process) below for details. | Yes            | Ideal for creating and managing concurrent processes. For example, a “primary” run that remains active while you start or end “secondary” runs. |
| `finish_previous` | Finish all active runs with `run.finish()` before creating a new one run with `wandb.init()`. Default behavior for non notebook environments.                                                                                                                                                                                          | Yes            | Ideal when you want to break sequential sub-processes into separate individual runs.                                                            |
| `return_previous` | Return the most recent, unfinished run. Default behavior for notebook environments.                                                                                                                                                                                                                                                    | No             |                                                                                                                                                 |

<Note>
  W\&B does not support `create_new` mode for [W\&B Integrations](/models/integrations) that assume a single global run, such as Hugging Face Trainer, Keras callbacks, and PyTorch Lightning. If you use these integrations, you should run each sub-experiment in a separate process.
</Note>

### Specifying `reinit`

* Use `wandb.init()` with the `reinit` argument directly:
  ```python theme={null}
  import wandb
  with wandb.init(reinit="<create_new|finish_previous|return_previous>") as run:
      # Your code here
  ```

* Use `wandb.init()` and pass a `wandb.Settings` object to the `settings` parameter. Specify `reinit` in the `Settings` object:

  ```python theme={null}
  import wandb
  with wandb.init(settings=wandb.Settings(reinit="<create_new|finish_previous|return_previous>")) as run:
      # Your code here
  ```

* Use `wandb.setup()` to set the `reinit` option globally for all runs in the current process. This is useful if you want to configure the behavior once and have it apply to all subsequent `wandb.init()` calls in that process.

  ```python theme={null}
  import wandb
  with wandb.setup(wandb.Settings(reinit="<create_new|finish_previous|return_previous>")) as run:
       # Your code here
  ```

* Specify the desired value for `reinit` in the environment variable `WANDB_REINIT`. Defining an environment variable applies the `reinit` option to `wandb.init()` calls.

  ```bash theme={null}
  export WANDB_REINIT="<create_new|finish_previous|return_previous>"
  ```

The following code snippet shows a high level overview how to set up W\&B to create a new run each time you call `wandb.init()`:

```python theme={null}
import wandb

wandb.setup(wandb.Settings(reinit="create_new"))

with wandb.init() as experiment_results_run:
    # This run will be used to log the results of each experiment.
    # You can think of this as a parent run that collects results
      with wandb.init() as run:
         # The do_experiment() function logs fine-grained metrics
         # to the given run and returns result metrics that
         # you want to track separately.
         experiment_results = do_experiment(run)

         # After each experiment, log its results to a parent
         # run. Each point in the parent run's charts corresponds
         # to one experiment's results.
         experiment_results_run.log(experiment_results)
```

### Example: Concurrent processes

Suppose you want to create a primary process that remains open for the script's entire lifespan, while periodically spawning short-lived secondary processes without finishing the primary process. For example, this pattern can be useful if you want to train a model in the primary run, but compute evaluations or do other work in separate runs.

To achieve this, use `reinit="create_new"` and initialize multiple runs. For this example, suppose "Run A" is the primary process that remains open throughout the script, while "Run B1", "Run B2", are short-lived secondary runs for tasks like evaluation.

The high level workflow might look like this:

1. Initialize the primary process Run A with `wandb.init()` and log training metrics.
2. Initialize Run B1 (with `wandb.init()`), log data, then finish it.
3. Log more data to Run A.
4. Initialize Run B2, log data, then finish it.
5. Continue logging to Run A.
6. Finally finish Run A at the end.

The following Python code example demonstrates this workflow:

```python theme={null}
import wandb

def train(name: str) -> None:
    """Perform one training iteration in its own W&B run.

    Using a 'with wandb.init()' block with `reinit="create_new"` ensures that
    this training sub-run can be created even if another run (like our primary
    tracking run) is already active.
    """
    with wandb.init(
        project="my_project",
        name=name,
        reinit="create_new"
    ) as run:
        # In a real script, you'd run your training steps inside this block.
        run.log({"train_loss": 0.42})  # Replace with your real metric(s)

def evaluate_loss_accuracy() -> (float, float):
    """Returns the current model's loss and accuracy.
    
    Replace this placeholder with your real evaluation logic.
    """
    return 0.27, 0.91  # Example metric values

# Create a 'primary' run that remains active throughout multiple train/eval steps.
with wandb.init(
    project="my_project",
    name="tracking_run",
    reinit="create_new"
) as tracking_run:
    # 1) Train once under a sub-run named 'training_1'
    train("training_1")
    loss, accuracy = evaluate_loss_accuracy()
    tracking_run.log({"eval_loss": loss, "eval_accuracy": accuracy})

    # 2) Train again under a sub-run named 'training_2'
    train("training_2")
    loss, accuracy = evaluate_loss_accuracy()
    tracking_run.log({"eval_loss": loss, "eval_accuracy": accuracy})
    
    # The 'tracking_run' finishes automatically when this 'with' block ends.
```

Note three key points from the previous example:

1. `reinit="create_new"` creates a new run each time you call `wandb.init()`.
2. You keep references of each run. `wandb.run` does not automatically point to the new run created with `reinit="create_new"`. Store new runs in variables like `run_a`, `run_b1`, etc., and call `.log()` or `.finish()` on those objects as needed.
3. You can finish sub-runs whenever you want while keeping the primary run open until.
4. Finish your runs with `run.finish()` when you are done logging to them. This ensures that all data is uploaded and the run is properly closed.


# Move a run to a different project or team
Source: https://docs.wandb.ai/models/runs/manage-runs

Move runs between projects or teams using the W&B App.

<Note>
  Before you begin, ensure you have the necessary permissions to move runs between projects or teams. You must have access to the run at its current and new locations.
</Note>

To move runs from one project to another or between teams:

1. Navigate to the project that contains the runs you want to move.
2. Select the **Runs** tab from the project sidebar.
3. Select the checkbox next to the runs you want to move.
4. Click the **Move to project** button above the table.
5. Select the destination team and project from the dropdown.

<Note>
  When you move a run, historical artifacts associated with it are not moved. To move an artifact manually, you can use the [`wandb artifact get`](/models/ref/cli/wandb-artifact/wandb-artifact-get/) SDK command or the [`Api.artifact` API](/models/ref/python/public-api/api/#artifact) to download the artifact, then use [`wandb artifact put`](/models/ref/cli/wandb-artifact/wandb-artifact-put/) or the `Api.artifact` API to upload it to the run's new location.
</Note>


# Resume a run
Source: https://docs.wandb.ai/models/runs/resuming

Resume paused, stopped, or crashed W&B runs using the resume parameter options in wandb.init().

Specify how W\&B should respond if a run stops or crashes by setting the `resume` parameter in `wandb.init()`. When you initialize a run, W\&B checks whether the run ID already exists and applies the behavior defined by the `resume` value.

The following table outlines the behavior of W\&B based on the argument passed to the `resume` parameter and whether the run ID exists or not.

| Argument  | Description                                                                                                                    | Run ID exists                                                  | Run ID does not exist                             | Use case                                         |
| --------- | ------------------------------------------------------------------------------------------------------------------------------ | -------------------------------------------------------------- | ------------------------------------------------- | ------------------------------------------------ |
| `"must"`  | W\&B must resume run specified by the run ID.                                                                                  | W\&B resumes run with the same run ID. Resumes from last step. | W\&B raises an error.                             | Resume a run that must use the same run ID.      |
| `"allow"` | Allow W\&B to resume run if run ID exists.                                                                                     | W\&B resumes run with the same run ID. Resumes from last step. | W\&B initializes a new run with specified run ID. | Resume a run without overriding an existing run. |
| `"never"` | Never allow W\&B to resume a run specified by the run ID.                                                                      | Raise an error if a run with the specified ID already exists.  | W\&B initializes a new run with specified run ID. |                                                  |
| `"auto"`  | Allow W\&B to automatically try to resume run if run ID exists. Restart the run from the same directory as the failed process. | W\&B resumes run with the same run ID.                         | W\&B initializes a new run with specified run ID. | Enable runs to automatically resume.             |

<Note>
  **When to use `auto` vs `allow`**

  W\&B recommends that you use `resume="allow"` and specify the specific run ID you want to resume.

  The `resume="auto"` option does not require you to specify a run ID but it can lead to unexpected behavior if you have multiple runs that fail in the same directory or if the file directory structure changes. You must also ensure that you restart the run from the same directory as the failed process when you use `resume="auto"`.
</Note>

For all the examples below, replace values enclosed within `<>` with your own.
<Tip>[View a live demo of a resumed run](https://wandb.ai/wandb/resume-run/workspace?nw=nwuserjuliarose).</Tip>

## Resume a run that must use the same run ID

If a run is stopped, crashes, or fails, you can resume it using the same run ID. To do so, initialize a run and specify the following:

* Set the `resume` parameter to `"must"` (`resume="must"`)
* Provide the run ID of the run that stopped or crashed

The following code snippet shows how to accomplish this with the W\&B Python SDK:

```python theme={null}
with wandb.init(entity="<entity>", project="<project>", resume="must") as run:
        # Your training code here
```

<Warning>
  Unexpected results will occur if multiple processes use the same `id` concurrently.

  For more information on  how to manage multiple processes, see the [Log distributed training experiments](/models/track/log/distributed-training/)
</Warning>

## Resume a run without overriding the existing run

Resume a run that stopped or crashed without overriding the existing run. This is especially helpful if your process doesn't exit successfully. The next time you start W\&B, W\&B will start logging from the last step.

Set the `resume` parameter to `"allow"` (`resume="allow"`) when you initialize a run with W\&B. Provide the run ID of the run that stopped or crashed. The following code snippet shows how to accomplish this with the W\&B Python SDK:

```python theme={null}
import wandb

with wandb.init(entity="<entity>", project="<project>", id="<run ID>", resume="allow") as run:
        # Your training code here
```

## Enable runs to automatically resume

The following code snippet shows how to enable runs to automatically resume with the Python SDK or with environment variables.

<Tabs>
  <Tab title="W&B Python SDK">
    Pass `auto` as an argument to the `resume` parameter when you initialize a run. Ensure that you restart the run from the same directory as the failed process.

    Copy and paste the following code snippet. Replace values enclosed within `<>` with your own:

    ```python theme={null}
    with wandb.init(entity="<entity>", project="<project>", id="<run ID>", resume="auto") as run:
            # Your training code here
    ```
  </Tab>

  <Tab title="Shell script">
    The following example shows how to specify the W\&B `WANDB_RUN_ID` variable in a bash script:

    ```bash title="run_experiment.sh" theme={null}
    RUN_ID="$1"

    WANDB_RESUME=auto WANDB_RUN_ID="$RUN_ID" python eval.py
    ```

    Within your terminal, you could run the shell script along with the W\&B run ID. The following code snippet passes the run ID `akj172`:

    ```bash theme={null}
    sh run_experiment.sh akj172 
    ```
  </Tab>
</Tabs>

<Warning>
  Automatic resuming only works if the process is restarted on top of the same filesystem as the failed process.
</Warning>

For example, suppose you execute a python script called `train.py` in a directory called `Users/AwesomeEmployee/Desktop/ImageClassify/training/`. Within `train.py`, the script creates a run that enables automatic resuming. Suppose next that the training script is stopped. To resume this run, you would need to restart your `train.py` script within `Users/AwesomeEmployee/Desktop/ImageClassify/training/` .

<Note>
  If you can not share a filesystem, specify the `WANDB_RUN_ID` environment variable or pass the run ID with the W\&B Python SDK. See the [Custom run IDs](./#custom-run-ids) section in the "What are runs?" page for more information on run IDs.
</Note>

## Resume preemptible Sweeps runs

When you handle preemption correctly, interrupted [sweep](/models/sweeps/) runs can be requeued automatically so another agent can pick them up. That pattern is especially helpful when the sweep agent runs in a preemptible environment, such as a SLURM job in a preemptible queue, an EC2 Spot instance, or a Google Cloud preemptible VM.

The behavior below applies when you run sweep agents with the [`wandb agent`](/models/ref/cli/wandb-agent) CLI, which starts your training program as a **subprocess**. It does not fully apply when you use only the Python API [`wandb.agent()`](/models/ref/python/functions/agent), because that path runs your training function in a thread rather than a separate process, so OS signal delivery and forwarding do not match the CLI agent model.

**Recommended pattern:** Register a signal handler for the preemption signal your scheduler or platform uses (for example `SIGUSR1` or `SIGTERM`). In the handler, call [`mark_preempting()`](/models/ref/python/experiments/run#mark_preempting) when a run is active, perform any cleanup (such as saving a checkpoint), then exit with a non-zero code (a common convention is `128 + signum` for signal termination). Do **not** call `mark_preempting()` unconditionally immediately after `wandb.init()`. Doing so can mark every failure, including code bugs, as preemption and requeue the run repeatedly.

For runnable examples, `--forward-signals` on the CLI agent, and a full reference table for different uses of `mark_preempting()`, see [Signal handling and sweep runs](/models/sweeps/signal-handling-sweep-runs).

When you follow that pattern, W\&B records run state roughly as follows:

| Scenario                                                                                                                                             | Run state                                               |
| ---------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------- |
| Run completes normally with exit code 0                                                                                                              | FINISHED                                                |
| Run fails with a non-zero exit code                                                                                                                  | FAILED                                                  |
| Run receives an unhandled signal (for example `SIGKILL`)                                                                                             | CRASHED after about five minutes                        |
| Run receives a handled preemption signal (for example `SIGTERM` or `SIGUSR1`), the handler calls `mark_preempting()`, and the process exits non-zero | PREEMPTED; the run is queued for the next agent request |

Sweep agents drain the run queue before the sweep generates new hyperparameter combinations from the search algorithm. Once the queue is empty, the sweep resumes normal scheduling.


# Rewind a run
Source: https://docs.wandb.ai/models/runs/rewind

Rewind a run to correct or modify its history without losing original data.

Rewind a run to modify the history of a run. When you rewind a run, W\&B resets the state of the run to the specified step while maintaining the same run ID.

<Note>
  The ability to rewind a run is in active development. It is in preview for Multi-tenant Cloud and Dedicated Cloud, and not yet available in Self-Managed.

  Due to known performance limitations with rewind, W\&B typically recommends [Forking](./forking) as an alternative.
</Note>

### Prerequisites

Before you rewind a run, ensure you meet the following prerequisites:

* To rewind a run, you must have [W\&B Python SDK](https://pypi.org/project/wandb/) version >= `0.17.1`.
* You must use monotonically increasing steps. This does not work with non-monotonic steps defined with [`define_metric()`](/models/ref/python/experiments/run#define_metric) because it disrupts the required chronological order of run history.

### Limitations

Rewind does not support the following:

* **Log rewind**: Logs are reset in the new run segment.
* **System metrics rewind**: W\&B logs only new system metrics after the rewind point.
* **Artifact association**: W\&B associates artifacts with the source run that produces them.

W\&B recomputes the summary metrics for the run you rewind based on the newly logged history, with the following results:

* **History truncation**: W\&B truncates the history to the rewind point, allowing new data logging.
* **Summary metrics**: Recomputed based on the newly logged history.
* **Configuration preservation**: W\&B preserves the original configurations and you can merge new configurations.

### Rewind and forking compatibility

Forking complements a rewind.

When you fork from a run, W\&B creates a new branch off a run at a specific point to try different parameters or models.

When you rewind a run, W\&B lets you correct or modify the run history itself.

## Rewind a run

Rewind a run from a specific step and log new data from that point in time. Pass both the run ID and the step you want to rewind from as arguments to the `resume_from` parameter in [`wandb.init()`](/models/ref/python/functions/init). The `resume_from` parameter accepts a string in the format of `<run ID>?_step=<step>`, where `<run ID>` is the run ID of the run you want to rewind and `<step>` is the step you want to rewind from.

Suppose you log a linear line for 300 steps:

```python theme={null}
import wandb

# Initialize the first run and log some metrics
with wandb.init(project="<project>", entity="wandb") as run:
    for i in range(300):
        # Plot a linear line
        run.log({"metric": i, "step": i})
```

Within your project's workspace, you see a line plot from step 0 to step 300:

<img alt="Line plot of the original run" />

At a later time, you want to rewind the run from step 200 and you want to log a new metric called `additional_metric` that logs `i*1.1` from step 200 to step 300. From step 250 you want to log a new subtle wavy pattern (`i**2 + 2*sin(i/3)`) instead of a linear line:

```python theme={null}
import math

run_ID = "<run_ID>" # Replace with the run ID of the run you want to rewind

# Rewind from the first run at a specific step and log the metric starting from step 200
with wandb.init(project="<project>", entity="wandb", resume_from=f"{run_ID}?_step=200") as run:

    # For the first few steps, log the metric as is from run
    # After step 250, start logging the wavy pattern
    for i in range(200, 300):
        if i < 250:
            run.log({"metric": i, "step": i})  # Continue logging from run without waves
        else:
            # Introduce the wavy behavior starting from step 250
            subtle_wave = i + (2 * math.sin(i / 3.0))  # Apply a subtle wavy pattern
            run.log({"metric": subtle_wave, "step": i})
        # Additionally log the new metric at all steps
        run.log({"additional_metric": i * 1.1, "step": i})
```

The following image shows the updated project's workspace. Note the following changes in the plot after the rewind:

* The line plot shows the original linear line from step 0 to step 200 and the new subtle wavy pattern starts from step 250 (left image).
* W\&B created a new plot (right plot) labeled `additional_metric` that starts from step 200.

<img alt="From left to right: original linear line and additional metric" />

## View an archived run

After you rewind a run, you can explore the original archived run in the W\&B App. Follow these steps to view an archived run:

1. **Access the Overview Tab:** Navigate to the [**Overview** tab](./#overview-tab) on the run's page. This tab provides a comprehensive view of the run's details and history.
2. **Locate the Forked From field:** Within the **Overview** tab, find the `Forked From` field. This field captures the history of the resumptions. The **Forked From** field includes a link to the source run, allowing you to trace back to the original run and understand the entire rewind history.

By using the `Forked From` field, you can effortlessly navigate the tree of archived resumptions and gain insights into the sequence and origin of each rewind.

## Fork from a run that you rewind

To fork from a rewound run, use the [`fork_from`](/models/runs/forking/) argument in `wandb.init()` and specify the source run ID and the step from the source run to fork from:

```python theme={null}
import wandb

# Fork the run from a specific step
forked_run = wandb.init(
    project="<project>",
    entity="<entity>",
    fork_from=f"{rewind_run.id}?_step=500",
)

# Continue logging in the new run
for i in range(500, 1000):
    forked_run.log({"metric": i*3})
forked_run.finish()
```


# Customize run colors
Source: https://docs.wandb.ai/models/runs/run-colors

Customize, randomize, and reset the colors assigned to individual runs in your W&B project workspace.

W\&B automatically assigns a color to each run that you create in your project. You can change the default color of a run to help you visually distinguish it from other runs in the table and graphs. Reset your project workspace to restore the default colors for all runs in the table.

Run colors are locally scoped. On the project page, custom colors apply only to your own workspace. In reports, custom colors for runs apply only at the section level. You can visualize the same run in different sections, which can use different custom colors per section.

## Edit default run colors

1. Click the **Runs** tab from the project sidebar.
2. Click the dot color next to the run name in the **Name** column.
3. Select a color from the color palette or the color picker, or enter a hex code.

<Image alt="Edit default run color in project workspace" />

## Randomize run colors

To randomize the colors of all runs in the table:

1. Click the **Runs** tab from the project sidebar.
2. Hover over the **Name** column header, click the **action (<Icon icon="ellipsis" />)** menu, and select **Randomize run colors** from the dropdown menu.

<Note>
  The option to randomize run colors is available only after modify the run's table in some way, such as by sorting, filtering, searching, or grouping.
</Note>

## Reset run colors

To restore the default colors for all runs in the table:

1. Click the **Runs** tab from the project sidebar.
2. Hover over the **Name** column header, click the **action (<Icon icon="ellipsis" />)** menu, and select **Reset colors** from the dropdown menu.

<Image alt="Reset run colors in project workspace" />


# Find and customize a run's ID or name
Source: https://docs.wandb.ai/models/runs/run-identifiers

Learn how to find a run's unique identifier and run name, how to create a custom run ID, and how to customize a run's name.

When you initialize a W\&B Run, W\&B assigns that run a [unique identifier known as a *run ID*](/models/runs/run-identifiers#run-id). Each run also has a human-readable [non-unique *run name*](/models/runs/run-identifiers#run-name) that you can customize.

## Run ID

A run's ID uniquely identifies the run. By default, W\&B generates a [random and unique run ID](#autogenerated-run-ids) automatically when you initialize a new run, unless you [specify your own unique run ID](#create-a-custom-run-id) when you [initialize the run](/models/runs/initialize-run).

### Find a run's ID

Find a run's unique ID programmatically with the W\&B Python SDK or interactively in the W\&B App.

<Tabs>
  <Tab title="Python SDK">
    When you initialize a run, W\&B returns the unique run ID in the terminal. For example, consider the following code snippet that initializes a W\&B run:

    ```python theme={null}
    import wandb
    entity = "nico"  # Replace with your W&B entity
    project = "awesome-project" 
    with wandb.init(entity=entity, project=project) as run:
        # Your code here
    ```

    Within the terminal, W\&B returns:

    ```bash theme={null}
    wandb: Syncing run earnest-sunset-1
    wandb: ⭐️ View project at https://wandb.ai/nico/awesome-project
    wandb: 🚀 View run at https://wandb.ai/nico/awesome-project/runs/1jx1ud12
    ```

    The last part of the run URL (`1jx1ud12`) is the unique run ID.
  </Tab>

  <Tab title="W&B App">
    You can also find a run's unique ID in the W\&B App:

    1. Navigate to the [W\&B App](https://wandb.ai/home).
    2. Navigate to the W\&B project you specified when you initialized the run.
    3. Within your project's workspace, select either the  **Workspace** or **Runs** tab.
    4. Select the run you want to view.
    5. Choose the **Overview** tab.

    W\&B displays the run ID in the **Run path** field. The run path consists of the name of your team, the name of the project, and the run ID. The unique ID is the last part of the run path.

    For example, in the following image, the unique run ID is `9mxi1arc`:

    <Frame>
      <img alt="Run ID location" />
    </Frame>
  </Tab>
</Tabs>

<Tip>
  Use a run's unique ID to directly navigate to that run's overview page in the W\&B App. The following code block shows the format of a URL path for a run:

  ```text title="W&B App URL for a specific run" theme={null}
  https://wandb.ai/<entity>/<project>/<run-id>
  ```

  Replace values enclosed in angle brackets (`< >`) with the actual values of the entity, project, and run ID.
</Tip>

### Create a custom run ID

Pass your desired run ID as a string to the `id` parameter when you initialize a run:

```python theme={null}
import wandb

with wandb.init(entity="<project>", project="<project>", id="<run-id>") as run:
    # Your code here
```

## Run name

Each run has a human-readable, non-unique run name. By default, W\&B generates a random run name when you initialize a new run if you do not specify a run name for it. The name of a run appears within your project's workspace and at the top of the [run's **Overview** page](#overview-tab).

Continuing from the previous example, the name of the run is `glowing-shadows-8`.

<Frame>
  <img alt="Run ID location" />
</Frame>

You can name your run when [you initialize it](/models/runs/run-identifiers#create-a-custom-run-name) or [rename](/models/runs/run-identifiers#rename-a-run) it at a later time.

### Create a custom run name

Specify a name for your run by passing the `name` parameter to the [`wandb.init()`](/models/ref/python/functions/init) method.

```python theme={null}
import wandb

with wandb.init(entity="<project>", project="<project>", name="<run-name>") as run:
    # Your code here
```

### Rename a run

Rename a run after initializing it programmatically with the Python SDK or interactively in the W\&B App.

<Tabs>
  <Tab title="Python SDK">
    Use [`wandb.Api.Run`](/models/ref/python/public-api/api#method-api-run) to access a run logged to W\&B. This method returns a [run object](/models/ref/python/public-api/run#property-run-name) that you can use to update the run name. Call `wandb.Api.Run.update()` method to persist changes.

    Replace the values enclosed in angle brackets (`< >`) with your own values.

    ```python theme={null}
    import wandb

    api = wandb.Api()

    # Access run by its path
    run = api.run(path = "<entity>/<project>/<run-id>")

    # Specify a new run name
    run.name = "<new-run-name>"
    run.update()
    ```
  </Tab>

  <Tab title="W&B App">
    1. Navigate to your W\&B project.
    2. Select the **Workspace** or **Runs** tab.
    3. Search or scroll to the run you want to rename.
    4. Hover over the run name, click the **action (<Icon icon="ellipsis-vertical" />)** menu, then click **Rename run**.
    5. To change the run name, update the **Run name** field.
    6. Click **Save**.
  </Tab>
</Tabs>

## Run display name

Each run also has a *run display name* that you can customize for each workspace.

<Note>
  If you change a run's display name in one workspace, the display name changes only for that workspace, not in other workspaces or projects.
</Note>

The display name defaults to the same value as the run name. The display name appears in the run's workspace and runs table.

Use the run display name to override the run name displayed in that workspace without renaming the run in the project.

### Rename a run's display name

Change a run's display name from the W\&B App:

1. Navigate to your W\&B project.
2. Select the **Workspace** or **Runs** tab.
3. Search or scroll to the run you want to rename.
4. Hover over the run name, click the **action (<Icon icon="ellipsis-vertical" />)** menu, then click **Rename run**.
5. Specify a new value for the **Display name** field.
6. Click **Save**.

## Customize run name truncation

By default, long run names are truncated in the middle for readability. To customize the truncation of run names:

1. Click the **action (<Icon icon="ellipsis" />)** menu at the top of the list of runs.
2. Set **Run name cropping** to crop the end, middle, or beginning.


# Run states
Source: https://docs.wandb.ai/models/runs/run-states

Learn about the different states a W&B run can have.

The [Run state](/models/runs/run-states#run-states) indicates the current status of a W\&B run. You can [view the state](/models/runs/run-states#view-the-state-of-a-run) of a run in the W\&B App or programmatically using the W\&B Python SDK.

## Run states

The following table describes the possible states a run can be in:

| State      | Description                                                                                      |
| ---------- | ------------------------------------------------------------------------------------------------ |
| `Crashed`  | Run stopped sending heartbeats in the internal process, which can happen if the machine crashes. |
| `Failed`   | Run ended with a non-zero exit status.                                                           |
| `Finished` | Run ended and fully synced data, or called `wandb.Run.finish()`.                                 |
| `Killed`   | Run was forcibly stopped before it could finish.                                                 |
| `Running`  | Run is still running and has recently sent a heartbeat.                                          |
| `Pending`  | Run is scheduled but not yet started (common in sweeps and Launch jobs).                         |

### Run states in sweeps

When runs are part of a [sweep](/models/sweeps/), their states behave independently from the sweep's status:

* **Individual run states** reflect each run's execution status (Running, Finished, Failed, etc.)
* **Sweep status** controls whether new runs are created, not how existing runs execute
* Pausing or stopping a sweep doesn't affect already-running runs
* Only cancelling a sweep forcibly kills running runs (changes their state to `Killed`)

For a detailed explanation of how sweep and run statuses interact, see [Understanding sweep and run statuses](/models/sweeps/pause-resume-and-cancel-sweeps#understanding-sweep-and-run-statuses).

## View the state of a run

Programmatically or interactively view a run's state with the Python SDK or W\&B App.

<Tabs>
  <Tab title="Python SDK">
    Use the `state` property of the [`wandb.Api.Run`](/models/ref/python/public-api/runs) object to access the current state of a run.

    The following code snippet retrieves and prints the state of all runs in a specified project. Copy and paste the following code snippet into your Python environment. Replace the values enclosed in angle brackets (`< >`) with your own values:

    ```python theme={null}
    import wandb

    api = wandb.Api()

    runs = api.runs(path="<entity>/<project>")

    # Access run object's properties
    for run in runs:
        print(f"Run: {run.name}")
        print(f"Run state: {run.state}")
        print()
    ```

    You can apply different filters to retrieve runs from your projects based on different criteria. See [Filter runs](/models/runs/filter-runs) to learn more about filtering runs programmatically.
  </Tab>

  <Tab title="W&B App">
    View the state of a run from the W\&B App:

    1. Navigate to your W\&B project.
    2. Select the **Workspace** or **Runs** tab from the project sidebar.
    3. Search or scroll to the run you want to view.
    4. Select the run to open the run overview page.
    5. Choose the **Overview** tab.

    Next to the **State** field, view the current state of the run.
  </Tab>
</Tabs>


# Search runs
Source: https://docs.wandb.ai/models/runs/search-runs

Learn how to search for specific runs by name or ID in your project's Runs table or Workspace.

Use the search box within your project's Runs table or Workspace to find specific [runs by name or ID](/models/runs/run-identifiers).

<Note>
  By default, the search box uses regular expressions (RegEx) to match your query against run names or IDs.
</Note>

## Search for runs by name or ID

1. Click either the **Runs** tab or **Workspace** from the project sidebar.
2. Click on the search box at the top of the runs table.
3. Enter the run name or run ID you want to search for.

## Turn off regular expressions search

1. Click either the **Runs** tab or **Workspace** from the project sidebar.
2. Click on the search box at the top of the runs table.
3. Toggle off the **RegEx** toggle (.\*) so that it is gray.


# Stop runs
Source: https://docs.wandb.ai/models/runs/stop-runs

Stop runs programmatically using the W&B Python SDK or manually from the W&B App.

Stop a run programmatically with the W\&B Python SDK or interactively in the W\&B App.

<Tabs>
  <Tab title="Python SDK">
    1. Navigate to the terminal or code editor where you initialized the run.
    2. Press `Ctrl+D` to stop the run.
  </Tab>

  <Tab title="W&B App">
    1. Navigate to the project that your run is logging to.
    2. Select the run you want to stop within the run selector.
    3. Choose the **Overview** tab.
    4. Select the stop button next to the **State** field.

    Next to the **State** field, the run's state changes from `running` to `Killed`.
  </Tab>
</Tabs>

See [State fields](/models/runs/run-states) for a full list of possible run states.


# Add labels to runs with tags
Source: https://docs.wandb.ai/models/runs/tags

Add, update, and remove tags on W&B runs using the Python SDK, Public API, or the W&B App UI for organization.

Add tags to label runs with particular features that might not be obvious from the logged metrics or artifact data.

For example, you can add a tag to a run to indicated that run's model is `in_production`, that run is `preemptible`, this run represents the `baseline`, and so forth.

## Add tags to one or more runs

Programmatically or interactively add tags to your runs.

Based on your use case, select a tab below that best fits your needs:

<Tabs>
  <Tab title="W&B Python SDK">
    Use `wandb.init()` to add tags when you initialize a run. Pass a list of strings to the `tags` parameter in `wandb.init()` to add tags to a run. For example:

    ```python theme={null}
    import wandb

    with wandb.init(
      entity="<entity>",
      project="<project>",
      tags=["<tag1>", "<tag2>"]
    ) as run:
        # Your training code here
    ```

    You can also add or update an existing tag during an active run by updating the `tags` attribute of the run object (`wandb.Run.tags`). The `tags` attribute accepts a tuple of strings. Concatenate one or more tags to the existing run tag property to add new tags after you initialize the run:

    ```python theme={null}
    import wandb

    with wandb.init(entity="<entity>", project="<project>", tags=["<tag1>"]) as run:
      # Training loop logic here

      # Add a new tag to the run object
      run.tags += ("<new_tag>",)
    ```
  </Tab>

  <Tab title="Public API">
    Use the [W\&B Public API](/models/ref/python/public-api) to add or update tags to a previously saved run.

    To update tags on an existing run, access the `wandb.Run.tags` property. `wandb.Run.tags` property consists of a list of strings. Concatenate the new tag or tags to the existing tags and then call `wandb.Run.update()` to update the run with the new tags. For example:

    ```python theme={null}
    with wandb.Api().run("{entity}/{project}/{run-id}") as run:
      run.tags.append("<tag1>")
      run.update()
    ```
  </Tab>

  <Tab title="Project page">
    This method is best suited to tagging large numbers of runs with the same tag or tags.

    1. Navigate to your project workspace.
    2. Select **Runs** from the project sidebar.
    3. Select one or more runs from the table.
    4. Once you select one or more runs, select the **Tag** button above the table.
    5. Type the tag you want to add and select the **Create new tag** checkbox to add the tag.
  </Tab>

  <Tab title="Run page">
    This method is best suited to applying a tag or tags to a single run manually.

    1. Navigate to your project workspace.
    2. Click a run to open it. The run page opens with the **Overview** tab shown by default.
    3. Select the gray plus icon (**+**) button next to **Tags**.
    4. Type a tag you want to add and select **Add** below the text box to add a new tag.
  </Tab>
</Tabs>

## Remove tags from one or more runs

Follow these steps to remove tags from a run in the W\&B App.

<Tabs>
  <Tab title="Project page">
    This method is best suited to removing tags from a large numbers of runs.

    1. In the Run sidebar of the project, select the table icon in the upper-right. This will expand the sidebar into the full runs table.
    2. Hover over a run in the table to see a checkbox on the left or look in the header row for a checkbox to select all runs.
    3. Select the checkbox to enable bulk actions.
    4. Select the runs you want to remove tags.
    5. Select the **Tag** button above the rows of runs.
    6. Select the checkbox next to a tag to remove it from the run.
  </Tab>

  <Tab title="Run page">
    1. In the left sidebar of the Run page, select the top **Overview** tab. The tags on the run are visible here.
    2. Hover over a tag and select the "x" to remove it from the run.
  </Tab>
</Tabs>


# View a specific run in a project
Source: https://docs.wandb.ai/models/runs/view-logged-runs

Learn how to view a specific logged run and its properties using the W&B App or the LEET terminal UI.

View information about a specific run, such as its current state, artifacts, metrics, and more. You can view and monitor runs using the W\&B App or the `wandb beta leet` terminal UI.

<Tabs>
  <Tab title="WA&B App">
    To view a specific run in the W\&B App:

    1. Navigate to the [W\&B App](https://wandb.ai/home).
    2. Navigate to the W\&B project you specified when you initialized the run.
    3. Within the project sidebar, select the **Workspace** tab.
    4. Within the run selector, click the run you want to view, or enter a partial run name to filter for matching runs.

    Alternatively, you can directly access a specific run's workspace by entering its URL in your browser. The URL path of a specific run has the following format:

    ```text theme={null}
    https://wandb.ai/<team-name>/<project-name>/runs/<run-id>
    ```

    Replace values enclosed in angle brackets (`< >`) with the actual values of the team name, project name, and [run ID](/models/runs/run-identifiers#run-id).

    Explore the run's properties by navigating through the tabs: [Overview](/models/runs/view-logged-runs#overview), [Logs](/models/runs/view-logged-runs#logs), [Files](/models/runs/view-logged-runs#files), [Code](/models/runs/view-logged-runs#code), and [Artifacts](/models/runs/view-logged-runs#artifacts).
  </Tab>

  <Tab title="LEET">
    To view a run locally in your terminal using the `wandb beta leet` terminal UI:

    1. If you started the run locally from a script, navigate to the directory where you ran your code. It contains a `wandb/` directory with a subdirectory per run and a `latest-run/` symbolic link. Each run directory contains a transaction log named in the format `run-<run-ID>.wandb`.

       If you did not start the run locally but downloaded a `.wandb` transaction log file instead, make a note of its location.
    2. Start `wandb beta leet` using one of these commands:

       ```bash theme={null}
       # View the latest run, stored in ./wandb/latest-run/
       wandb beta leet

       # Specify a run directory
       wandb beta leet ./wandb/run-20250813_124246-n67z9ude

       # Specify a .wandb file
       wandb beta leet ./wandb/run-20250813_124246-n67z9ude/run-n67z9ude.wandb
       ```

    LEET displays a three-panel interface:

    * **Left panel**: Run overview with environment variables, configuration, and summary statistics
    * **Center panel**: Metrics grid with Braille-style line charts showing your logged metrics
    * **Right panel**: System metrics including GPU/CPU/RAM utilization

    Get started with these keyboard shortcuts:

    * `h` or `?` - View all keyboard shortcuts
    * `/` - Filter metrics by pattern
    * `[` / `]` - Toggle left/right panels
    * `n` / `N` - Navigate between metric pages
    * `q` / `CMD+C` - Quit

    See the [`wandb beta leet`](/models/ref/cli/wandb-beta/wandb-beta-leet) reference for more details.
  </Tab>
</Tabs>

## Overview

Use the **Overview** tab to learn about specific run information in a project, such as:

* **References**: Dictionary keys from your experiment's [config](/models/track/config) that you want to show prominently in the W\&B App. See [Highlight config values](/models/track/config#highlight-config-values) for more details.
* **Notes**: Any notes you added to the run. You can add notes to a run with the W\&B App or programmatically with the Python SDK.
* **Tags**: A list of strings. Tags are useful for organizing related runs together or applying temporary labels like `baseline` or `production`. You can add tags to a run with the W\&B App or programmatically with the Python SDK.
* **Author**: The W\&B entity that creates the run.
* **Command**: The command that initializes the run.
* **Description**: A description of the run that you provided. This field is empty if you do not specify a description when you create the run. You can add a description to a run with the W\&B App or programmatically with the Python SDK.
* **Tracked Hours**: The amount of time the run is actively computing or logging data, excluding any pauses or waiting periods. This metric helps you understand the actual computational time spent on your run. You are not billed for tracked hours, which are unlimited for all plans.
* **Runtime**: Measures the total time from the start to the end of the run. It's the wall-clock time for the run, including any time where the run is paused or waiting for resources. This metric provides the complete elapsed time for your run.
* **Git repository**: The git repository associated with the run. You must [enable git](/platform/app/settings-page/user-settings/#personal-github-integration) to view this field.
* **Host name**: Where W\&B computes the run. W\&B displays the name of your machine if you initialize the run locally on your machine.
* **Name**: The name of the run.
* **OS**: Operating system that initializes the run.
* **Python executable**: The command that starts the run.
* **Python version**: Specifies the Python version that creates the run.
* **Run path**: Identifies the unique run identifier in the form `entity/project/run-ID`.
* **Start time**: The timestamp when you initialize the run.
* **State**: The [state of the run](/models/runs/run-states).
* **System hardware**: The hardware W\&B uses to compute the run.
* **W\&B CLI version**: The W\&B CLI version installed on the machine that hosted the run command.
* **Git state**: The most recent git commit SHA of a repository or working directory where the run is initialized. This field is empty if you do not enable Git when you create the run or if the git information is not available.

W\&B stores the following information below the overview section:

* **Artifact Outputs**: Artifact outputs produced by the run.
* **Config**: List of config parameters saved with [`wandb.Run.config`](/models/track/config/).
* **Summary**: List of summary parameters saved with [`wandb.Run.log()`](/models/track/log/). By default, W\&B sets this value to the last value logged.

You can use the search box aboce the Config and Summary sections to filter for specific parameters. For example, if you enter `acc` in the search box, W\&B filters for all parameters with `acc` in their name such as `accuracy` and `val_acc`.

<Note>
  W\&B does not search for nested config or summary parameters. For example, if you log a nested config parameter with `wandb.Run.config.update({"model": {"learning_rate": 0.01}})`, W\&B does not return the `learning_rate` parameter if you search for `learning_rate` in the search box. You can only find the `learning_rate` parameter by searching for `model`.
</Note>

View an example project overview [here](https://wandb.ai/stacey/deep-drive/overview).

## Logs

The **Logs** tab shows output printed on the command line such as the standard output (`stdout`) and standard error (`stderr`).

<Frame>
  <img alt="Run logs tab" />
</Frame>

Click the **Download** button in the upper right hand corner to download the log file.

View an example logs tab [here](https://app.wandb.ai/stacey/deep-drive/runs/pr0os44x/logs).

## Files

Use the **Files** tab to view files associated with a specific run such as model checkpoints, validation set examples, and more

<Frame>
  <img alt="Run files tab" />
</Frame>

View an example files tab [here](https://app.wandb.ai/stacey/deep-drive/runs/pr0os44x/files/media/images).

## Code

The **Code** tab displays the code files associated with a specific run. This includes the main script that was executed as well as any additional code files that were part of the run's environment.

## Artifacts

The **Artifacts** tab lists the input and output [artifacts](/models/artifacts/) for the specified run.

<Frame>
  <img alt="Run artifacts tab" />
</Frame>


# Sweeps overview
Source: https://docs.wandb.ai/models/sweeps

Hyperparameter search and model optimization with W&B Sweeps

<CardGroup>
  <ColabLink />

  <TryProductLink />
</CardGroup>

Use W\&B Sweeps to automate hyperparameter search and visualize rich, interactive experiment tracking. Pick from popular search methods such as Bayesian, grid search, and random to search the hyperparameter space. Scale and parallelize sweep across one or more machines.

<Frame>
  <img alt="Hyperparameter tuning insights" />
</Frame>

## How it works

Create a sweep with two [W\&B CLI](/models/ref/cli/) commands:

1. Initialize a sweep.

```bash theme={null}
wandb sweep --project <project-name> <path-to-config file>
```

2. Start the sweep agent.

```bash theme={null}
wandb agent <sweep-ID>
```

<Note>
  The preceding code snippet, and the colab linked on this page, show how to initialize and create a sweep with the W\&B CLI. See the [Sweeps walkthrough](/models/sweeps/walkthrough/) to use the Python SDK to configure, initialize, and run a sweep.
</Note>

## How to get started

Depending on your use case, explore the following resources to get started with W\&B Sweeps:

* Read through the [sweeps walkthrough](/models/sweeps/walkthrough/) for a step-by-step outline of the W\&B Python SDK commands to use to define a sweep configuration, initialize a sweep, and start a sweep.
* Explore this chapter to learn how to:
  * [Add W\&B to your code](/models/sweeps/add-w-and-b-to-your-code/)
  * [Define sweep configuration](/models/sweeps/define-sweep-configuration/)
  * [Initialize sweeps](/models/sweeps/initialize-sweeps/)
  * [Start sweep agents](/models/sweeps/start-sweep-agents/)
  * [Visualize sweep results](/models/sweeps/visualize-sweep-results/)
* Explore a [curated list of Sweep experiments](/models/sweeps/useful-resources/) that explore hyperparameter optimization with W\&B Sweeps. Results are stored in W\&B Reports.

For a step-by-step video, see: [Tune Hyperparameters Easily with W\&B Sweeps](https://www.youtube.com/watch?v=9zrmUIlScdY\&ab_channel=Weights%26Biases).

### Notebook examples

The following notebook examples explore how to use W\&B Sweeps for hyperparameter optimization across a variety of frameworks and use cases:

* [Hyperparameter optimization with Sweeps](https://colab.research.google.com/github/wandb/examples/blob/master/colabs/tensorflow/Hyperparameter_Optimization_in_TensorFlow_using_W\&B_Sweeps.ipynb)
* [Using XGBoost with W\&B Sweeps](https://colab.research.google.com/github/wandb/examples/blob/master/colabs/boosting/Using_W\&B_Sweeps_with_XGBoost.ipynb)


# Add W&B (wandb) to your code
Source: https://docs.wandb.ai/models/sweeps/add-w-and-b-to-your-code

Add W&B to your Python code script or Jupyter Notebook.

This guide provides recommendations on how to integrate W\&B into your Python training script or notebook for hyperparameter search optimization.

## Original training script

Suppose you have a Python script that trains a model (see below). Your goal is to find the hyperparameters that maxmimizes the validation accuracy(`val_acc`).

In your Python script, you define two functions: `train_one_epoch` and `evaluate_one_epoch`. The `train_one_epoch` function simulates training for one epoch and returns the training accuracy and loss. The `evaluate_one_epoch` function simulates evaluating the model on the validation data set and returns the validation accuracy and loss.

You define a configuration dictionary (`config`) that contains hyperparameter values such as the learning rate (`lr`), batch size (`batch_size`), and number of epochs (`epochs`). The values in the configuration dictionary control the training process.

Next you define a function called `main` that mimics a typical training loop. For each epoch, the accuracy and loss is computed on the training and validation data sets.

<Note>
  This code is a mock training script. It does not train a model, but simulates the training process by generating random accuracy and loss values. The purpose of this code is to demonstrate how to integrate W\&B into your training script.
</Note>

```python theme={null}
import random
import numpy as np

def train_one_epoch(epoch, lr, batch_size):
    acc = 0.25 + ((epoch / 30) + (random.random() / 10))
    loss = 0.2 + (1 - ((epoch - 1) / 10 + random.random() / 5))
    return acc, loss

def evaluate_one_epoch(epoch):
    acc = 0.1 + ((epoch / 20) + (random.random() / 10))
    loss = 0.25 + (1 - ((epoch - 1) / 10 + random.random() / 6))
    return acc, loss

# config variable with hyperparameter values
config = {"lr": 0.0001, "batch_size": 16, "epochs": 5}

def main():
    lr = config["lr"]
    batch_size = config["batch_size"]
    epochs = config["epochs"]

    for epoch in np.arange(1, epochs):
        train_acc, train_loss = train_one_epoch(epoch, lr, batch_size)
        val_acc, val_loss = evaluate_one_epoch(epoch)

        print("epoch: ", epoch)
        print("training accuracy:", train_acc, "training loss:", train_loss)
        print("validation accuracy:", val_acc, "validation loss:", val_loss)

if __name__ == "__main__":
    main()
```

In the next section, you will add W\&B to your Python script to track hyperparameters and metrics during training. You want to use W\&B to find the best hyperparameters that maximize the validation accuracy (`val_acc`).

## Add W\&B to your training script

Update you training script to include W\&B. How you integrate W\&B to your Python script or notebook depends on how you manage sweeps.

To use the W\&B Python SDK to start, stop, and manage sweeps, follow the instructions in the **Python script or notebook** tab. To use the W\&B CLI  instead, follow the instructions in the **CLI** tab.

<Tabs>
  <Tab title="CLI">
    Create a YAML configuration file with your sweep configuration. The
    configuration file contains the hyperparameters you want the sweep to explore. In
    the following example, the batch size (`batch_size`), epochs (`epochs`), and
    the learning rate (`lr`) hyperparameters are varied during each sweep.

    ```yaml theme={null}
    # config.yaml
    program: train.py
    method: random
    name: sweep
    metric:
      goal: maximize
      name: val_acc
    parameters:
      batch_size:
        values: [16, 32, 64]
      lr:
        min: 0.0001
        max: 0.1
      epochs:
        values: [5, 10, 15]
    ```

    For more information on how to create a W\&B Sweep configuration, see [Define sweep configuration](/models/sweeps/define-sweep-configuration/).

    You must provide the name of your Python script for the `program` key
    in your YAML file.

    Next, add the following to the code example:

    1. Import the W\&B Python SDK (`wandb`) and PyYAML (`yaml`). PyYAML is used to read in our YAML configuration file.
    2. Read in the configuration file.
    3. Use [`wandb.init()`](/models/ref/python/functions/init) to start a background process to sync and log data as a [W\&B Run](/models/ref/python/experiments/run). Pass the config object to the config parameter.
    4. Define hyperparameter values from `wandb.Run.config` instead of using hard coded values.
    5. Log the metric you want to optimize with [`wandb.Run.log()`](/models/ref/python/experiments/run.md/#method-runlog). You must log the metric defined in your configuration. Within the configuration dictionary (`sweep_configuration` in this example) you define the sweep to maximize the `val_acc` value.

    ```python theme={null}
    import wandb
    import yaml
    import random
    import numpy as np


    def train_one_epoch(epoch, lr, batch_size):
        acc = 0.25 + ((epoch / 30) + (random.random() / 10))
        loss = 0.2 + (1 - ((epoch - 1) / 10 + random.random() / 5))
        return acc, loss


    def evaluate_one_epoch(epoch):
        acc = 0.1 + ((epoch / 20) + (random.random() / 10))
        loss = 0.25 + (1 - ((epoch - 1) / 10 + random.random() / 6))
        return acc, loss


    def main():
        # Set up your default hyperparameters
        with open("./config.yaml") as file:
            config = yaml.load(file, Loader=yaml.FullLoader)

        with wandb.init(config=config) as run:
            for epoch in np.arange(1, run.config['epochs']):
                train_acc, train_loss = train_one_epoch(epoch, run.config['lr'], run.config['batch_size'])
                val_acc, val_loss = evaluate_one_epoch(epoch)
                run.log(
                    {
                        "epoch": epoch,
                        "train_acc": train_acc,
                        "train_loss": train_loss,
                        "val_acc": val_acc,
                        "val_loss": val_loss,
                    }
                )

    # Call the main function.
    main()
    ```

    In your CLI, set a maximum number of runs for the sweep
    agent to try. This is optional. This example we set the
    maximum number to 5.

    ```bash theme={null}
    NUM=5
    ```

    Next, initialize the sweep with the [`wandb sweep`](/models/ref/cli/wandb-sweep) command. Provide the name of the YAML file. Optionally provide the name of the project for the project flag (`--project`):

    ```bash theme={null}
    wandb sweep --project sweep-demo-cli config.yaml
    ```

    This returns a sweep ID. For more information on how to initialize sweeps, see
    [Initialize sweeps](./initialize-sweeps).

    Copy the sweep ID and replace `sweepID` in the following code snippet to start
    the sweep job with the [`wandb agent`](/models/ref/cli/wandb-agent)
    command:

    ```bash theme={null}
    wandb agent --count $NUM your-entity/sweep-demo-cli/sweepID
    ```

    For more information, see [Start sweep jobs](./start-sweep-agents).
  </Tab>

  <Tab title="Python script or notebook">
    Follow these steps to add W\&B to your Python script:

    1. Create a dictionary object where the key-value pairs define a [sweep configuration](/models/sweeps/define-sweep-configuration/). The sweep configuration defines the hyperparameters you want W\&B to explore on your behalf along with the metric you want to optimize. Continuing from the previous example, the batch size (`batch_size`), epochs (`epochs`), and the learning rate (`lr`) are the hyperparameters to vary during each sweep. You want to maximize the accuracy of the validation score so you set `"goal": "maximize"` and the name of the variable you want to optimize for, in this case `val_acc` (`"name": "val_acc"`).
    2. Pass the sweep configuration dictionary to [`wandb.sweep()`](/models/ref/python/functions/sweep). This initializes the sweep and returns a sweep ID (`sweep_id`). For more information, see [Initialize sweeps](./initialize-sweeps).
    3. At the top of your script, import the W\&B Python SDK (`wandb`).
    4. Within your `main` function, use [`wandb.init()`](/models/ref/python/functions/init) to generate a background process to sync and log data as a [W\&B Run](/models/ref/python/experiments/run). Pass the project name as a parameter to the `wandb.init()` method. If you do not pass a project name, W\&B uses the default project name.
    5. Fetch the hyperparameter values from the `wandb.Run.config` object. This allows you to use the hyperparameter values defined in the sweep configuration dictionary instead of hard coded values.
    6. Log the metric you are optimizing for to W\&B using [`wandb.Run.log()`](/models/ref/python/experiments/run.md/#method-runlog). You must log the metric defined in your configuration. For example, if you define the metric to optimize as `val_acc`, you must log `val_acc`. If you do not log the metric, W\&B does not know what to optimize for. Within the configuration dictionary (`sweep_configuration` in this example), you define the sweep to maximize the `val_acc` value.
    7. Start the sweep with [`wandb.agent()`](/models/ref/python/functions/agent). Provide the sweep ID and the name of the function the sweep will execute (`function=main`), and specify the maximum number of runs to try to four (`count=4`).

    Putting this all together, your script might look similar to the following:

    ```python theme={null}
    import wandb # Import the W&B Python SDK
    import numpy as np
    import random
    import argparse

    def train_one_epoch(epoch, lr, batch_size):
        acc = 0.25 + ((epoch / 30) + (random.random() / 10))
        loss = 0.2 + (1 - ((epoch - 1) / 10 + random.random() / 5))
        return acc, loss

    def evaluate_one_epoch(epoch):
        acc = 0.1 + ((epoch / 20) + (random.random() / 10))
        loss = 0.25 + (1 - ((epoch - 1) / 10 + random.random() / 6))
        return acc, loss

    def main(args=None):
        # When called by sweep agent, args will be None,
        # so we use the project from sweep config
        project = args.project if args else None
        
        with wandb.init(project=project) as run:
            # Fetches the hyperparameter values from `wandb.Run.config` object
            lr = run.config["lr"]
            batch_size = run.config["batch_size"]
            epochs = run.config["epochs"]

            # Execute the training loop and log the performance values to W&B
            for epoch in np.arange(1, epochs):
                train_acc, train_loss = train_one_epoch(epoch, lr, batch_size)
                val_acc, val_loss = evaluate_one_epoch(epoch)
                run.log(
                    {
                        "epoch": epoch,
                        "train_acc": train_acc,
                        "train_loss": train_loss,
                        "val_acc": val_acc, # Metric optimized
                        "val_loss": val_loss,
                    }
                )

    if __name__ == "__main__":
        parser = argparse.ArgumentParser()
        parser.add_argument("--project", type=str, default="sweep-example", help="W&B project name")
        args = parser.parse_args()

        # Define a sweep config dictionary
        sweep_configuration = {
            "method": "random",
            "name": "sweep",
            # Metric that you want to optimize
            # For example, if you want to maximize validation
            # accuracy set "goal": "maximize" and the name of the variable 
            # you want to optimize for, in this case "val_acc"
            "metric": {
                "goal": "maximize",
                "name": "val_acc"
                },
            "parameters": {
                "batch_size": {"values": [16, 32, 64]},
                "epochs": {"values": [5, 10, 15]},
                "lr": {"max": 0.1, "min": 0.0001},
            },
        }

        # Initialize the sweep by passing in the config dictionary
        sweep_id = wandb.sweep(sweep=sweep_configuration, project=args.project)

        # Start the sweep job
        wandb.agent(sweep_id, function=main, count=4)
    ```
  </Tab>
</Tabs>

<Note>
  **Logging metrics to W\&B in a sweep**

  You must log the metric you define and are optimizing for in both your sweep configuration and with `wandb.Run.log()`. For example, if you define the metric to optimize as `val_acc` within your sweep configuration, you must also log `val_acc` to W\&B. If you do not log the metric, W\&B does not know what to optimize for.

  ```python theme={null}
  with wandb.init() as run:
      val_loss, val_acc = train()
      run.log(
          {
              "val_loss": val_loss,
              "val_acc": val_acc
              }
          )
  ```

  The following is an incorrect example of logging the metric to W\&B. The metric that is optimized for in the sweep configuration is `val_acc`, but the code logs `val_acc` within a nested dictionary under the key `validation`. You must log the metric directly, not within a nested dictionary.

  ```python theme={null}
  with wandb.init() as run:
      val_loss, val_acc = train()
      run.log(
          {
              "validation": {
                  "val_loss": val_loss, 
                  "val_acc": val_acc
                  }
              }
          )
  ```
</Note>


# Overview
Source: https://docs.wandb.ai/models/sweeps/define-sweep-configuration

Learn how to create configuration files for sweeps.

A W\&B Sweep combines a strategy for exploring hyperparameter values with the code that evaluates them. The strategy can be as simple as trying every option or as complex as Bayesian Optimization and Hyperband ([BOHB](https://arxiv.org/abs/1807.01774)).

Define a sweep configuration either in a [Python dictionary](https://docs.python.org/3/tutorial/datastructures.html#dictionaries) or a [YAML](https://yaml.org/) file. How you define your sweep configuration depends on how you want to manage your sweep.

<Note>
  Define your sweep configuration in a YAML file if you want to initialize a sweep and start a sweep agent from the command line. Define your sweep in a Python dictionary if you initialize a sweep and start a sweep entirely within a Python script or notebook.
</Note>

The following guide describes how to format your sweep configuration. See [Sweep configuration options](./sweep-config-keys) for a comprehensive list of top-level sweep configuration keys.

## Basic structure

Both sweep configuration format options (YAML and Python dictionary) use key-value pairs and nested structures.

Use top-level keys within your sweep configuration to define qualities of your sweep search such as the name of the sweep ([`name`](./sweep-config-keys) key), the parameters to search through ([`parameters`](./sweep-config-keys#parameters) key), the methodology to search the parameter space ([`method`](./sweep-config-keys#method) key), and more.

For example, the following code snippets show the same sweep configuration defined within a YAML file and within a Python dictionary. Within the sweep configuration there are five top level keys specified: `program`, `name`, `method`, `metric` and `parameters`.

<Tabs>
  <Tab title="CLI">
    Define a sweep configuration in a YAML file if you want to manage sweeps interactively from the command line (CLI)

    ```yaml title="config.yaml" theme={null}
    program: train.py
    name: sweepdemo
    method: bayes
    metric:
      goal: minimize
      name: validation_loss
    parameters:
      learning_rate:
        min: 0.0001
        max: 0.1
      batch_size:
        values: [16, 32, 64]
      epochs:
        values: [5, 10, 15]
      optimizer:
        values: ["adam", "sgd"]
    ```
  </Tab>

  <Tab title="Python script or notebook">
    Define a sweep in a Python dictionary data structure if you define training algorithm in a Python script or notebook.

    The following code snippet stores a sweep configuration in a variable named `sweep_configuration`:

    ```python title="train.py" theme={null}
    sweep_configuration = {
        "name": "sweepdemo",
        "method": "bayes",
        "metric": {"goal": "minimize", "name": "validation_loss"},
        "parameters": {
            "learning_rate": {"min": 0.0001, "max": 0.1},
            "batch_size": {"values": [16, 32, 64]},
            "epochs": {"values": [5, 10, 15]},
            "optimizer": {"values": ["adam", "sgd"]},
        },
    }
    ```
  </Tab>
</Tabs>

Within the top level `parameters` key, the following keys are nested: `learning_rate`, `batch_size`, `epoch`, and `optimizer`. For each of the nested keys you specify, you can provide one or more values, a distribution, a probability, and more. For more information, see the [parameters](./sweep-config-keys#parameters) section in [Sweep configuration options](./sweep-config-keys).

## Double nested parameters

Sweep configurations support nested parameters. To define a nested parameter, include an additional `parameters` key under the top-level parameter name.

The following example shows a sweep configuration with three nested parameters: `nested_category_1`, `nested_category_2`, and `nested_category_3`. Each nested parameter includes two additional parameters: `momentum` and `weight_decay`.

<Note>
  `nested_category_1`, `nested_category_2`, and `nested_category_3` are placeholders. Replace them with names that fit your use case.
</Note>

The following code snippets show how to define nested parameters in both a YAML file and a Python dictionary.

<Tabs>
  <Tab title="CLI">
    ```yaml theme={null}
    program: sweep_nest.py
    name: nested_sweep
    method: random
    metric:
      name: loss
      goal: minimize
    parameters:
      optimizer:
        values: ['adam', 'sgd']
      fc_layer_size:
        values: [128, 256, 512]
      dropout:
        values: [0.3, 0.4, 0.5]
      epochs:
        value: 1
      learning_rate:
        distribution: uniform
        min: 0
        max: 0.1
      batch_size:
        distribution: q_log_uniform_values
        q: 8
        min: 32
        max: 256
      nested_category_1:
        parameters:
          momentum:
            distribution: uniform
            min: 0.0
            max: 0.9
          weight_decay:
            values: [0.0001, 0.0005, 0.001]
      nested_category_2:
        parameters:
          momentum:
            distribution: uniform
            min: 0.0
            max: 0.9
          weight_decay:
            values: [0.1, 0.2, 0.3]
      nested_category_3:
        parameters:
          momentum:
            distribution: uniform
            min: 0.5
            max: 0.7
          weight_decay:
            values: [0.2, 0.3, 0.4]
    ```
  </Tab>

  <Tab title="Python script or notebook">
    ```python theme={null}
    {
      "program": "sweep_nest.py",
      "name": "nested_sweep",
      "method": "random",
      "metric": {
        "name": "loss",
        "goal": "minimize"
      },
      "parameters": {
        "optimizer": {
          "values": ["adam", "sgd"]
        },
        "fc_layer_size": {
          "values": [128, 256, 512]
        },
        "dropout": {
          "values": [0.3, 0.4, 0.5]
        },
        "epochs": {
          "value": 1
        },
        "learning_rate": {
          "distribution": "uniform",
          "min": 0,
          "max": 0.1
        },
        "batch_size": {
          "distribution": "q_log_uniform_values",
          "q": 8,
          "min": 32,
          "max": 256
        },
        "nested_category_1": {
          "parameters": {
            "momentum": {
              "distribution": "uniform",
              "min": 0.0,
              "max": 0.9
            },
            "weight_decay": {
              "values": [0.0001, 0.0005, 0.001]
            }
          }
        },
        "nested_category_2": {
          "parameters": {
            "momentum": {
              "distribution": "uniform",
              "min": 0.0,
              "max": 0.9
            },
            "weight_decay": {
              "values": [0.1, 0.2, 0.3]
            }
          }
        },
        "nested_category_3": {
          "parameters": {
            "momentum": {
              "distribution": "uniform",
              "min": 0.5,
              "max": 0.7
            },
            "weight_decay": {
              "values": [0.2, 0.3, 0.4]
            }
          }
        }
      }
    }
    ```
  </Tab>
</Tabs>

<Warning>
  Nested parameters defined in sweep configuration overwrite keys specified in a W\&B run configuration.

  As an example, suppose you have `train.py` script that initializes a run with a nested default:

  ```python theme={null}
  def main():
      with  wandb.init(config={"nested_param": {"manual_key": 1}}) as run:
          # Your training code here
  ```

  Your sweep configuration defines nested parameters under a top-level `"parameters"` key:

  ```python theme={null}
  sweep_configuration = {
      "method": "grid",
      "metric": {"name": "score", "goal": "minimize"},
      "parameters": {
          "top_level_param": {"value": 0},
          "nested_param": {
              "parameters": {
                  "learning_rate": {"value": 0.01},
                  "double_nested_param": {
                      "parameters": {"x": {"value": 0.9}, "y": {"value": 0.8}}
                  },
              }
          },
      },
  }

  sweep_id = wandb.sweep(sweep=sweep_configuration, project="<project>")
  wandb.agent(sweep_id, function=main, count=4)
  ```

  During a sweep run, `run.config["nested_param"]` reflects the subtree defined by the
  sweep (`learning_rate`, `double_nested_param`) config and does not include `manual_key` defined
  in `wandb.init(config=...)`.
</Warning>

## Sweep configuration template

The following template shows how you can configure parameters and specify search constraints. Replace `hyperparameter_name` with the name of your hyperparameter and any values enclosed in `<>`.

```yaml title="config.yaml" theme={null}
program: <insert>
method: <insert>
parameter:
  hyperparameter_name0:
    value: 0  
  hyperparameter_name1: 
    values: [0, 0, 0]
  hyperparameter_name: 
    distribution: <insert>
    value: <insert>
  hyperparameter_name2:  
    distribution: <insert>
    min: <insert>
    max: <insert>
    q: <insert>
  hyperparameter_name3: 
    distribution: <insert>
    values:
      - <list_of_values>
      - <list_of_values>
      - <list_of_values>
early_terminate:
  type: hyperband
  s: 0
  eta: 0
  max_iter: 0
command:
- ${Command macro}
- ${Command macro}
- ${Command macro}
- ${Command macro}      
```

To express a numeric value using scientific notation, add the YAML `!!float` operator, which casts the value to a floating point number. For example, `min: !!float 1e-5`. See [Command example](#command-example).

## Sweep configuration examples

<Tabs>
  <Tab title="CLI">
    ```yaml title="config.yaml"  theme={null}
    program: train.py
    method: random
    metric:
      goal: minimize
      name: loss
    parameters:
      batch_size:
        distribution: q_log_uniform_values
        max: 256 
        min: 32
        q: 8
      dropout: 
        values: [0.3, 0.4, 0.5]
      epochs:
        value: 1
      fc_layer_size: 
        values: [128, 256, 512]
      learning_rate:
        distribution: uniform
        max: 0.1
        min: 0
      optimizer:
        values: ["adam", "sgd"]
    ```
  </Tab>

  <Tab title="Python script or notebook">
    ```python title="train.py"  theme={null}
    sweep_config = {
        "method": "random",
        "metric": {"goal": "minimize", "name": "loss"},
        "parameters": {
            "batch_size": {
                "distribution": "q_log_uniform_values",
                "max": 256,
                "min": 32,
                "q": 8,
            },
            "dropout": {"values": [0.3, 0.4, 0.5]},
            "epochs": {"value": 1},
            "fc_layer_size": {"values": [128, 256, 512]},
            "learning_rate": {"distribution": "uniform", "max": 0.1, "min": 0},
            "optimizer": {"values": ["adam", "sgd"]},
        },
    }
    ```
  </Tab>
</Tabs>

### Bayes hyperband example

```yaml theme={null}
program: train.py
method: bayes
metric:
  goal: minimize
  name: val_loss
parameters:
  dropout:
    values: [0.15, 0.2, 0.25, 0.3, 0.4]
  hidden_layer_size:
    values: [96, 128, 148]
  layer_1_size:
    values: [10, 12, 14, 16, 18, 20]
  layer_2_size:
    values: [24, 28, 32, 36, 40, 44]
  learn_rate:
    values: [0.001, 0.01, 0.003]
  decay:
    values: [1e-5, 1e-6, 1e-7]
  momentum:
    values: [0.8, 0.9, 0.95]
  epochs:
    value: 27
early_terminate:
  type: hyperband
  s: 2
  eta: 3
  max_iter: 27
```

The following tabs show how to specify either a minimum or maximum number of iterations for `early_terminate`:

<Tabs>
  <Tab title="Maximum number of iterations">
    The brackets for this example are: `[3, 3*eta, 3*eta*eta, 3*eta*eta*eta]`, which equals `[3, 9, 27, 81]`.

    ```yaml theme={null}
    early_terminate:
      type: hyperband
      min_iter: 3
    ```
  </Tab>

  <Tab title="Minimum number of iterations">
    The brackets for this example are `[27/eta, 27/eta/eta]`, which equals `[9, 3]`.

    ```yaml theme={null}
    early_terminate:
      type: hyperband
      max_iter: 27
      s: 2
    ```
  </Tab>
</Tabs>

### Macro and custom command arguments example

For more complex command line arguments, you can use macros to pass environment variables, the Python interpreter, and additional arguments. [W\&B supports pre defined macros](./sweep-config-keys#command-macros) and custom command line arguments that you can specify in your sweep configuration.

For example, the following sweep configuration (`sweep.yaml`) defines a command that runs a Python script (`run.py`) with the `${env}`, `${interpreter}`, and `${program}` macros replaced with the appropriate values when the sweep runs.

The `--batch_size=${batch_size}`, `--test=True`, and `--optimizer=${optimizer}` arguments use custom macros to pass the values of the `batch_size`, `test`, and `optimizer` parameters defined in the sweep configuration.

```yaml title="sweep.yaml" theme={null}
program: run.py
method: random
metric:
  name: validation_loss
parameters:
  learning_rate:
    min: 0.0001
    max: 0.1
command:
  - ${env}
  - ${interpreter}
  - ${program}
  - "--batch_size=${batch_size}"
  - "--optimizer=${optimizer}"
  - "--test=True"
```

The associated Python script (`run.py`) can then parse these command line arguments using the `argparse` module.

```python title="run.py" theme={null}
# run.py  
import wandb
import argparse

parser = argparse.ArgumentParser()
parser.add_argument('--batch_size', type=int)
parser.add_argument('--optimizer', type=str, choices=['adam', 'sgd'], required=True)
parser.add_argument('--test', type=str2bool, default=False)
args = parser.parse_args()

# Initialize a W&B Run
with wandb.init('test-project') as run:
    run.log({'validation_loss':1})
```

See the [Command macros](./sweep-config-keys#command-macros) section in [Sweep configuration options](./sweep-config-keys) for a list of pre-defined macros you can use in your sweep configuration.

#### Boolean arguments

The `argparse` module does not support boolean arguments by default. To define a boolean argument, you can use the [`action`](https://docs.python.org/3/library/argparse.html#action) parameter or use a custom function to convert the string representation of the boolean value to a boolean type.

As an example, you can use the following code snippet to define a boolean argument. Pass `store_true` or `store_false` as an argument to `ArgumentParser`.

```python theme={null}
import wandb
import argparse

parser = argparse.ArgumentParser()
parser.add_argument('--test', action='store_true')
args = parser.parse_args()

args.test  # This will be True if --test is passed, otherwise False
```

You can also define a custom function to convert the string representation of the boolean value to a boolean type. For example, the following code snippet defines the `str2bool` function, which converts a string to a boolean value.

```python theme={null}
def str2bool(v: str) -> bool:
  """Convert a string to a boolean. This is required because
  argparse does not support boolean arguments by default.
  """
  if isinstance(v, bool):
      return v
  return v.lower() in ('yes', 'true', 't', '1')
```


# Tutorial: Define, initialize, and run a sweep
Source: https://docs.wandb.ai/models/sweeps/walkthrough

Sweeps quickstart shows how to define, initialize, and run a sweep. There are four main steps

This page shows how to define, initialize, and run a sweep. There are four main steps:

1. [Set up your training code](#set-up-your-training-code)
2. [Define the search space with a sweep configuration](#define-the-search-space-with-a-sweep-configuration)
3. [Initialize the sweep](#initialize-the-sweep)
4. [Start the sweep agent](#start-the-sweep)

Copy and paste the following code into a Jupyter Notebook or Python script:

```python theme={null}
# Import the W&B Python Library and log into W&B
import wandb

# 1: Define objective/training function
def objective(config):
    score = config.x**3 + config.y
    return score

def main():
    with wandb.init(project="my-first-sweep") as run:
        score = objective(run.config)
        run.log({"score": score})

# 2: Define the search space
sweep_configuration = {
    "method": "random",
    "metric": {"goal": "minimize", "name": "score"},
    "parameters": {
        "x": {"max": 0.1, "min": 0.01},
        "y": {"values": [1, 3, 7]},
    },
}

# 3: Start the sweep
sweep_id = wandb.sweep(sweep=sweep_configuration, project="my-first-sweep")

wandb.agent(sweep_id, function=main, count=10)
```

The following sections break down and explains each step in the code sample.

## Set up your training code

Define a training function that takes in hyperparameter values from `wandb.Run.config` and uses them to train a model and return metrics.

Optionally provide the name of the project where you want the output of the W\&B Run to be stored (project parameter in [`wandb.init()`](/models/ref/python/functions/init)). If the project is not specified, the run is put in an "Uncategorized" project.

<Note>
  Both the sweep and the run must be in the same project. Therefore, the name you provide when you initialize W\&B must match the name of the project you provide when you initialize a sweep.
</Note>

```python theme={null}
# 1: Define objective/training function
def objective(config):
    score = config.x**3 + config.y
    return score


def main():
    with wandb.init(project="my-first-sweep") as run:
        score = objective(run.config)
        run.log({"score": score})
```

## Define the search space with a sweep configuration

Specify the hyperparameters to sweep in a dictionary. For configuration options, see [Define sweep configuration](/models/sweeps/define-sweep-configuration/).

The following example demonstrates a sweep configuration that uses a random search (`'method':'random'`). The sweep will randomly select a random set of values listed in the configuration for the batch size, epoch, and the learning rate.

W\&B minimizes the metric specified in the `metric` key when `"goal": "minimize"` is associated with it. In this case, W\&B will optimize for minimizing the metric `score` (`"name": "score"`).

```python theme={null}
# 2: Define the search space
sweep_configuration = {
    "method": "random",
    "metric": {"goal": "minimize", "name": "score"},
    "parameters": {
        "x": {"max": 0.1, "min": 0.01},
        "y": {"values": [1, 3, 7]},
    },
}
```

## Initialize the Sweep

W\&B uses a *Sweep Controller* to manage sweeps on the cloud (standard), locally (local) across one or more machines. For more information about Sweep Controllers, see [Search and stop algorithms locally](./local-controller).

A sweep identification number is returned when you initialize a sweep:

```python theme={null}
sweep_id = wandb.sweep(sweep=sweep_configuration, project="my-first-sweep")
```

For more information about initializing sweeps, see [Initialize sweeps](./initialize-sweeps).

## Start the Sweep

Use the [`wandb.agent()`](/models/ref/python/functions/agent) API call to start a sweep.

```python theme={null}
wandb.agent(sweep_id, function=main, count=10)
```

<Warning>
  **Multiprocessing**

  You must wrap your `wandb.agent()` and `wandb.sweep()` calls with `if __name__ == '__main__':` if you use Python standard library's `multiprocessing` or PyTorch's `pytorch.multiprocessing` package. For example:

  ```python theme={null}
  if __name__ == '__main__':
      wandb.agent(sweep_id="<sweep_id>", function="<function>", count="<count>")
  ```

  Wrapping your code with this convention ensures that it is only executed when the script is run directly, and not when it is imported as a module in a worker process.

  See [Python standard library `multiprocessing`](https://docs.python.org/3/library/multiprocessing.html#the-spawn-and-forkserver-start-methods) or [PyTorch `multiprocessing`](https://docs.pytorch.org/docs/stable/notes/multiprocessing.html#asynchronous-multiprocess-training-e-g-hogwild) for more information about multiprocessing. See [https://realpython.com/if-name-main-python/](https://realpython.com/if-name-main-python/) for information about the `if __name__ == '__main__':` convention.
</Warning>

## Visualize results (optional)

Open your project to see your live results in the W\&B App dashboard. With just a few clicks, construct rich, interactive charts like [parallel coordinates plots](/models/app/features/panels/parallel-coordinates/),[ parameter importance analyzes](/models/app/features/panels/parameter-importance/), and [additional chart types](/models/app/features/panels/).

<Frame>
  <img alt="Sweeps Dashboard example" />
</Frame>

For more information about how to visualize results, see [Visualize sweep results](./visualize-sweep-results). For an example dashboard, see this sample [Sweeps Project](https://wandb.ai/anmolmann/pytorch-cnn-fashion/sweeps/pmqye6u3).

## Stop the agent (optional)

In the terminal, press `Ctrl+C` to stop the current run. Press it again to terminate the agent.


# Experiments overview
Source: https://docs.wandb.ai/models/track

Track machine learning experiments with W&B to log metrics, hyperparameters, system metrics, and model artifacts.

<CardGroup>
  <ColabLink />

  <TryProductLink />
</CardGroup>

Track machine learning experiments with a few lines of code. You can then review the results in an [interactive dashboard](/models/track/workspaces/) or export your data to Python for programmatic access using our [Public API](/models/ref/python/public-api/).

Utilize W\&B Integrations if you use popular frameworks such as [Keras](/models/integrations/keras). See [W\&B Integrations](/models/integrations) for a full list of integrations and information on how to add W\&B to your code.

<Frame>
  <img alt="Experiments dashboard" />
</Frame>

The image above shows an example dashboard where you can view and compare metrics across multiple [runs](/models/runs/).

## How it works

Track a machine learning experiment with a few lines of code:

1. Create a [W\&B Run](/models/runs/).
2. Store a dictionary of hyperparameters, such as learning rate or model type, into your configuration ([`wandb.Run.config`](/models/track/config/)).
3. Log metrics ([`wandb.Run.log()`](/models/track/log/)) over time in a training loop, such as accuracy and loss.
4. Save outputs of a run, like the model weights or a table of predictions.

The following code demonstrates a common W\&B experiment tracking workflow:

```python theme={null}
# Start a run.
#
# When this block exits, it waits for logged data to finish uploading.
# If an exception is raised, the run is marked failed.
with wandb.init(entity="", project="my-project-name") as run:
  # Save mode inputs and hyperparameters.
  run.config.learning_rate = 0.01

  # Run your experiment code.
  for epoch in range(num_epochs):
    # Do some training...

    # Log metrics over time to visualize model performance.
    run.log({"loss": loss})

  # Upload model outputs as artifacts.
  run.log_artifact(model)
```

## Get started

Depending on your use case, explore the following resources to get started with W\&B Experiments:

* Read the [W\&B Quickstart](/models/quickstart/) for a step-by-step outline of the W\&B Python SDK commands you could use to create, track, and use a dataset artifact.
* Explore this chapter to learn how to:
  * Create an experiment
  * Configure experiments
  * Log data from experiments
  * View results from experiments
* Explore the [W\&B Python Library](/models/ref/python/) within the [W\&B API Reference Guide](/models/ref/python/).

## Best practices and tips

For best practices and tips for experiments and logging, see [Best Practices: Experiments and Logging](https://wandb.ai/wandb/pytorch-lightning-e2e/reports/W-B-Best-Practices-Guide--VmlldzozNTU1ODY1#w\&b-experiments-and-logging).


# Configure experiments
Source: https://docs.wandb.ai/models/track/config

Use a dictionary-like object to save your experiment configuration

<ColabLink />

Use the `config` property of a run to save your training configuration:

* hyperparameter
* input settings such as the dataset name or model type
* any other independent variables for your experiments.

The `wandb.Run.config` property makes it easy to analyze your experiments and reproduce your work in the future. You can group by configuration values in the W\&B App, compare the configurations of different W\&B runs, and evaluate how each training configuration affects the output. The `config` property is a dictionary-like object that can be composed from multiple dictionary-like objects.

<Note>
  To save output metrics or dependent variables like loss and accuracy, use `wandb.Run.log()` instead of `wandb.Run.config`.
</Note>

## Set up an experiment configuration

Configurations are typically defined in the beginning of a training script. Machine learning workflows may vary, however, so you are not required to define a configuration at the beginning of your training script.

Use dashes (`-`) or underscores (`_`) instead of periods (`.`) in your config variable names.

Use the dictionary access syntax `["key"]["value"]` instead of the attribute access syntax `config.key.value` if your script accesses `wandb.Run.config` keys below the root.

The following sections outline different common scenarios of how to define your experiments configuration.

### Set the configuration at initialization

Pass a dictionary at the beginning of your script when you call the `wandb.init()` API to generate a background process to sync and log data as a W\&B Run.

The following code snippet demonstrates how to define a Python dictionary with configuration values and how to pass that dictionary as an argument when you initialize a W\&B Run.

```python theme={null}
import wandb

# Define a config dictionary object
config = {
    "hidden_layer_sizes": [32, 64],
    "kernel_sizes": [3],
    "activation": "ReLU",
    "pool_sizes": [2],
    "dropout": 0.5,
    "num_classes": 10,
}

# Pass the config dictionary when you initialize W&B
with wandb.init(project="config_example", config=config) as run:
    ...
```

If you pass a nested dictionary as the `config`, W\&B flattens the names using dots.

Access the values from the dictionary similarly to how you access other dictionaries in Python:

```python theme={null}
# Access values with the key as the index value
hidden_layer_sizes = run.config["hidden_layer_sizes"]
kernel_sizes = run.config["kernel_sizes"]
activation = run.config["activation"]

# Python dictionary get() method
hidden_layer_sizes = run.config.get("hidden_layer_sizes")
kernel_sizes = run.config.get("kernel_sizes")
activation = run.config.get("activation")
```

<Note>
  Throughout the Developer Guide and examples we copy the configuration values into separate variables. This step is optional. It is done for readability.
</Note>

### Set the configuration with argparse

You can set your configuration with an argparse object. [argparse](https://docs.python.org/3/library/argparse.html), short for argument parser, is a standard library module in Python 3.2 and above that makes it easy to write scripts that take advantage of all the flexibility and power of command line arguments.

This is useful for tracking results from scripts that are launched from the command line.

The following Python script demonstrates how to define a parser object to define and set your experiment config. The functions `train_one_epoch` and `evaluate_one_epoch` are provided to simulate a training loop for the purpose of this demonstration:

```python theme={null}
# config_experiment.py
import argparse
import random

import numpy as np
import wandb


# Training and evaluation demo code
def train_one_epoch(epoch, lr, bs):
    acc = 0.25 + ((epoch / 30) + (random.random() / 10))
    loss = 0.2 + (1 - ((epoch - 1) / 10 + random.random() / 5))
    return acc, loss


def evaluate_one_epoch(epoch):
    acc = 0.1 + ((epoch / 20) + (random.random() / 10))
    loss = 0.25 + (1 - ((epoch - 1) / 10 + random.random() / 6))
    return acc, loss


def main(args):
    # Start a W&B Run
    with wandb.init(project="config_example", config=args) as run:
        # Access values from config dictionary and store them
        # into variables for readability
        lr = run.config["learning_rate"]
        bs = run.config["batch_size"]
        epochs = run.config["epochs"]

        # Simulate training and logging values to W&B
        for epoch in np.arange(1, epochs):
            train_acc, train_loss = train_one_epoch(epoch, lr, bs)
            val_acc, val_loss = evaluate_one_epoch(epoch)

            run.log(
                {
                    "epoch": epoch,
                    "train_acc": train_acc,
                    "train_loss": train_loss,
                    "val_acc": val_acc,
                    "val_loss": val_loss,
                }
            )


if __name__ == "__main__":
    parser = argparse.ArgumentParser(
        formatter_class=argparse.ArgumentDefaultsHelpFormatter
    )

    parser.add_argument("-b", "--batch_size", type=int, default=32, help="Batch size")
    parser.add_argument(
        "-e", "--epochs", type=int, default=50, help="Number of training epochs"
    )
    parser.add_argument(
        "-lr", "--learning_rate", type=int, default=0.001, help="Learning rate"
    )

    args = parser.parse_args()
    main(args)
```

### Set the configuration throughout your script

You can add more parameters to your config object throughout your script. The following code snippet demonstrates how to add new key-value pairs to your config object:

```python theme={null}
import wandb

# Define a config dictionary object
config = {
    "hidden_layer_sizes": [32, 64],
    "kernel_sizes": [3],
    "activation": "ReLU",
    "pool_sizes": [2],
    "dropout": 0.5,
    "num_classes": 10,
}

# Pass the config dictionary when you initialize W&B
with wandb.init(project="config_example", config=config) as run:
    # Update config after you initialize W&B
    run.config["dropout"] = 0.2
    run.config.epochs = 4
    run.config["batch_size"] = 32
```

You can update multiple values at a time:

```python theme={null}
run.config.update({"lr": 0.1, "channels": 16})
```

### Set the configuration after your run finishes

Use the [W\&B Public API](/models/ref/python/public-api/) to update a completed run's config.

You must provide the API with your entity, project name and the run's ID. You can find these details in the Run object or in the [W\&B App](/models/track/workspaces/):

```python theme={null}
with wandb.init() as run:
    ...

# Find the following values from the Run object if it was initiated from the
# current script or notebook, or you can copy them from the W&B App UI.
username = run.entity
project = run.project
run_id = run.id

# Note that api.run() returns a different type of object than wandb.init().
api = wandb.Api()
api_run = api.run(f"{username}/{project}/{run_id}")
api_run.config["bar"] = 32
api_run.update()
```

## Highlight config values

Pin config keys to the **References** section at the top of a run's overview page.

Use [`wandb.Run.pin_config_keys`](/models/ref/python/experiments/run#method-run-pin_config_keys) to pin one or more config keys with the Python SDK.

For example, if you use a Grafana dashboard to monitor training runs, add the dashboard URL to your config and pin the `grafana_url` key:

```python theme={null}
config = {
    "hidden_layer_sizes": [32, 64],
    "kernel_sizes": [3],
    "activation": "ReLU",
    "pool_sizes": [2],
    "dropout": 0.5,
    "num_classes": 10,
    "grafana_url": "[Grafana dashboard](https://my-grafana-instance.com/)"
}

with wandb.init(config=config) as run:
    # Add the "grafana_url" config key to the References section.
    run.pin_config_keys(["grafana_url"])
```

## `absl.FLAGS`

You can also pass in [`absl` flags](https://abseil.io/docs/python/guides/flags).

```python theme={null}
flags.DEFINE_string("model", None, "model to run")  # name, default, help

run.config.update(flags.FLAGS)  # adds absl flags to config
```

## File-Based Configs

If you place a file named `config-defaults.yaml` in the same directory as your run script, the run automatically picks up the key-value pairs defined in the file and passes them to `wandb.Run.config`.

The following code snippet shows a sample `config-defaults.yaml` YAML file:

```yaml theme={null}
batch_size:
  desc: Size of each mini-batch
  value: 32
```

You can override the default values automatically loaded from `config-defaults.yaml` by setting updated values in the `config` argument of `wandb.init()`. For example:

```python theme={null}
import wandb

# Override config-defaults.yaml by passing custom values
with wandb.init(config={"epochs": 200, "batch_size": 64}) as run:
    ...
```

To load a configuration file other than `config-defaults.yaml`, use the `--configs command-line` argument and specify the path to the file:

```bash theme={null}
python train.py --configs other-config.yaml
```

### Example use case for file-based configs

Suppose you have a YAML file with some metadata for the run, and then a dictionary of hyperparameters in your Python script. You can save both in the nested `config` object:

```python theme={null}
hyperparameter_defaults = dict(
    dropout=0.5,
    batch_size=100,
    learning_rate=0.001,
)

config_dictionary = dict(
    yaml=my_yaml_file,
    params=hyperparameter_defaults,
)

with wandb.init(config=config_dictionary) as run:
    ...
```

## View config values in the W\&B App

View your config values in the **Config** section of a run's Overview tab. The config values are also available in the **References** section if you pin them.

1. Navigate to your project in the W\&B App.
2. Click the run you want to view the config values for.
3. Select the **Overview** tab.
4. Scroll to the **Config** section.
5. (Optional) Click on **View raw data** to view config values in JSON format.

The raw JSON format is useful when you write expressions to create or transform line plots from config values, logged metrics, or summary values. See [Expressions](/models/app/features/panels/line-plot/reference#expressions) for more details.

## TensorFlow v1 flags

You can pass TensorFlow flags into the `wandb.Run.config` object directly.

```python theme={null}
with wandb.init() as run:
    run.config.epochs = 4

    flags = tf.app.flags
    flags.DEFINE_string("data_dir", "/tmp/data")
    flags.DEFINE_integer("batch_size", 128, "Batch size.")
    run.config.update(flags.FLAGS)  # add tensorflow flags as config
```


# Create an experiment
Source: https://docs.wandb.ai/models/track/create-an-experiment

Create a W&B Experiment using the Python SDK to track run initialization, hyperparameters, and metric logging.

Use the W\&B Python SDK to track machine learning experiments. You can then review the results in an interactive dashboard or export your data to Python for programmatic access with the [W\&B Public API](/models/ref/python/public-api/).

This guide describes how to use W\&B building blocks to create a W\&B Experiment.

## How to create a W\&B Experiment

Create a W\&B Experiment in four steps:

1. [Initialize a W\&B Run](#initialize-a-wb-run)
2. [Capture a dictionary of hyperparameters](#capture-a-dictionary-of-hyperparameters)
3. [Log metrics inside your training loop](#log-metrics-inside-your-training-loop)
4. [Log an artifact to W\&B](#log-an-artifact-to-wb)

### Initialize a W\&B run

Use [`wandb.init()`](/models/ref/python/functions/init) to create a W\&B Run.

The following snippet creates a run in a W\&B project named `“cat-classification”` with the description `“My first experiment”` to help identify this run. Tags `“baseline”` and `“paper1”` are included to remind us that this run is a baseline experiment intended for a future paper publication.

```python theme={null}
import wandb

with wandb.init(
    project="cat-classification",
    notes="My first experiment",
    tags=["baseline", "paper1"],
) as run:
    ...
```

`wandb.init()` returns a [Run](/models/ref/python/experiments/run) object.

<Note>
  Note: Runs are added to pre-existing projects if that project already exists when you call `wandb.init()`. For example, if you already have a project called `“cat-classification”`, that project will continue to exist and not be deleted. Instead, a new run is added to that project.
</Note>

### Capture a dictionary of hyperparameters

Save a dictionary of hyperparameters such as learning rate or model type. The model settings you capture in config are useful later to organize and query your results.

```python theme={null}
with wandb.init(
    ...,
    config={"epochs": 100, "learning_rate": 0.001, "batch_size": 128},
) as run:
    ...
```

For more information on how to configure an experiment, see [Configure Experiments](./config).

### Log metrics inside your training loop

Call [`run.log()`](/models/ref/python/experiments/run/#method-runlog) to log metrics about each training step such as accuracy and loss.

```python theme={null}
model, dataloader = get_model(), get_data()

for epoch in range(run.config.epochs):
    for batch in dataloader:
        loss, accuracy = model.training_step()
        run.log({"accuracy": accuracy, "loss": loss})
```

For more information on different data types you can log with W\&B, see [Log Data During Experiments](/models/track/log/).

### Log an artifact to W\&B

Optionally log a W\&B Artifact. Artifacts make it easy to version datasets and models.

```python theme={null}
# You can save any file or even a directory. In this example, we pretend
# the model has a save() method that outputs an ONNX file.
model.save("path_to_model.onnx")
run.log_artifact("path_to_model.onnx", name="trained-model", type="model")
```

Learn more about [Artifacts](/models/artifacts/) or about versioning models in [Registry](/models/registry/).

### Putting it all together

The full script with the preceding code snippets is found below:

```python theme={null}
import wandb

with wandb.init(
    project="cat-classification",
    notes="",
    tags=["baseline", "paper1"],
    # Record the run's hyperparameters.
    config={"epochs": 100, "learning_rate": 0.001, "batch_size": 128},
) as run:
    # Set up model and data.
    model, dataloader = get_model(), get_data()

    # Run your training while logging metrics to visualize model performance.
    for epoch in range(run.config["epochs"]):
        for batch in dataloader:
            loss, accuracy = model.training_step()
            run.log({"accuracy": accuracy, "loss": loss})

    # Upload the trained model as an artifact.
    model.save("path_to_model.onnx")
    run.log_artifact("path_to_model.onnx", name="trained-model", type="model")
```

## Next steps: Visualize your experiment

Use the W\&B Dashboard as a central place to organize and visualize results from your machine learning models. With just a few clicks, construct rich, interactive charts like [parallel coordinates plots](/models/app/features/panels/parallel-coordinates/), [parameter importance analyses](/models/app/features/panels/parameter-importance/), and [additional chart types](/models/app/features/panels/).

<Frame>
  <img alt="Quickstart Sweeps Dashboard example" />
</Frame>

For more information on how to view experiments and specific runs, see [Visualize results from experiments](/models/track/workspaces/).

## Best practices

The following are some suggested guidelines to consider when you create experiments:

1. **Finish your runs**: Use `wandb.init()` in a `with` statement to automatically mark the run as finished when the code completes or raises an exception.
   * In Jupyter notebooks, it may be more convenient to manage the Run object yourself. In this case, you can explicitly call `finish()` on the Run object to mark it complete:

     ```python theme={null}
     # In a notebook cell:
     run = wandb.init()

     # In a different cell:
     run.finish()
     ```
2. **Config**: Track hyperparameters, architecture, dataset, and anything else you'd like to use to reproduce your model. These will show up in columns— use config columns to group, sort, and filter runs dynamically in the app.
3. **Project**: A project is a set of experiments you can compare together. Each project gets a dedicated dashboard page, and you can easily turn on and off different groups of runs to compare different model versions.
4. **Notes**: Set a quick commit message directly from your script. Edit and access notes in the Overview section of a run in the W\&B App.
5. **Tags**: Identify baseline runs and favorite runs. You can filter runs using tags. You can edit tags at a later time on the Overview section of your project's dashboard on the W\&B App.
6. **Create multiple run sets to compare experiments**: When comparing experiments, create multiple run sets to make metrics easy to compare. You can toggle run sets on or off on the same chart or group of charts.

The following code snippet demonstrates how to define a W\&B Experiment using the best practices listed above:

```python theme={null}
import wandb

config = {
    "learning_rate": 0.01,
    "momentum": 0.2,
    "architecture": "CNN",
    "dataset_id": "cats-0192",
}

with wandb.init(
    project="detect-cats",
    notes="tweak baseline",
    tags=["baseline", "paper1"],
    config=config,
) as run:
    ...
```

For more information about available parameters when defining a W\&B Experiment, see the [`wandb.init()`](/models/ref/python/functions/init) API docs in the [API Reference Guide](/models/ref/python/).


# Environment variables
Source: https://docs.wandb.ai/models/track/environment-variables

Configure W&B SDK behavior using environment variables for authentication, project settings, logging modes, and more.

When you're running a script in an automated environment, you can control W\&B with environment variables set before the script runs or within the script.

```bash theme={null}
# This is secret and shouldn't be checked into version control
WANDB_API_KEY=$YOUR_API_KEY
# Name and notes optional
WANDB_NAME="My first run"
WANDB_NOTES="Smaller learning rate, more regularization."
```

```bash theme={null}
# Only needed if you don't check in the wandb/settings file
WANDB_ENTITY=$username
WANDB_PROJECT=$project
```

```python theme={null}
# If you don't want your script to sync to the cloud
os.environ["WANDB_MODE"] = "offline"

# Add sweep ID tracking to Run objects and related classes
os.environ["WANDB_SWEEP_ID"] = "b05fq58z"
```

## Optional environment variables

Use these optional environment variables to do things like set up authentication on remote machines.

| Variable name           | Usage                                                                                                                                                                                                                                                                                                                                        |
| ----------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `WANDB_API_KEY`         | Sets the authentication key associated with your account. Create an API key at [User Settings](https://wandb.ai/settings). This must be set if `wandb login` hasn't been run on the remote machine.                                                                                                                                          |
| `WANDB_BASE_URL`        | If you're using [wandb/local](/platform/hosting/) you should set this environment variable to `http://YOUR_IP:YOUR_PORT`                                                                                                                                                                                                                     |
| `WANDB_CACHE_DIR`       | This defaults to \~/.cache/wandb, you can override this location with this environment variable                                                                                                                                                                                                                                              |
| `WANDB_CONFIG_DIR`      | This defaults to \~/.config/wandb, you can override this location with this environment variable                                                                                                                                                                                                                                             |
| `WANDB_CONFIG_PATHS`    | Comma separated list of yaml files to load into wandb.config. See [config](./config#file-based-configs).                                                                                                                                                                                                                                     |
| `WANDB_CONSOLE`         | Set this to "off" to disable stdout / stderr logging. This defaults to "on" in environments that support it.                                                                                                                                                                                                                                 |
| `WANDB_DATA_DIR`        | Where to upload staging artifacts. The default location depends on your platform, because it uses the value of `user_data_dir` from the `platformdirs` Python package. Make sure this directory exists and the running user has permission to write to it.                                                                                   |
| `WANDB_DIR`             | Where to store all generated files. If unset, defaults to the `wandb` directory relative to your training script. Make sure this directory exists and the running user has permission to write to it. This does not control the location of downloaded artifacts, which you can set using the `WANDB_ARTIFACT_DIR` environment variable.     |
| `WANDB_ARTIFACT_DIR`    | Where to store all downloaded artifacts. If unset, defaults to the `artifacts` directory relative to your training script. Make sure this directory exists and the running user has permission to write to it. This does not control the location of generated metadata files, which you can set using the `WANDB_DIR` environment variable. |
| `WANDB_DISABLE_GIT`     | Prevent wandb from probing for a git repository and capturing the latest commit / diff.                                                                                                                                                                                                                                                      |
| `WANDB_DISABLE_CODE`    | Set this to true to prevent wandb from saving notebooks or git diffs. We'll still save the current commit if we're in a git repo.                                                                                                                                                                                                            |
| `WANDB_DOCKER`          | Set this to a docker image digest to enable restoring of runs. This is set automatically with the wandb docker command. You can obtain an image digest by running `wandb docker my/image/name:tag --digest`                                                                                                                                  |
| `WANDB_ENTITY`          | The entity associated with your run. If you have run `wandb init` in the directory of your training script, it will create a directory named *wandb* and will save a default entity which can be checked into source control. If you don't want to create that file or want to override the file you can use the environmental variable.     |
| `WANDB_ERROR_REPORTING` | Set this to false to prevent wandb from logging fatal errors to its error tracking system.                                                                                                                                                                                                                                                   |
| `WANDB_HOST`            | Set this to the hostname you want to see in the wandb interface if you don't want to use the system provided hostname                                                                                                                                                                                                                        |
| `WANDB_IGNORE_GLOBS`    | Set this to a comma separated list of file globs to ignore. These files will not be synced to the cloud.                                                                                                                                                                                                                                     |
| `WANDB_JOB_NAME`        | Specify a name for any jobs created by `wandb`.                                                                                                                                                                                                                                                                                              |
| `WANDB_JOB_TYPE`        | Specify the job type, like "training" or "evaluation" to indicate different types of runs. See [grouping](/models/runs/grouping/) for more info.                                                                                                                                                                                             |
| `WANDB_MODE`            | If you set this to "offline" wandb will save your run metadata locally and not sync to the server. If you set this to `disabled` wandb will turn off completely.                                                                                                                                                                             |
| `WANDB_NAME`            | The human-readable name of your run. If not set it will be randomly generated for you                                                                                                                                                                                                                                                        |
| `WANDB_NOTEBOOK_NAME`   | If you're running in jupyter you can set the name of the notebook with this variable. We attempt to auto detect this.                                                                                                                                                                                                                        |
| `WANDB_NOTES`           | Longer notes about your run. Markdown is allowed and you can edit this later in the UI.                                                                                                                                                                                                                                                      |
| `WANDB_PROJECT`         | The project associated with your run. This can also be set with `wandb init`, but the environmental variable will override the value.                                                                                                                                                                                                        |
| `WANDB_RESUME`          | By default this is set to *never*. If set to *auto* wandb will automatically resume failed runs. If set to *must* forces the run to exist on startup. If you want to always generate your own unique ids, set this to *allow* and always set `WANDB_RUN_ID`.                                                                                 |
| `WANDB_RUN_GROUP`       | Specify the experiment name to automatically group runs together. See [grouping](/models/runs/grouping/) for more info.                                                                                                                                                                                                                      |
| `WANDB_RUN_ID`          | Set this to a globally unique string (per project) corresponding to a single run of your script. It must be no longer than 64 characters. All non-word characters will be converted to dashes. This can be used to resume an existing run in cases of failure.                                                                               |
| `WANDB_QUIET`           | Set this to `true` to limit statements logged to standard output to critical statements only. If this is set all logs will be written to `$WANDB_DIR/debug.log`.                                                                                                                                                                             |
| `WANDB_SILENT`          | Set this to `true` to silence wandb log statements. This is useful for scripted commands. If this is set all logs will be written to `$WANDB_DIR/debug.log`.                                                                                                                                                                                 |
| `WANDB_SHOW_RUN`        | Set this to `true` to automatically open a browser with the run url if your operating system supports it.                                                                                                                                                                                                                                    |
| `WANDB_SWEEP_ID`        | Add sweep ID tracking to `Run` objects and related classes, and display in the UI.                                                                                                                                                                                                                                                           |
| `WANDB_TAGS`            | A comma separated list of tags to be applied to the run.                                                                                                                                                                                                                                                                                     |
| `WANDB_USERNAME`        | The username of a member of your team associated with the run. This can be used along with a service account API key to enable attribution of automated runs to members of your team.                                                                                                                                                        |
| `WANDB_USER_EMAIL`      | The email of a member of your team associated with the run. This can be used along with a service account API key to enable attribution of automated runs to members of your team.                                                                                                                                                           |

## Singularity environments

If you're running containers in [Singularity](https://singularity.lbl.gov/index.html) you can pass environment variables by pre-pending the above variables with `SINGULARITYENV_`. More details about Singularity environment variables can be found [here](https://singularity.lbl.gov/docs-environment-metadata#environment).

## Running on AWS

If you're running batch jobs in AWS, it's easy to authenticate your machines with your W\&B credentials. Create an API key at [User Settings](https://wandb.ai/settings), and set the `WANDB_API_KEY` environment variable in the [AWS batch job spec](https://docs.aws.amazon.com/batch/latest/userguide/job_definition_parameters.html#parameters).


# Track Jupyter notebooks
Source: https://docs.wandb.ai/models/track/jupyter

Use W&B with Jupyter to get interactive visualizations without leaving your notebook.

Use W\&B with Jupyter to get interactive visualizations without leaving your notebook. Combine custom analysis, experiments, and prototypes, all fully logged.

## Use cases for W\&B with Jupyter notebooks

1. **Iterative experimentation**: Run and re-run experiments, tweaking parameters, and have all the runs you do saved automatically to W\&B without having to take manual notes along the way.
2. **Code saving**: When reproducing a model, it's hard to know which cells in a notebook ran, and in which order. Turn on code saving on your [settings page](/platform/app/settings-page/) to save a record of cell execution for each experiment.
3. **Custom analysis**: Once runs are logged to W\&B, it's easy to get a dataframe from the API and do custom analysis, then log those results to W\&B to save and share in reports.

## Getting started in a notebook

Start your notebook with the following code to install W\&B and link your account:

```notebook theme={null}
!pip install wandb -qqq
import wandb
wandb.login()
```

Next, set up your experiment and save hyperparameters:

```python theme={null}
wandb.init(
    project="jupyter-projo",
    config={
        "batch_size": 128,
        "learning_rate": 0.01,
        "dataset": "CIFAR-100",
    },
)
```

After running `wandb.init()` , start a new cell with `%%wandb` to see live graphs in the notebook. If you run this cell multiple times, data will be appended to the run.

```notebook theme={null}
%%wandb

# Your training loop here
```

Try it for yourself in this [example notebook](https://wandb.me/jupyter-interact-colab).

<Frame>
  <img alt="Jupyter W&B widget" />
</Frame>

### Rendering live W\&B interfaces directly in your notebooks

You can also display any existing dashboards, sweeps, or reports directly in your notebook using the `%wandb` magic:

```notebook theme={null}
# Display a project workspace
%wandb USERNAME/PROJECT
# Display a single run
%wandb USERNAME/PROJECT/runs/RUN_ID
# Display a sweep
%wandb USERNAME/PROJECT/sweeps/SWEEP_ID
# Display a report
%wandb USERNAME/PROJECT/reports/REPORT_ID
# Specify the height of embedded iframe
%wandb USERNAME/PROJECT -h 2048
```

As an alternative to the `%%wandb` or `%wandb` magics, after running `wandb.init()` you can end any cell with `wandb.Run.finish()` to show in-line graphs, or call `ipython.display(...)` on any report, sweep, or run object returned from our apis.

```python theme={null}
import wandb
from IPython.display import display
# Initialize a run
run = wandb.init()

# If cell outputs run.finish(), you'll see live graphs
run.finish()
```

<Note>
  Want to know more about what you can do with W\&B? Check out our [guide to logging data and media](/models/track/log/), learn [how to integrate us with your favorite ML toolkits](/models/integrations), or just dive straight into the [reference docs](/models/ref/python/) or our [repo of examples](https://github.com/wandb/examples).
</Note>

## Additional Jupyter features in W\&B

1. **Easy authentication in Colab**: When you call `wandb.init()` for the first time in a Colab, we automatically authenticate your runtime if you're currently logged in to W\&B in your browser. On the overview tab of your run page, you'll see a link to the Colab.
2. **Jupyter Magic:** Display dashboards, sweeps and reports directly in your notebooks. The `%wandb` magic accepts a path to your project, sweeps or reports and will render the W\&B interface directly in the notebook.
3. **Launch dockerized Jupyter**: Call `wandb docker --jupyter` to launch a docker container, mount your code in it, ensure Jupyter is installed, and launch on port 8888.
4. **Run cells in arbitrary order without fear**: By default, we wait until the next time `wandb.init()` is called to mark a run as `finished`. That allows you to run multiple cells (say, one to set up data, one to train, one to test) in whatever order you like and have them all log to the same run. If you turn on code saving in [User Settings](https://wandb.ai/settings), you'll also log the cells that were executed, in order and in the state in which they were run, enabling you to reproduce even the most non-linear of pipelines. To mark a run as complete manually in a Jupyter notebook, call `wandb.Run.finish()`.

```python theme={null}
import wandb

run = wandb.init()

# training script and logging goes here

run.finish()
```


# Logging at scale and performance
Source: https://docs.wandb.ai/models/track/limits

This page describes how logging patterns impact performance in W&B and provides guidance for scaling experiment tracking in large projects.

Performance is usually influenced by a combination of:

* the number of runs in a project
* the number of steps in each run
* the number of distinct metrics you log
* how often you call `wandb.Run.log()`
* how much data you send in each log call
* how your workspace is configured

In most cases, performance issues are caused more often by logging too many distinct metrics than by logging too many steps.

## Key terms

The following terms are used throughout this page.

### Steps

A **step** is a single logical row of metrics in a run. A step is finalized when `wandb.Run.log()` is called with `commit=True`, or implicitly when neither `commit` nor `step` is specified.

```python theme={null}
import wandb

with wandb.init() as run:
    run.log({"loss": 0.42}, commit=True)
```

### Metric cardinality

**Metric cardinality** is the number of distinct metric keys logged in a project, including keys in nested dictionaries.

For example, the following logs 4 distinct metric keys: `a`, `b.c`, `b.d.e`, and `b.d.f`.

```python theme={null}
import wandb

with wandb.init() as run:
    run.log(
        {
            "a": 1,
            "b": {
                "c": 2,
                "d": {
                    "e": 3,
                    "f": 4,
                },
            },
        }
    )
```

W\&B flattens nested dictionaries into dot-separated metric names.

### Logged points

**Logged points** are the total number of metric values recorded.

For example, both of the following code samples produce three logged points:

```python theme={null}
import wandb

with wandb.init() as run:
    run.log({"a": 1, "b": 2, "c": 3})
```

```python theme={null}
import wandb

with wandb.init() as run:
    run.log({"a": 1})
    run.log({"a": 2})
    run.log({"a": 3})
```

### Log frequency

**Log frequency** is the number of `wandb.Run.log()` calls per minute.

```text theme={null}
log frequency = wandb.Run.log() calls per minute
```

### Throughput

**Throughput** is the total number of logged points recorded per minute.

You can think of throughput as:

```text theme={null}
throughput = logged points per minute
```

Or, equivalently:

```text theme={null}
throughput = logged points × log frequency
```

## Recommendations at scale

<Warning>
  The recommendations described in this section only apply to W\&B Multi-tenant Cloud. If you use a different W\&B deployment type, check with your administrator for deployment-specific guidance or limits.
</Warning>

The following table summarizes recommended operating ranges for large-scale logging.

| Dimension                      | Guidance at scale         |
| ------------------------------ | ------------------------- |
| Runs per project               | 10,000                    |
| Steps per run                  | 500,000                   |
| Metric cardinality per project | 100,000                   |
| Log frequency                  | 1,000 rows per minute     |
| Throughput                     | 100,000 values per minute |
| Video throughput               | 40 MB per minute          |

<Note>
  These values are guidelines for maintaining good performance at scale. W\&B may continue to accept data beyond these recommendations, but pages can become slower to load and use.
</Note>

## Throughput examples

Different logging patterns can produce the same throughput.

### Scalar logging examples

<Warning>
  The values listed in the table only apply to W\&B Multi-tenant Cloud. If you use a different W\&B deployment type, check with your administrator for deployment-specific guidance or limits.
</Warning>

| Metrics per log call | Log frequency (per minute) | Throughput (values per minute) |
| -------------------- | -------------------------- | ------------------------------ |
| 100                  | 1,000                      | 100,000                        |
| 1,000                | 100                        | 100,000                        |
| 10,000               | 10                         | 100,000                        |
| 20,000               | 5                          | 100,000                        |

### Video logging examples

<Warning>
  The values listed in the table only apply to W\&B Multi-tenant Cloud. If you use a different W\&B deployment type, check with your administrator for deployment-specific guidance or limits.
</Warning>

| Video size (MB) | Log frequency (per minute) | Video throughput (MB per minute) |
| --------------- | -------------------------- | -------------------------------- |
| 1               | 46                         | 46                               |
| 5               | 8                          | 40                               |
| 10              | 4                          | 40                               |
| 50              | 1                          | 50                               |
| 100             | 0.3                        | 30                               |
| 250             | 0.1                        | 25                               |
| 500             | 0.07                       | 35                               |

## Logging considerations

Use `wandb.Run.log()` to track experiment metrics.

### Metric cardinality

Keep the total metric cardinality (number of distinct metrics) in a project within the recommended range for your workload. High metric cardinality is one of the most common causes of slow workspaces.

<Tip>
  Performance issues are often caused by logging too many distinct metrics, not by logging too many steps.
</Tip>

Because W\&B flattens nested keys into dot-separated metric names, metric cardinality can increase more than you expect.

For example, the following logs 3 distinct metric keys: `a`, `b.c`, and `b.d`.

```python theme={null}
import wandb

with wandb.init() as run:
    run.log(
        {
            "a": 1,
            "b": {
                "c": "hello",
                "d": [1, 2, 3],
            },
        }
    )
```

If your workspace suddenly slows down, check whether recent runs introduced a large number of new metric keys. This often appears as many plots with only one or two runs visible. If this happened unintentionally, consider deleting and recreating those runs with a smaller, more stable set of metric names.

### Value size

Keep the size of a single logged value under 1 MB and the total size of a single `wandb.Run.log()` call under 25 MB.

These recommendations do not apply to `wandb.Media` types such as `wandb.Image` and `wandb.Audio`, which are handled differently.

```python theme={null}
import json
import wandb

with wandb.init(project="wide-values") as run:
    # Not recommended
    run.log({"wide_key": list(range(10000000))})

    # Not recommended
    with open("large_file.json", "r") as f:
        large_data = json.load(f)
        run.log(large_data)
```

Large values can slow plot loading for the entire run, not just for the metric that contains the large value.

<Note>
  W\&B still stores logged data that exceeds these recommendations, but pages may load more slowly.
</Note>

### Log frequency and throughput

Choose a logging frequency that matches the value of the data you are collecting. Logging too often can increase SDK overhead and make the app slower, especially when combined with high metric cardinality or large payloads.

As a starting point, keep logging within these guidelines:

* Log frequency: less than 1,000 `wandb.Run.log()` calls per minute
* Throughput: less than 100,000 logged values per minute
* Video throughput: less than 40 MB per minute

Batch related metrics into the same step when possible. For example, the following code snippet logs three metrics in the same step, which is more efficient than logging them separately.

```python theme={null}
import wandb

with wandb.init(project="metric-frequency") as run:
    # Recommended: batch related scalar metrics together
    run.log(
        {
            "loss": 0.12,
            "accuracy": 0.98,
            "lr": 1e-4,
        },
        commit=True,
    )
```

### Config size

Keep the total size of a run config under 10 MB.

Large configs can slow project workspaces and runs table operations.

```python theme={null}
import json
import wandb

# Recommended
with wandb.init(
    project="config-size",
    config={
        "lr": 0.1,
        "batch_size": 32,
        "epochs": 4,
    },
) as run:
    pass

# Not recommended
with wandb.init(
    project="config-size",
    config={
        "large_list": list(range(10000000)),
        "large_string": "a" * 10000000,
    },
) as run:
    pass

# Not recommended
with open("large_config.json", "r") as f:
    large_config = json.load(f)
    wandb.init(config=large_config)
```

## Workspace performance

Workspace performance depends on both the underlying project data and workspace configuration.

### Runs per project

For large projects, keep the number of runs in a project under 10,000 for best performance.

If your team regularly works with only a subset of runs, consider moving older or less frequently used runs into a separate archive project. See [Manage runs](/models/runs/manage-runs/).

### Panel count

By default, a workspace in automatic mode creates standard panels for each logged key. In large projects, this can produce too many panels and slow the workspace.

To improve performance:

1. Reset the workspace to manual mode.
2. Use [Quick add](/models/app/features/panels/#quick-add) to add only the panels you need.

<Note>
  Deleting unused panels one at a time usually has little effect. Reset the workspace and add back only the panels you want.
</Note>

See [Panels](/models/app/features/panels/) for details.

### Section count

Hundreds of sections in a workspace can hurt performance.

Create sections based on high-level metric groupings rather than one section per metric. If you have too many sections, consider creating sections by prefix rather than suffix so that related metrics are grouped into fewer sections.

<Frame>
  <img alt="Toggling section creation" />
</Frame>

### Many metrics per run

When logging thousands of metrics per run, use a manual workspace so that you can choose which metrics to visualize.

A focused set of panels loads faster. Metrics that are not plotted are still collected and stored.

To reset a workspace to manual mode, click the workspace's **action (<Icon icon="ellipsis" />)** menu, then click **Reset workspace**. Resetting a workspace has no impact on stored metrics for runs. See [workspace panel management](/models/app/features/panels/).

### File count

Keep the number of files uploaded for a single run under 1,000.

If you need to log a large number of files, use W\&B Artifacts instead. Exceeding 1,000 files in a single run can slow run pages.

### Reports and workspaces

A report is designed for communication and presentation. A workspace is designed for dense, interactive analysis across many runs and metrics.

Use a workspace when you need to compare large numbers of runs or view many plots together. Use a report when you want to present curated results.

## Python script performance

Logging can add overhead to your training script. The main contributors are:

1. Large payloads
2. Network speed and backend configuration
3. Very frequent calls to `wandb.Run.log()`

If you call `wandb.Run.log()` too often, each call can add a small amount of latency to the training loop. Batching multiple metrics into fewer log calls usually improves performance.

<Note>
  Is frequent logging slowing your training runs down? See [this Colab](https://wandb.me/log-hf-colab) for strategies to improve performance by adjusting your logging pattern.
</Note>

W\&B does not enforce hard product limits for these recommendations beyond API rate limiting. If you exceed the guidance on this page, W\&B may continue to accept your data, but the app or SDK may become slower.

## Rate limits

W\&B Multi-tenant Cloud APIs use rate limits to maintain service reliability and availability.

<Note>
  Rate limits are subject to change.
</Note>

If you hit a rate limit, the server returns HTTP `429 Rate limit exceeded` and includes rate-limit headers in the response.

### Rate-limit HTTP headers

| Header name           | Description                                                       |
| --------------------- | ----------------------------------------------------------------- |
| `RateLimit-Limit`     | Quota available in the current time window, scaled from 0 to 1000 |
| `RateLimit-Remaining` | Remaining quota in the current window, scaled from 0 to 1000      |
| `RateLimit-Reset`     | Number of seconds until the current quota resets                  |

### Metric logging API rate limits

`wandb.Run.log()` sends training data to W\&B, either directly online or later through [offline syncing](/models/ref/cli/wandb-sync).

Rate limits for metric logging apply at the project level and include both request rate and total request size over a rolling time window. Paid plans have higher limits than free plans.

If you exceed a rate limit, the W\&B SDK automatically retries requests with backoff. In some cases, this can delay `run.finish()` until the rate-limit window resets.

To reduce the chance of rate limiting:

* Use the latest W\&B SDK version.
* Reduce logging frequency.
* Batch related metrics into fewer log calls.
* Use offline logging and sync later when appropriate.

```python theme={null}
import random
import wandb

with wandb.init(project="basic-intro") as run:
    for epoch in range(10):
        accuracy = 1 - 2 ** -epoch - random.random() / (epoch + 1)
        loss = 2 ** -epoch + random.random() / (epoch + 1)

        if epoch % 5 == 0:
            run.log({"acc": accuracy, "loss": loss})
```

For manual syncing, use `wandb sync <run-file-path>`. See [`wandb sync`](/models/ref/cli/wandb-sync).

### GraphQL API rate limits

The W\&B app and the [public API](/models/ref/python/public-api/api) use GraphQL requests to query and modify data.

For Multi-tenant Cloud:

* unauthorized requests are rate-limited per IP address
* authorized requests are rate-limited per user
* some SDK requests that specify a project path can also be limited per project based on database query time

Teams and Enterprise plans have higher limits than Free plans.

If you are making a large number of public API requests, wait at least one second between requests when possible. If you receive HTTP `429 Rate limit exceeded` or see `RateLimit-Remaining=0`, wait for the number of seconds in `RateLimit-Reset` before retrying.

## Troubleshooting slow projects

If a project or workspace feels slow, check the following first:

1. Did recent runs introduce a large number of new metric names?
2. Are you logging too frequently?
3. Are individual `run.log()` calls very large?
4. Is the workspace in automatic mode with too many panels or sections?
5. Does the project contain more runs than your team actively uses?

In many cases, performance improves after reducing metric cardinality, batching log calls, and switching large workspaces to manual mode.

## Browser considerations

The W\&B app can be memory-intensive and performs best in Chrome. Depending on your computer's memory, having W\&B active in 3+ tabs at once can cause performance to degrade. If you encounter unexpectedly slow performance, consider closing other tabs or applications.

## Reporting performance issues to W\&B

W\&B takes performance seriously and investigates every report of lag. To expedite investigation, when reporting slow loading times consider invoking W\&B's built-in performance logger that captures key metrics and performance events. Append the URL parameter `&PERF_LOGGING` to a page that is loading slowly, then share the output of your console with your account team or Support.

<Frame>
  <img alt="Adding PERF_LOGGING" />
</Frame>


# Overview
Source: https://docs.wandb.ai/models/track/log

Keep track of metrics, videos, custom plots, and more

Log a dictionary of metrics, media, or custom objects to a step with the W\&B Python SDK. W\&B collects the key-value pairs during each step and stores them in one unified dictionary each time you log data with `wandb.Run.log()`. Data logged from your script is saved locally to your machine in a directory called `wandb`, then synced to the W\&B cloud or your [private server](/platform/hosting/).

<Note>
  Key-value pairs are stored in one unified dictionary only if you pass the same value for each step. W\&B writes all of the collected keys and values to memory if you log a different value for `step`.
</Note>

Each call to `wandb.Run.log()` is a new `step` by default. W\&B uses steps as the default x-axis when it creates charts and panels. You can optionally create and use a custom x-axis or capture a custom summary metric. For more information, see [Customize log axes](/models/track/log/customize-logging-axes/).

<Warning>
  Use `wandb.Run.log()` to log consecutive values for each `step`: 0, 1, 2, and so on. It is not possible to write to a specific history step. W\&B only writes to the "current" and "next" step.
</Warning>

## Automatically logged data

W\&B automatically logs the following information during a W\&B Experiment:

* **System metrics**: CPU and GPU utilization, network, etc. For the GPU, these are fetched with [`nvidia-smi`](https://developer.nvidia.com/nvidia-system-management-interface).
* **Command line**: The stdout and stderr are picked up and show in the logs tab on the [run page.](/models/runs/)

Turn on [Code Saving](https://wandb.me/code-save-colab) in your account's [Settings page](https://wandb.ai/settings) to log:

* **Git commit**: Pick up the latest git commit and see it on the overview tab of the run page, as well as a `diff.patch` file if there are any uncommitted changes.
* **Dependencies**: The `requirements.txt` file will be uploaded and shown on the files tab of the run page, along with any files you save to the `wandb` directory for the run.

## What data is logged with specific W\&B API calls?

With W\&B, you can decide exactly what you want to log. The following lists some commonly logged objects:

* **Datasets**: You have to specifically log images or other dataset samples for them to stream to W\&B.
* **Plots**: Use `wandb.plot()` with `wandb.Run.log()` to track charts. See [Log Plots](/models/track/log/plots/) for more information.
* **Tables**: Use `wandb.Table` to log data to visualize and query with W\&B. See [Log Tables](/models/track/log/log-tables/) for more information.
* **PyTorch gradients**: Add `wandb.Run.watch(model)` to see gradients of the weights as histograms in the UI.
* **Configuration information**: Log hyperparameters, a link to your dataset, or the name of the architecture you're using as config parameters, passed in like this: `wandb.init(config=your_config_dictionary)`.
* **Metrics**: Use `wandb.Run.log()` to see metrics from your model. If you log metrics like accuracy and loss from inside your training loop, you'll get live updating graphs in the UI.

## Metric naming constraints

Due to GraphQL limitations, metric names in W\&B must follow specific naming rules:

* **Allowed characters**: Letters (A-Z, a-z), digits (0-9), and underscores (\_)
* **Starting character**: Names must start with a letter or underscore
* **Pattern**: Metric names should match `/^[_a-zA-Z][_a-zA-Z0-9]*$/`

<Warning>
  Avoid naming metrics with invalid characters (such as commas, spaces, or special symbols), which may cause problems with sorting, querying, or display in the W\&B UI.
</Warning>

**Valid metric names:**

```python theme={null}
with wandb.init() as run:
  run.log({"accuracy": 0.9, "val_loss": 0.1, "epoch_5": 5})
  run.log({"modelAccuracy": 0.95, "learning_rate": 0.001})
```

**Invalid metric names (avoid these):**

```python theme={null}
with wandb.init() as run:
  run.log({"acc,val": 0.9})  # Contains comma
  run.log({"loss-train": 0.1})  # Contains hyphen
  run.log({"test acc": 0.95})  # Contains space
  run.log({"5_fold_cv": 0.8})  # Starts with number
```

## Common workflows

1. **Compare the best accuracy**: To compare the best value of a metric across runs, set the summary value for that metric. By default, summary is set to the last value you logged for each key. This is useful in the table in the UI, where you can sort and filter runs based on their summary metrics, to help compare runs in a table or bar chart based on their *best* accuracy, instead of final accuracy. For example: `wandb.run.summary["best_accuracy"] = best_accuracy`

2. **View multiple metrics on one chart**: Log multiple metrics in the same call. For example:
   ```python theme={null}
   with wandb.init() as run:
     run.log({"acc": 0.9, "loss": 0.1})
   ```
   You can then plot both metrics in the UI.

3. **Customize the x-axis**: Add a custom x-axis to the same log call to visualize your metrics against a different axis in the W\&B dashboard. For example:
   ```python theme={null}
   with wandb.init() as run:
     run.log({'acc': 0.9, 'epoch': 3, 'batch': 117})
   ```
   To set the default x-axis for a given metric use [Run.define\_metric()](/models/ref/python/experiments/run#define_metric).

4. **Log rich media and charts**: `wandb.Run.log()` supports the logging of a wide variety of data types, from [media like images and videos](/models/track/log/media/) to [tables](/models/track/log/log-tables/) and [charts](/models/app/features/custom-charts/).

## Best practices and tips

For best practices and tips for Experiments and logging, see [Best Practices: Experiments and Logging](https://wandb.ai/wandb/pytorch-lightning-e2e/reports/W-B-Best-Practices-Guide--VmlldzozNTU1ODY1#w\&b-experiments-and-logging).


# Customize log axes
Source: https://docs.wandb.ai/models/track/log/customize-logging-axes

Use define_metric() to set a custom x-axis for logged metrics instead of the default W&B step counter.

Set a custom x-axis when you log metrics to W\&B. By default, W\&B logs metrics as *steps*. Each step corresponds to a `wandb.Run.log()` API call.

For example, the following script has a `for` loop that iterates 10 times. In each iteration, the script logs a metric called `validation_loss` and increments the step number by 1.

```python theme={null}
import wandb

with wandb.init() as run:
  # range function creates a sequence of numbers from 0 to 9
  for i in range(10):
    log_dict = {
        "validation_loss": 1/(i+1)   
    }
    run.log(log_dict)
```

In the project's workspace, the `validation_loss` metric is plotted against the `step` x-axis, which increments by 1 each time `wandb.Run.log()` is called. From the previous code, the x-axis shows the step numbers 0, 1, 2, ..., 9.

<Frame>
  <img alt="Line plot panel that uses `step` as the x-axis." />
</Frame>

In certain situations, it makes more sense to log metrics against a different x-axis such as a logarithmic x-axis. Use the [`define_metric()`](/models/ref/python/experiments/run/#define_metric) method to use any metric you log  as a custom x-axis.

Specify the metric that you want to appear as the y-axis with the `name` parameter. The `step_metric` parameter specifies the metric you want to use as the x-axis. When you log a custom metric, specify a value for both the x-axis and the y-axis as key-value pairs in a dictionary.

Copy and paste the following code snippet to set a custom x-axis metric. Replace the values within `<>` with your own values:

```python theme={null}
import wandb

custom_step = "<custom_step>"  # Name of custom x-axis
metric_name = "<metric>"  # Name of y-axis metric

with wandb.init() as run:
    # Specify the step metric (x-axis) and the metric to log against it (y-axis)
    run.define_metric(step_metric = custom_step, name = metric_name)

    for i in range(10):
        log_dict = {
            custom_step : int,  # Value of x-axis
            metric_name : int,  # Value of y-axis
        }
        run.log(log_dict)
```

As an example, the following code snippet creates a custom x-axis called `x_axis_squared`. The value of the custom x-axis is the square of the for loop index `i` (`i**2`). The y-axis consists of mock values for validation loss (`"validation_loss"`) using Python's built-in `random` module:

```python theme={null}
import wandb
import random

with wandb.init() as run:
    run.define_metric(step_metric = "x_axis_squared", name = "validation_loss")

    for i in range(10):
        log_dict = {
            "x_axis_squared": i**2,
            "validation_loss": random.random(),
        }
        run.log(log_dict)
```

The following image shows the resulting plot in the W\&B App UI. The `validation_loss` metric is plotted against the custom x-axis `x_axis_squared`, which is the square of the for loop index `i`. Note that the x-axis values are `0, 1, 4, 9, 16, 25, 36, 49, 64, 81`, which correspond to the squares of `0, 1, 2, ..., 9` respectively.

<Frame>
  <img alt="Line plot panel that uses a custom x axis. Values are logged to W&B as the square of the loop number." />
</Frame>

You can set a custom x-axis for multiple metrics using `globs` with string prefixes. As an example, the following code snippet plots logged metrics with the prefix `train/*` to the x-axis `train/step`:

```python theme={null}
import wandb

with wandb.init() as run:

    # set all other train/ metrics to use this step
    run.define_metric("train/*", step_metric="train/step")

    for i in range(10):
        log_dict = {
            "train/step": 2**i,  # exponential growth w/ internal W&B step
            "train/loss": 1 / (i + 1),  # x-axis is train/step
            "train/accuracy": 1 - (1 / (1 + i)),  # x-axis is train/step
            "val/loss": 1 / (1 + i),  # x-axis is internal wandb step
        }
        run.log(log_dict)
```


# Log distributed training experiments
Source: https://docs.wandb.ai/models/track/log/distributed-training

Use W&B to log distributed training experiments with multiple GPUs.

During a distributed training experiment, you train a model using multiple machines or clients in parallel. W\&B can help you track distributed training experiments. Based on your use case, track distributed training experiments using one of the following approaches:

* **Track a single process**: Track a rank 0 process (also known as a "leader" or "coordinator") with W\&B. This is a common solution for logging distributed training experiments with the [PyTorch Distributed Data Parallel](https://pytorch.org/docs/stable/generated/torch.nn.parallel.DistributedDataParallel.html#torch.nn.parallel.DistributedDataParallel) (DDP) Class.
* **Track multiple processes**: For multiple processes, you can either:
  * Track each process separately using one run per process. You can optionally group them together in the W\&B App UI.
  * Track all processes to a single run.

<Tip>
  **Concurrent connections**

  Each concurrent connection takes compute, memory, and network resources. Even empty client connections that don't log metrics regularly push system metric updates, leading to slower performance when loading charts.

  W\&B recommends that you limit the maximum number of concurrent client connections as appropriate for your workload and that you monitor resource usage over time. W\&B has tested with a hard limit of 300 concurrent client connections in **Dedicated Cloud**.

  In **Multi-tenant Cloud** organizations, client connections for distributed training are subject to the same [rate limits](/models/track/limits#rate-limits) as regular training runs. Users on [Teams and Enterprise plans](https://wandb.ai/site/pricing) receive higher rate limits than those on the Free plan.
</Tip>

## Track a single process

This section describes how to track values and metrics available to your rank 0 process. Use this approach to track only metrics that are available from a single process. Typical metrics include GPU/CPU utilization, behavior on a shared validation set, gradients and parameters, and loss values on representative data examples.

Within the rank 0 process, initialize a W\&B run with [`wandb.init()`](/models/ref/python/functions/init) and log experiments ([`wandb.Run.log()`](/models/ref/python/experiments/run/#method-runlog)) to that run.

The following [sample Python script (`log-ddp.py`)](https://github.com/wandb/examples/blob/master/examples/pytorch/pytorch-ddp/log-ddp.py) demonstrates one way to track metrics on two GPUs on a single machine using PyTorch DDP. [PyTorch DDP](https://pytorch.org/tutorials/intermediate/ddp_tutorial.html) (`DistributedDataParallel` in`torch.nn`) is a popular library for distributed training. The basic principles apply to any distributed training setup, but the implementation may differ.

The Python script:

1. Starts multiple processes with `torch.distributed.launch`.
2. Checks the rank with the `--local_rank` command line argument.
3. If the rank is set to 0, sets up `wandb` logging conditionally in the [`train()`](https://github.com/wandb/examples/blob/master/examples/pytorch/pytorch-ddp/log-ddp.py#L24) function.

```python theme={null}
if __name__ == "__main__":
    # Get args
    args = parse_args()

    if args.local_rank == 0:  # only on main process
        # Initialize wandb run
        run = wandb.init(
            entity=args.entity,
            project=args.project,
        )
        # Train model with DDP
        train(args, run)
    else:
        train(args)
```

Explore an [example dashboard showing metrics tracked from a single process](https://wandb.ai/ayush-thakur/DDP/runs/1s56u3hc/system).

The dashboard displays system metrics for both GPUs, such as temperature and utilization.

<Frame>
  <img alt="GPU metrics dashboard" />
</Frame>

However, the loss values as a function epoch and batch size were only logged from a single GPU.

<Frame>
  <img alt="Loss function plots" />
</Frame>

## Track multiple processes

Track multiple processes with W\&B with one of the following approaches:

* [Tracking each process separately](/models/track/log/distributed-training/#track-each-process-separately) by creating a run for each process.
* [Tracking all processes to a single run](/models/track/log/distributed-training/#track-all-processes-to-a-single-run).

### Track each process separately

This section describes how to track each process separately by creating a run for each process. Within each run you log metrics, artifacts, and forth to their respective run. Call `wandb.Run.finish()` at the end of training, to mark that the run has completed so that all processes exit properly.

You might find it difficult to keep track of runs across multiple experiments. To mitigate this, provide a value to the `group` parameter when you initialize W\&B (`wandb.init(group='group-name')`) to keep track of which run belongs to a given experiment. For more information about how to keep track of training and evaluation W\&B Runs in experiments, see [Group Runs](/models/runs/grouping/).

<Note>
  **Use this approach if you want to track metrics from individual processes**. Typical examples include the data and predictions on each node (for debugging data distribution) and metrics on individual batches outside of the main node. This approach is not necessary to get system metrics from all nodes nor to get summary statistics available on the main node.
</Note>

The following Python code snippet demonstrates how to set the group parameter when you initialize W\&B:

```python theme={null}
if __name__ == "__main__":
    # Get args
    args = parse_args()
    # Initialize run
    run = wandb.init(
        entity=args.entity,
        project=args.project,
        group="DDP",  # all runs for the experiment in one group
    )
    # Train model with DDP
    train(args, run)

    run.finish()  # mark the run as finished
```

Explore the W\&B App UI to view an [example dashboard](https://wandb.ai/ayush-thakur/DDP?workspace=user-noahluna) of metrics tracked from multiple processes. Note that there are two W\&B Runs grouped together in the left sidebar. Click on a group to view the dedicated group page for the experiment. The dedicated group page displays metrics from each process separately.

<Frame>
  <img alt="Grouped distributed runs" />
</Frame>

The preceding image demonstrates the W\&B App UI dashboard. On the sidebar we see two experiments. One labeled 'null' and a second (bound by a yellow box) called 'DPP'. If you expand the group (select the Group dropdown) you will see the W\&B Runs that are associated to that experiment.

### Organize distributed runs

Set the `job_type` parameter when you initialize W\&B (`wandb.init(job_type='type-name')`) to categorize your nodes based on their function. For example, you might have a main coordinating node and several reporting worker nodes. You can set `job_type` to `main` for the main coordinating node and `worker` for the reporting worker nodes:

```python theme={null}
# Main coordinating node
with wandb.init(project="<project>", job_type="main", group="experiment_1") as run:
     # Training code

# Reporting worker nodes
with wandb.init(project="<project>", job_type="worker", group="experiment_1") as run:
     # Training code
```

Once you have set the `job_type` for your nodes, you can create [saved views](/models/track/workspaces/#create-a-new-saved-workspace-view) in your workspace to organize your runs. Click the **action (<Icon icon="ellipsis" />)** menu at the top right and click **Save as new view**.

For example, you could create the following saved views:

* **Default view**: Filter out worker nodes to reduce noise
  * Click **Filter**, then set **Job Type** to `worker`.
  * Shows only your reporting nodes

* **Debug view**: Focus on worker nodes for troubleshooting
  * Click **Filter**, then set **Job Type** `==` `worker` and set **State** to  `IN` `crashed`.
  * Shows only worker nodes that have crashed or are in error states

* **All nodes view**: See everything together
  * No filter
  * Useful for comprehensive monitoring

To open a saved view, click **Workspaces** in the project sidebar, then click the menu. Workspaces appear at the top of the list and saved views appear at the bottom.

### Track all processes to a single run

<Warning>
  Parameters prefixed by `x_` (such as `x_label`) are in public preview. Create a [GitHub issue in the W\&B repository](https://github.com/wandb/wandb) to provide feedback.
</Warning>

<Note>
  **Requirements**

  To track multiple processes to a single run, you must have:

  * W\&B Python SDK version `v0.19.9` or newer.

  * W\&B Server v0.68 or newer.
</Note>

In this approach you use a primary node and one or more worker nodes. Within the primary node you initialize a W\&B run. For each worker node, initialize a run using the run ID used by the primary node. During training each worker node logs to the same run ID as the primary node. W\&B aggregates metrics from all nodes and displays them in the W\&B App UI.

Within the primary node, initialize a W\&B run with [`wandb.init()`](/models/ref/python/functions/init). Pass in a `wandb.Settings` object to the `settings` parameter (`wandb.init(settings=wandb.Settings()`) with the following:

1. The `mode` parameter set to `"shared"` to enable shared mode.
2. A unique label for [`x_label`](https://github.com/wandb/wandb/blob/main/wandb/sdk/wandb_settings.py#L638). You use the value you specify for `x_label` to identify which node the data is coming from in logs and system metrics in the W\&B App UI. If left unspecified, W\&B creates a label for you using the hostname and a random hash.
3. Set the [`x_primary`](https://github.com/wandb/wandb/blob/main/wandb/sdk/wandb_settings.py#L660) parameter to `True` to indicate that this is the primary node.
4. Optionally provide a list of GPU indexes (\[0,1,2]) to `x_stats_gpu_device_ids` to specify which GPUs W\&B tracks metrics for. If you do not provide a list, W\&B tracks metrics for all GPUs on the machine.

Make note of the run ID of the primary node. Each worker node needs the run ID of the primary node.

<Note>
  `x_primary=True` distinguishes a primary node from worker nodes. Primary nodes are the only nodes that upload files shared across nodes such as configuration files, telemetry and more. Worker nodes do not upload these files.
</Note>

For each worker node, initialize a W\&B run with [`wandb.init()`](/models/ref/python/functions/init) and provide the following:

1. A `wandb.Settings` object to the `settings` parameter (`wandb.init(settings=wandb.Settings()`) with:
   * The `mode` parameter set to `"shared"` to enable shared mode.
   * A unique label for `x_label`. You use the value you specify for `x_label` to identify which node the data is coming from in logs and system metrics in the W\&B App UI. If left unspecified, W\&B creates a label for you using the hostname and a random hash.
   * Set the `x_primary` parameter to `False` to indicate that this is a worker node.
2. Pass the run ID used by the primary node to the `id` parameter.
3. Optionally set [`x_update_finish_state`](https://github.com/wandb/wandb/blob/main/wandb/sdk/wandb_settings.py#L772) to `False`. This prevents non-primary nodes from updating the [run's state](/models/runs/run-states#run-states) to `finished` prematurely, ensuring the run state remains consistent and managed by the primary node.

<Note>
  * Use the same entity and project for all nodes. This helps ensure the correct run ID is found.
  * Consider defining an environment variable on each worker node to set the run ID of the primary node.
</Note>

The following sample code demonstrates the high level requirements for tracking multiple processes to a single run:

```python theme={null}
import wandb

entity = "<team_entity>"
project = "<project_name>"

# Initialize a run in the primary node
run = wandb.init(
    entity=entity,
    project=project,
	settings=wandb.Settings(
        x_label="rank_0", 
        mode="shared", 
        x_primary=True,
        x_stats_gpu_device_ids=[0, 1],  # (Optional) Only track metrics for GPU 0 and 1
        )
)

# Note the run ID of the primary node.
# Each worker node needs this run ID.
run_id = run.id

# Initialize a run in a worker node using the run ID of the primary node
run = wandb.init(
    entity=entity, # Use the same entity as the primary node
    project=project, # Use the same project as the primary node
	settings=wandb.Settings(x_label="rank_1", mode="shared", x_primary=False),
	id=run_id,
)

# Initialize a run in a worker node using the run ID of the primary node
run = wandb.init(
    entity=entity, # Use the same entity as the primary node
    project=project, # Use the same project as the primary node
	settings=wandb.Settings(x_label="rank_2", mode="shared", x_primary=False),
	id=run_id,
)
```

In a real world example, each worker node might be on a separate machine.

<Note>
  See the [Distributed Training with Shared Mode](https://wandb.ai/dimaduev/simple-cnn-ddp/reports/Distributed-Training-with-Shared-Mode--VmlldzoxMTI0NTE1NA) report for an end-to-end example on how to train a model on a multi-node and multi-GPU Kubernetes cluster in GKE.
</Note>

View console logs from multi node processes in the project that the run logs to:

1. Navigate to the project that contains the run.
2. Click on the **Runs** tab in the project sidebar.
3. Click on the run you want to view.
4. Click on the **Logs** tab in the project sidebar.

You can filter console logs based on the labels you provide for `x_label` in the UI search bar located at the top of the console log page. For example, the following image shows which options are available to filter the console log by if values  `rank0`, `rank1`, `rank2`, `rank3`, `rank4`, `rank5`, and `rank6` are provided to `x_label`.\`

<Frame>
  <img alt="Multi-node console logs" />
</Frame>

See [Console logs](/models/app/console-logs/) for more information.

W\&B aggregates system metrics from all nodes and displays them in the W\&B App UI. For example, the following image shows a sample dashboard with system metrics from multiple nodes. Each node possesses a unique label (`rank_0`, `rank_1`, `rank_2`) that you specify in the `x_label` parameter.

<Frame>
  <img alt="Multi-node system metrics" />
</Frame>

See [Line plots](/models/app/features/panels/line-plot/) for information on how to customize line plot panels.

## Example use cases

The following code snippets demonstrate common scenarios for advanced distributed use cases.

### Spawn process

Use the `wandb.setup()`method in your main function if you initiate a run in a spawned process:

```python theme={null}
import multiprocessing as mp

def do_work(n):
    with wandb.init(config=dict(n=n)) as run:
        run.log(dict(this=n * n))

def main():
    wandb.setup()
    pool = mp.Pool(processes=4)
    pool.map(do_work, range(4))


if __name__ == "__main__":
    main()
```

### Share a run

Pass a run object as an argument to share runs between processes:

```python theme={null}
def do_work(run):
    with wandb.init() as run:
        run.log(dict(this=1))

def main():
    run = wandb.init()
    p = mp.Process(target=do_work, kwargs=dict(run=run))
    p.start()
    p.join()
    run.finish()  # mark the run as finished


if __name__ == "__main__":
    main()
```

W\&B can not guarantee the logging order. Synchronization should be done by the author of the script.

## Troubleshooting

There are two common issues you might encounter when using W\&B and distributed training:

1. **Hanging at the beginning of training** - A `wandb` process can hang if the `wandb` multiprocessing interferes with the multiprocessing from distributed training.
2. **Hanging at the end of training** - A training job might hang if the `wandb` process does not know when it needs to exit. Call the `wandb.Run.finish()` API at the end of your Python script to tell W\&B that the run finished. The `wandb.Run.finish()` API will finish uploading data and will cause W\&B to exit.
   W\&B recommends using `wandb service` command to improve the reliability of your distributed jobs. Both of the preceding training issues are commonly found in versions of the W\&B SDK where wandb service is unavailable.

### Enable W\&B Service

Depending on your version of the W\&B SDK, you might already have W\&B Service enabled by default.

#### W\&B SDK 0.13.0 and above

W\&B Service is enabled by default for versions of the W\&B SDK `0.13.0` and above.

#### W\&B SDK 0.12.5 and above

Modify your Python script to enable W\&B Service for W\&B SDK version 0.12.5 and above. Use the `wandb.require()` method and pass the string `"service"` within your main function:

```python theme={null}
if __name__ == "__main__":
    main()


def main():
    wandb.require("service")
    # rest-of-your-script-goes-here
```

For optimal experience we do recommend you upgrade to the latest version.

**W\&B SDK 0.12.4 and below**

Set the `WANDB_START_METHOD` environment variable to `"thread"` to use multithreading instead if you use a W\&B SDK version 0.12.4 and below.


# Log models
Source: https://docs.wandb.ai/models/track/log/log-models

Log model artifacts to a W&B run and retrieve them later using the log_model and use_model SDK methods.

<ColabLink />

# Log models

The following guide describes how to log models to a W\&B run and interact with them.

<Note>
  The following APIs are useful for tracking models as a part of your experiment tracking workflow. Use the APIs listed on this page to log models to a run, and to access metrics, tables, media, and other objects.

  W\&B suggests that you use [W\&B Artifacts](/models/artifacts/) if you want to:

  * Create and keep track of different versions of serialized data besides models, such as datasets, prompts, and more.
  * Explore [lineage graphs](/models/artifacts/explore-and-traverse-an-artifact-graph/) of a model or any other objects tracked in W\&B.
  * Interact with the model artifacts these methods created, such as [updating properties](/models/artifacts/update-an-artifact/) (metadata, aliases, and descriptions)

  For more information on W\&B Artifacts and advanced versioning use cases, see the [Artifacts](/models/artifacts/) documentation.
</Note>

## Log a model to a run

Use the [`log_model`](/models/ref/python/experiments/run#log_model) to log a model artifact that contains content within a directory you specify. The [`log_model`](/models/ref/python/experiments/run#log_model) method also marks the resulting model artifact as an output of the W\&B run.

You can track a model's dependencies and the model's associations if you mark the model as the input or output of a W\&B run. View the lineage of the model within the W\&B App UI. See the [Explore and traverse artifact graphs](/models/artifacts/explore-and-traverse-an-artifact-graph/) page within the [Artifacts](/models/artifacts/) chapter for more information.

Provide the path where your model files are saved to the `path` parameter. The path can be a local file, directory, or [reference URI](/models/artifacts/track-external-files/#amazon-s3--gcs--azure-blob-storage-references) to an external bucket such as `s3://bucket/path`.

Ensure to replace values enclosed in `<>` with your own.

```python theme={null}
import wandb

# Initialize a W&B run
with wandb.init(project="<your-project>", entity="<your-entity>") as run:

    # Log the model
    run.log_model(path="<path-to-model>", name="<name>")
```

Optionally provide a name for the model artifact for the `name` parameter. If `name` is not specified, W\&B will use the basename of the input path prepended with the run ID as the name.

<Note>
  Keep track of the `name` that you, or W\&B assigns, to the model. You will need the name of the model to retrieve the model path with the [`wandb.Run.use_model()`](/models/ref/python/experiments/run#use_model) method.
</Note>

See [`log_model`](/models/ref/python/experiments/run#log_model) in the API Reference for parameters.

<details>
  <summary>Example: Log a model to a run</summary>

  ```python theme={null}
  import os
  import wandb
  from tensorflow import keras
  from tensorflow.keras import layers

  config = {"optimizer": "adam", "loss": "categorical_crossentropy"}

  # Initialize a W&B run
  with wandb.init(entity="charlie", project="mnist-experiments", config=config) as run:

      # Hyperparameters
      loss = run.config["loss"]
      optimizer = run.config["optimizer"]
      metrics = ["accuracy"]
      num_classes = 10
      input_shape = (28, 28, 1)

      # Training algorithm
      model = keras.Sequential(
          [
              layers.Input(shape=input_shape),
              layers.Conv2D(32, kernel_size=(3, 3), activation="relu"),
              layers.MaxPooling2D(pool_size=(2, 2)),
              layers.Conv2D(64, kernel_size=(3, 3), activation="relu"),
              layers.MaxPooling2D(pool_size=(2, 2)),
              layers.Flatten(),
              layers.Dropout(0.5),
              layers.Dense(num_classes, activation="softmax"),
          ]
      )

      # Configure the model for training
      model.compile(loss=loss, optimizer=optimizer, metrics=metrics)

      # Save model
      model_filename = "model.h5"
      local_filepath = "./"
      full_path = os.path.join(local_filepath, model_filename)
      model.save(filepath=full_path)

      # Log the model to the W&B run
      run.log_model(path=full_path, name="MNIST")
  ```

  When you call `wandb.Run.log_model()`, a model artifact named `MNIST` is created and the file `model.h5` is added to the model artifact. Your terminal or notebook will print information of where to find information about the run the model was logged to.

  ```python theme={null}
  View run different-surf-5 at: https://wandb.ai/charlie/mnist-experiments/runs/wlby6fuw
  Synced 5 W&B file(s), 0 media file(s), 1 artifact file(s) and 0 other file(s)
  Find logs at: ./wandb/run-20231206_103511-wlby6fuw/logs
  ```
</details>

## Download and use a logged model

Use the [`use_model`](/models/ref/python/experiments/run#use_model) function to access and download models files previously logged to a W\&B run.

Provide the name of the model artifact where the model files you want to retrieve are stored. The name you provide must match the name of an existing logged model artifact.

If you did not define `name` when you originally logged the files with `log_model`, the default name assigned is the basename of the input path, prepended with the run ID.

Ensure to replace the values enclosed in `<>` with your own:

```python theme={null}
import wandb

# Initialize a run
with wandb.init(project="<your-project>", entity="<your-entity>") as run:

    # Access and download model. Returns path to downloaded artifact
    downloaded_model_path = run.use_model(name="<your-model-name>")
```

The [use\_model](/models/ref/python/experiments/run#use_model) function returns the path of downloaded model files. Keep track of this path if you want to link this model later. In the preceding code snippet, the returned path is stored in a variable called `downloaded_model_path`.

<details>
  <summary>Example: Download and use a logged model</summary>

  For example, in the following code snippet a user called the `use_model` API. They specified the name of the model artifact they want to fetch and they also provided a version/alias. They then stored the path that is returned from the API to the `downloaded_model_path` variable.

  ```python theme={null}
  import wandb

  entity = "luka"
  project = "NLP_Experiments"
  alias = "latest"  # semantic nickname or identifier for the model version
  model_artifact_name = "fine-tuned-model"

  # Initialize a run
  with wandb.init(project=project, entity=entity) as run:
      # Access and download model. Returns path to downloaded artifact
      downloaded_model_path = run.use_model(name = f"{model_artifact_name}:{alias}") 
  ```
</details>

See [`use_model`](/models/ref/python/experiments/run#use_model) in the API Reference for parameters and return type.


# Log summary metrics
Source: https://docs.wandb.ai/models/track/log/log-summary

Track and customize single summary metrics like best accuracy or minimum loss on a W&B run using run.summary.

In addition to values that change over time during training, it is often important to track a single value that summarizes a model or a preprocessing step. Log this information in a W\&B Run's `summary` dictionary. A Run's summary dictionary can handle numpy arrays, PyTorch tensors or TensorFlow tensors. When a value is one of these types we persist the entire tensor in a binary file and store high level metrics in the summary object, such as min, mean, variance, percentiles, and more.

The last value logged with `wandb.Run.log()` is automatically set as the summary dictionary in a W\&B Run. If a summary metric dictionary is modified, the previous value is lost.

The following code snippet demonstrates how to provide a custom summary metric to W\&B:

```python theme={null}
import wandb
import argparse

with wandb.init(config=args) as run:
  best_accuracy = 0
  for epoch in range(1, args.epochs + 1):
      test_loss, test_accuracy = test()
      if test_accuracy > best_accuracy:
          run.summary["best_accuracy"] = test_accuracy
          best_accuracy = test_accuracy
```

You can update the summary attribute of an existing W\&B Run after training has completed. Use the [W\&B Public API](/models/ref/python/public-api/) to update the summary attribute:

```python theme={null}
api = wandb.Api()
run = api.run("username/project/run_id")
run.summary["tensor"] = np.random.random(1000)
run.summary.update()
```

## Customize summary metrics

Custom summary metrics are useful for capturing model performance at the best step of training in your `run.summary`. For example, you might want to capture the maximum accuracy or the minimum loss value, instead of the final value.

By default, the summary uses the final value from history. To customize summary metrics, pass the `summary` argument in `define_metric`. It accepts the following values:

* `"min"`
* `"max"`
* `"mean"`
* `"best"`
* `"last"`
* `"none"`

You can use `"best"` only when you also set the optional `objective` argument to `"minimize"` or `"maximize"`.

The following example adds the min and max values of loss and accuracy to the summary:

```python theme={null}
import wandb
import random

random.seed(1)

with wandb.init() as run:
    # Min and max summary values for loss
    run.define_metric("loss", summary="min")
    run.define_metric("loss", summary="max")

    # Min and max summary values for accuracy
    run.define_metric("acc", summary="min")
    run.define_metric("acc", summary="max")

    for i in range(10):
        log_dict = {
            "loss": random.uniform(0, 1 / (i + 1)),
            "acc": random.uniform(1 / (i + 1), 1),
        }
        run.log(log_dict)
```

## View summary metrics

View summary values in a run's **Overview** page or the project's runs table.

<Tabs>
  <Tab title="Run Overview">
    1. Navigate to the W\&B App.
    2. Select the **Workspace** tab from the project sidebar.
    3. Click the run that logged the summary values. The run page opens with the **Overview** tab shown by default.
    4. View the summary values in the **Summary** section.

    <Frame>
      <img alt="Run overview" />
    </Frame>
  </Tab>

  <Tab title="Run Table">
    1. Navigate to the W\&B App.
    2. Select the **Runs** tab.
    3. Within the runs table, you can view the summary values within the columns based on the name of the summary value.
  </Tab>

  <Tab title="W&B Public API">
    You can use the W\&B Public API to fetch the summary values of a run.

    The following code example demonstrates one way to retrieve the summary values logged to a specific run using the W\&B Public API and pandas:

    ```python theme={null}
    import wandb
    import pandas

    entity = "<your-entity>"
    project = "<your-project>"
    run_name = "<your-run-name>" # Name of run with summary values

    all_runs = []

    for run in api.runs(f"{entity}/{project_name}"):
        print("Fetching details for run: ", run.id, run.name)
        run_data = {
                  "id": run.id,
                  "name": run.name,
                  "url": run.url,
                  "state": run.state,
                  "tags": run.tags,
                  "config": run.config,
                  "created_at": run.created_at,
                  "system_metrics": run.system_metrics,
                  "summary": run.summary,
                  "project": run.project,
                  "entity": run.entity,
                  "user": run.user,
                  "path": run.path,
                  "notes": run.notes,
                  "read_only": run.read_only,
                  "history_keys": run.history_keys,
                  "metadata": run.metadata,
              }
        all_runs.append(run_data)
      
    # Convert to DataFrame  
    df = pd.DataFrame(all_runs)

    # Get row based on the column name (run) and convert to dictionary
    df[df['name']==run_name].summary.reset_index(drop=True).to_dict()
    ```
  </Tab>
</Tabs>


# Log tables
Source: https://docs.wandb.ai/models/track/log/log-tables

Create, populate, and log W&B Tables to visualize and query structured data from your experiment runs.

Use `wandb.Table` to log data to visualize and query with W\&B. In this guide, learn how to:

1. [Create Tables](./log-tables#create-tables)
2. [Add Data](./log-tables#add-data)
3. [Retrieve Data](./log-tables#retrieve-data)
4. [Save Tables](./log-tables#save-tables)

## Create tables

To define a Table, specify the columns you want to see for each row of data. Each row might be a single item in your training dataset, a particular step or epoch during training, a prediction made by your model on a test item, an object generated by your model, etc. Each column has a fixed type: numeric, text, boolean, image, video, audio, etc. You do not need to specify the type in advance. Give each column a name, and make sure to only pass data of that type into that column index. For a more detailed example, see the [W\&B Tables guide](https://wandb.ai/stacey/mnist-viz/reports/Guide-to-W-B-Tables--Vmlldzo2NTAzOTk#1.-how-to-log-a-wandb.table).

Use the `wandb.Table` constructor in one of two ways:

1. **List of Rows:**

   Log named columns and rows of data. For example the following code snippet generates a table with two rows and three columns:

   ```python theme={null}
   wandb.Table(columns=["a", "b", "c"], data=[["1a", "1b", "1c"], ["2a", "2b", "2c"]])
   ```

2. **Pandas DataFrame:** Log a DataFrame using `wandb.Table(dataframe=my_df)`. Column names will be extracted from the DataFrame.

#### From an existing array or dataframe

```python theme={null}
# assume a model has returned predictions on four images
# with the following fields available:
# - the image id
# - the image pixels, wrapped in a wandb.Image()
# - the model's predicted label
# - the ground truth label
my_data = [
    [0, wandb.Image("img_0.jpg"), 0, 0],
    [1, wandb.Image("img_1.jpg"), 8, 0],
    [2, wandb.Image("img_2.jpg"), 7, 1],
    [3, wandb.Image("img_3.jpg"), 1, 1],
]

# create a wandb.Table() with corresponding columns
columns = ["id", "image", "prediction", "truth"]
test_table = wandb.Table(data=my_data, columns=columns)
```

## Add data

Tables are mutable. As your script executes you can add more data to your table, up to 200,000 rows. There are two ways to add data to a table:

1. **Add a Row**: `table.add_data("3a", "3b", "3c")`. Note that the new row is not represented as a list. If your row is in list format, use the star notation, `*` ,to expand the list to positional arguments: `table.add_data(*my_row_list)`. The row must contain the same number of entries as there are columns in the table.
2. **Add a Column**: `table.add_column(name="col_name", data=col_data)`. Note that the length of `col_data` must be equal to the table's current number of rows. Here, `col_data` can be a list data, or a NumPy NDArray.

### Adding data incrementally

This code sample shows how to create and populate a W\&B table incrementally. You define the table with predefined columns, including confidence scores for all possible labels, and add data row by row during inference. You can also [add data to tables incrementally when resuming runs](#adding-data-to-resumed-runs).

```python theme={null}
# Define the columns for the table, including confidence scores for each label
columns = ["id", "image", "guess", "truth"]
for digit in range(10):  # Add confidence score columns for each digit (0-9)
    columns.append(f"score_{digit}")

# Initialize the table with the defined columns
test_table = wandb.Table(columns=columns)

# Iterate through the test dataset and add data to the table row by row
# Each row includes the image ID, image, predicted label, true label, and confidence scores
for img_id, img in enumerate(mnist_test_data):
    true_label = mnist_test_data_labels[img_id]  # Ground truth label
    guess_label = my_model.predict(img)  # Predicted label
    test_table.add_data(
        img_id, wandb.Image(img), guess_label, true_label
    )  # Add row data to the table
```

#### Adding data to resumed runs

You can incrementally update a W\&B table in resumed runs by loading an existing table from an artifact, retrieving the last row of data, and adding the updated metrics. Then, reinitialize the table for compatibility and log the updated version back to W\&B.

```python theme={null}
import wandb

# Initialize a run 
with wandb.init(project="my_project") as run:

    # Load the existing table from the artifact
    best_checkpt_table = run.use_artifact(table_tag).get(table_name)

    # Get the last row of data from the table for resuming
    best_iter, best_metric_max, best_metric_min = best_checkpt_table.data[-1]

    # Update the best metrics as needed

    # Add the updated data to the table
    best_checkpt_table.add_data(best_iter, best_metric_max, best_metric_min)

    # Reinitialize the table with its updated data to ensure compatibility
    best_checkpt_table = wandb.Table(
        columns=["col1", "col2", "col3"], data=best_checkpt_table.data
    )

    # Initialize the Run
    with wandb.init() as run:

        # Log the updated table to W&B
        run.log({table_name: best_checkpt_table})
```

## Retrieve data

Once data is in a Table, access it by column or by row:

1. **Row Iterator**: Use the row iterator of Table such as `for ndx, row in table.iterrows(): ...` to efficiently iterate over the data's rows.
2. **Get a Column**: Retrieve a column of data using `table.get_column("col_name")` . As a convenience, you can pass `convert_to="numpy"` to convert the column to a NumPy NDArray of primitives. This is useful if your column contains media types such as `wandb.Image` so that you can access the underlying data directly.

## Save tables

After you generate a table of data in your script, for example a table of model predictions, save it to W\&B to visualize the results live.

### Log a table to a run

Use `wandb.Run.log()` to save your table to the run, like so:

```python theme={null}
with wandb.init() as run:
    my_table = wandb.Table(columns=["a", "b"], data=[["1a", "1b"], ["2a", "2b"]])
    run.log({"table_key": my_table})
```

Each time a table is logged to the same key, a new version of the table is created and stored in the backend. This means you can log the same table across multiple training steps to see how model predictions improve over time, or compare tables across different runs, as long as they're logged to the same key. You can log up to 200,000 rows.

<Note>
  To log more than 200,000 rows, you can override the limit with:

  `wandb.Table.MAX_ARTIFACT_ROWS = X`

  However, this would likely cause performance issues, such as slower queries, in the UI.
</Note>

### Access tables programmatically

In the backend, Tables are persisted as Artifacts. If you are interested in accessing a specific version, you can do so with the artifact API:

```python theme={null}
with wandb.init() as run:
    my_table = run.use_artifact("run-<run-id>-<table-name>:<tag>").get("<table-name>")
```

For more information on Artifacts, see the [Artifacts Chapter](/models/artifacts/) in the Developer Guide.

### Visualize tables

Any table logged this way will show up in your Workspace on both the Run Page and the Project Page. For more information, see [Visualize and Analyze Tables](/models/tables/visualize-tables/).

## Artifact tables

Use `artifact.add()` to log tables to the Artifacts section of your run instead of the workspace. This could be useful if you have a dataset that you want to log once and then reference for future runs.

```python theme={null}
with wandb.init(project="my_project") as run:
    # create a wandb Artifact for each meaningful step
    test_predictions = wandb.Artifact("mnist_test_preds", type="predictions")

    # [build up your predictions data as above]
    test_table = wandb.Table(data=data, columns=columns)
    test_predictions.add(test_table, "my_test_key")
    run.log_artifact(test_predictions)
```

Refer to this Colab for a [detailed example of artifact.add() with image data](https://wandb.me/dsviz-nature-colab) and this Report for an example of how to use Artifacts and Tables to [version control and deduplicate tabular data](https://wandb.me/TBV-Dedup).

### Join Artifact tables

You can join tables you have locally constructed or tables you have retrieved from other artifacts using `wandb.JoinedTable(table_1, table_2, join_key)`.

| Args      | Description                                                                                                        |
| --------- | ------------------------------------------------------------------------------------------------------------------ |
| table\_1  | (str, `wandb.Table`, ArtifactEntry) the path to a `wandb.Table` in an artifact, the table object, or ArtifactEntry |
| table\_2  | (str, `wandb.Table`, ArtifactEntry) the path to a `wandb.Table` in an artifact, the table object, or ArtifactEntry |
| join\_key | (str, \[str, str]) key or keys on which to perform the join                                                        |

To join two Tables you have logged previously in an artifact context, fetch them from the artifact and join the result into a new Table.

For example, the following code example demonstrates how to read one Table of original songs called `'original_songs'` and another Table of synthesized versions of the same songs called `'synth_songs'`. The code joins the two tables on `"song_id"`, and uploads the resulting table as a new W\&B Table:

```python theme={null}
import wandb

with wandb.init(project="my_project") as run:

    # fetch original songs table
    orig_songs = run.use_artifact("original_songs:latest")
    orig_table = orig_songs.get("original_samples")

    # fetch synthesized songs table
    synth_songs = run.use_artifact("synth_songs:latest")
    synth_table = synth_songs.get("synth_samples")

    # join tables on "song_id"
    join_table = wandb.JoinedTable(orig_table, synth_table, "song_id")
    join_at = wandb.Artifact("synth_summary", "analysis")

    # add table to artifact and log to W&B
    join_at.add(join_table, "synth_explore")
    run.log_artifact(join_at)
```

[Read this tutorial](https://wandb.ai/stacey/cshanty/reports/Whale2Song-W-B-Tables-for-Audio--Vmlldzo4NDI3NzM) for an example on how to combine two previously stored tables stored in different Artifact objects.


# Log media and objects
Source: https://docs.wandb.ai/models/track/log/media

Log rich media, from 3D point clouds and molecules to HTML and histograms

<ColabLink />

We support images, video, audio, and more. Log rich media to explore your results and visually compare your runs, models, and datasets. Read on for examples and how-to guides.

<Note>
  For details, see the [Data types reference](/models/ref/python/data-types/).
</Note>

<Note>
  For more details, check out a [demo report about visualize model predictions](https://wandb.ai/lavanyashukla/visualize-predictions/reports/Visualize-Model-Predictions--Vmlldzo1NjM4OA) or watch a [video walkthrough](https://www.youtube.com/watch?v=96MxRvx15Ts).
</Note>

## Pre-requisites

In order to log media objects with the W\&B SDK, you may need to install additional dependencies.
You can install these dependencies by running the following command:

```bash theme={null}
pip install wandb[media]
```

## Images

Log images to track inputs, outputs, filter weights, activations, and more.

<Frame>
  <img alt="Autoencoder inputs and outputs" />
</Frame>

Images can be logged directly from NumPy arrays, as PIL images, or from the filesystem.

Each time you log images from a step, they are available in the UI. Expand the image panel, then use the step slider to look at images from different steps. This makes it easy to compare how a model's output changes during training. Click a media panel to view an image in full-screen mode; there you can zoom and pan, including with [keyboard shortcuts](/models/app/keyboard-shortcuts#media-panels).

To compare images or videos from different runs, steps, or indices in one view, use [Compare mode](/models/app/features/panels/media#compare-mode) in a media panel.

<Note>
  It's recommended to log fewer than 50 images per step to prevent logging from becoming a bottleneck during training and image loading from becoming a bottleneck when viewing results.
</Note>

<Tabs>
  <Tab title="Logging arrays as Images">
    Provide arrays directly when constructing images manually, such as by using [`make_grid` from `torchvision`](https://pytorch.org/vision/stable/utils.html#torchvision.utils.make_grid).

    Arrays are converted to png using [Pillow](https://pillow.readthedocs.io/en/stable/index.html).

    ```python theme={null}
    import wandb

    with wandb.init(project="image-log-example") as run:

        images = wandb.Image(image_array, caption="Top: Output, Bottom: Input")

        run.log({"examples": images})
    ```

    We assume the image is gray scale if the last dimension is 1, RGB if it's 3, and RGBA if it's 4. If the array contains floats, we convert them to integers between `0` and `255`. If you want to normalize your images differently, you can specify the [`mode`](https://pillow.readthedocs.io/en/stable/handbook/concepts.html#modes) manually or just supply a [`PIL.Image`](https://pillow.readthedocs.io/en/stable/reference/Image.html), as described in the "Logging PIL Images" tab of this panel.
  </Tab>

  <Tab title="Logging PIL Images">
    For full control over the conversion of arrays to images, construct the [`PIL.Image`](https://pillow.readthedocs.io/en/stable/reference/Image.html) yourself and provide it directly.

    ```python theme={null}
    from PIL import Image

    with wandb.init(project="") as run:
        # Create a PIL image from a NumPy array
        image = Image.fromarray(image_array)

        # Optionally, convert to RGB if needed
        if image.mode != "RGB":
            image = image.convert("RGB")

        # Log the image
        run.log({"example": wandb.Image(image, caption="My Image")})
    ```
  </Tab>

  <Tab title="Logging Images from Files">
    For even more control, create images however you like, save them to disk, and provide a filepath.

    ```python theme={null}
    import wandb
    from PIL import Image

    with wandb.init(project="") as run:

        im = Image.fromarray(...)
        rgb_im = im.convert("RGB")
        rgb_im.save("myimage.jpg")

        run.log({"example": wandb.Image("myimage.jpg")})
    ```
  </Tab>
</Tabs>

## Image overlays

<Tabs>
  <Tab title="Segmentation Masks">
    Log semantic segmentation masks and interact with them (altering opacity, viewing changes over time, and more) via the W\&B UI.

    <Frame>
      <img alt="Interactive mask viewing" />
    </Frame>

    To log an overlay, provide a dictionary with the following keys and values to the `masks` keyword argument of `wandb.Image`:

    * one of two keys representing the image mask:
      * `"mask_data"`: a 2D NumPy array containing an integer class label for each pixel
      * `"path"`: (string) a path to a saved image mask file
    * `"class_labels"`: (optional) a dictionary mapping the integer class labels in the image mask to their readable class names

    To log multiple masks, log a mask dictionary with multiple keys, as in the code snippet below.

    [See a live example](https://app.wandb.ai/stacey/deep-drive/reports/Image-Masks-for-Semantic-Segmentation--Vmlldzo4MTUwMw)

    [Sample code](https://colab.research.google.com/drive/1SOVl3EvW82Q4QKJXX6JtHye4wFix_P4J)

    ```python theme={null}
    mask_data = np.array([[1, 2, 2, ..., 2, 2, 1], ...])

    class_labels = {1: "tree", 2: "car", 3: "road"}

    mask_img = wandb.Image(
        image,
        masks={
            "predictions": {"mask_data": mask_data, "class_labels": class_labels},
            "ground_truth": {
                # ...
            },
            # ...
        },
    )
    ```

    Segmentation masks for a key are defined at each step (each call to `run.log()`).

    * If steps provide different values for the same mask key, only the most recent value for the key is applied to the image.
    * If steps provide different mask keys, all values for each key are shown, but only those defined in the step being viewed are applied to the image. Toggling the visibility of masks not defined in the step do not change the image.
  </Tab>

  <Tab title="Bounding Boxes">
    Log bounding boxes with images, and use filters and toggles to dynamically visualize different sets of boxes in the UI.

    <Frame>
      <img alt="Bounding box example" />
    </Frame>

    [See a live example](https://app.wandb.ai/stacey/yolo-drive/reports/Bounding-Boxes-for-Object-Detection--Vmlldzo4Nzg4MQ)

    To log a bounding box, you'll need to provide a dictionary with the following keys and values to the boxes keyword argument of `wandb.Image`:

    * `box_data`: a list of dictionaries, one for each box. The box dictionary format is described below.
      * `position`: a dictionary representing the position and size of the box in one of two formats, as described below. Boxes need not all use the same format.
        * *Option 1:* `{"minX", "maxX", "minY", "maxY"}`. Provide a set of coordinates defining the upper and lower bounds of each box dimension.
        * *Option 2:* `{"middle", "width", "height"}`. Provide a set of coordinates specifying the `middle` coordinates as `[x,y]`, and `width` and `height` as scalars.
      * `class_id`: an integer representing the class identity of the box. See `class_labels` key below.
      * `scores`: a dictionary of string labels and numeric values for scores. Can be used for filtering boxes in the UI.
      * `domain`: specify the units/format of the box coordinates. **Set this to "pixel"** if the box coordinates are expressed in pixel space, such as integers within the bounds of the image dimensions. By default, the domain is assumed to be a fraction/percentage of the image, expressed as a floating point number between 0 and 1.
      * `box_caption`: (optional) a string to be displayed as the label text on this box
    * `class_labels`: (optional) A dictionary mapping `class_id`s to strings. By default we will generate class labels `class_0`, `class_1`, etc.

    Check out this example:

    ```python theme={null}
    import wandb

    class_id_to_label = {
        1: "car",
        2: "road",
        3: "building",
        # ...
    }

    img = wandb.Image(
        image,
        boxes={
            "predictions": {
                "box_data": [
                    {
                        # one box expressed in the default relative/fractional domain
                        "position": {"minX": 0.1, "maxX": 0.2, "minY": 0.3, "maxY": 0.4},
                        "class_id": 2,
                        "box_caption": class_id_to_label[2],
                        "scores": {"acc": 0.1, "loss": 1.2},
                        # another box expressed in the pixel domain
                        # (for illustration purposes only, all boxes are likely
                        # to be in the same domain/format)
                        "position": {"middle": [150, 20], "width": 68, "height": 112},
                        "domain": "pixel",
                        "class_id": 3,
                        "box_caption": "a building",
                        "scores": {"acc": 0.5, "loss": 0.7},
                        # ...
                        # Log as many boxes an as needed
                    }
                ],
                "class_labels": class_id_to_label,
            },
            # Log each meaningful group of boxes with a unique key name
            "ground_truth": {
                # ...
            },
        },
    )

    with wandb.init(project="my_project") as run:
        run.log({"driving_scene": img})
    ```
  </Tab>
</Tabs>

## Image overlays in Tables

<Tabs>
  <Tab title="Segmentation Masks">
    <Frame>
      <img alt="Interactive Segmentation Masks in Tables" />
    </Frame>

    To log Segmentation Masks in tables, you will need to provide a `wandb.Image` object for each row in the table.

    An example is provided in the Code snippet below:

    ```python theme={null}
    table = wandb.Table(columns=["ID", "Image"])

    for id, img, label in zip(ids, images, labels):
        mask_img = wandb.Image(
            img,
            masks={
                "prediction": {"mask_data": label, "class_labels": class_labels}
                # ...
            },
        )

        table.add_data(id, mask_img)

    with wandb.init(project="my_project") as run:
        run.log({"Table": table})
    ```
  </Tab>

  <Tab title="Bounding Boxes">
    <Frame>
      <img alt="Interactive Bounding Boxes in Tables" />
    </Frame>

    To log Images with Bounding Boxes in tables, you will need to provide a `wandb.Image` object for each row in the table.

    An example is provided in the code snippet below:

    ```python theme={null}
    table = wandb.Table(columns=["ID", "Image"])

    for id, img, boxes in zip(ids, images, boxes_set):
        box_img = wandb.Image(
            img,
            boxes={
                "prediction": {
                    "box_data": [
                        {
                            "position": {
                                "minX": box["minX"],
                                "minY": box["minY"],
                                "maxX": box["maxX"],
                                "maxY": box["maxY"],
                            },
                            "class_id": box["class_id"],
                            "box_caption": box["caption"],
                            "domain": "pixel",
                        }
                        for box in boxes
                    ],
                    "class_labels": class_labels,
                }
            },
        )
    ```
  </Tab>
</Tabs>

## Histograms

<Tabs>
  <Tab title="Basic Histogram Logging">
    If a sequence of numbers, such as a list, array, or tensor, is provided as the first argument, we will construct the histogram automatically by calling `np.histogram()`. All arrays/tensors are flattened. You can use the optional `num_bins` keyword argument to override the default of `64` bins. The maximum number of bins supported is `512`.

    In the UI, histograms are plotted with the training step on the x-axis, the metric value on the y-axis, and the count represented by color, to ease comparison of histograms logged throughout training. See the "Histograms in Summary" tab of this panel for details on logging one-off histograms.

    ```python theme={null}
    run.log({"gradients": wandb.Histogram(grads)})
    ```

    <Frame>
      <img alt="GAN discriminator gradients" />
    </Frame>
  </Tab>

  <Tab title="Flexible Histogram Logging">
    If you want more control, call `np.histogram()` and pass the returned tuple to the `np_histogram` keyword argument.

    ```python theme={null}
    np_hist_grads = np.histogram(grads, density=True, range=(0.0, 1.0))
    run.log({"gradients": wandb.Histogram(np_hist_grads)})
    ```
  </Tab>
</Tabs>

If histograms are in your summary they will appear on the Overview tab of the [Run Page](/models/runs/). If they are in your history, we plot a heatmap of bins over time on the Charts tab.

## 3D visualizations

Log 3D point clouds and Lidar scenes with bounding boxes. Pass in a NumPy array containing coordinates and colors for the points to render.

```python theme={null}
point_cloud = np.array([[0, 0, 0, COLOR]])

run.log({"point_cloud": wandb.Object3D(point_cloud)})
```

<Note>
  The W\&B UI truncates the data at 300,000 points.
</Note>

#### NumPy array formats

Three different formats of NumPy arrays are supported for flexible color schemes.

* `[[x, y, z], ...]` `nx3`
* `[[x, y, z, c], ...]` `nx4` `| c is a category` in the range `[1, 14]` (Useful for segmentation)
* `[[x, y, z, r, g, b], ...]` `nx6 | r,g,b` are values in the range `[0,255]`for red, green, and blue color channels.

#### Python object

Using this schema, you can define a Python object and pass it in to [the `from_point_cloud` method](/models/ref/python/#from_point_cloud).

* `points`is a NumPy array containing coordinates and colors for the points to render using [the same formats as the simple point cloud renderer shown above](#python-object).
* `boxes` is a NumPy array of python dictionaries with three attributes:
  * `corners`- a list of eight corners
  * `label`- a string representing the label to be rendered on the box (Optional)
  * `color`- rgb values representing the color of the box
  * `score` - a numeric value that will be displayed on the bounding box that can be used to filter the bounding boxes shown (for example, to only show bounding boxes where `score` > `0.75`). (Optional)
* `type` is a string representing the scene type to render. Currently the only supported value is `lidar/beta`

```python theme={null}
point_list = [
    [
        2566.571924017235, # x
        746.7817289698219, # y
        -15.269245470863748,# z
        76.5, # red
        127.5, # green
        89.46617199365393 # blue
    ],
    [ 2566.592983606823, 746.6791987335685, -15.275803826279521, 76.5, 127.5, 89.45471117247024 ],
    [ 2566.616361739416, 746.4903185513501, -15.28628929674075, 76.5, 127.5, 89.41336375503832 ],
    [ 2561.706014951675, 744.5349468458361, -14.877496818222781, 76.5, 127.5, 82.21868245418283 ],
    [ 2561.5281847916694, 744.2546118233013, -14.867862032341005, 76.5, 127.5, 81.87824684536432 ],
    [ 2561.3693562897465, 744.1804761656741, -14.854129178142523, 76.5, 127.5, 81.64137897587152 ],
    [ 2561.6093071504515, 744.0287526628543, -14.882135189841177, 76.5, 127.5, 81.89871499537098 ],
    # ... and so on
]

run.log({"my_first_point_cloud": wandb.Object3D.from_point_cloud(
     points = point_list,
     boxes = [{
         "corners": [
                [ 2601.2765123137915, 767.5669506323393, -17.816764802288663 ],
                [ 2599.7259021588347, 769.0082337923552, -17.816764802288663 ],
                [ 2599.7259021588347, 769.0082337923552, -19.66876480228866 ],
                [ 2601.2765123137915, 767.5669506323393, -19.66876480228866 ],
                [ 2604.8684867834395, 771.4313904894723, -17.816764802288663 ],
                [ 2603.3178766284827, 772.8726736494882, -17.816764802288663 ],
                [ 2603.3178766284827, 772.8726736494882, -19.66876480228866 ],
                [ 2604.8684867834395, 771.4313904894723, -19.66876480228866 ]
        ],
         "color": [0, 0, 255], # color in RGB of the bounding box
         "label": "car", # string displayed on the bounding box
         "score": 0.6 # numeric displayed on the bounding box
     }],
     vectors = [
        {"start": [0, 0, 0], "end": [0.1, 0.2, 0.5], "color": [255, 0, 0]}, # color is optional
     ],
     point_cloud_type = "lidar/beta",
)})
```

When viewing a point cloud, you can hold control and use the mouse to move around inside the space.

#### Point cloud files

You can use [the `from_file` method](/models/ref/python/#from_file) to load in a JSON file full of point cloud data.

```python theme={null}
run.log({"my_cloud_from_file": wandb.Object3D.from_file(
     "./my_point_cloud.pts.json"
)})
```

An example of how to format the point cloud data is shown below.

```json theme={null}
{
    "boxes": [
        {
            "color": [
                0,
                255,
                0
            ],
            "score": 0.35,
            "label": "My label",
            "corners": [
                [
                    2589.695869075582,
                    760.7400443552185,
                    -18.044831294622487
                ],
                [
                    2590.719039645323,
                    762.3871153874499,
                    -18.044831294622487
                ],
                [
                    2590.719039645323,
                    762.3871153874499,
                    -19.54083129462249
                ],
                [
                    2589.695869075582,
                    760.7400443552185,
                    -19.54083129462249
                ],
                [
                    2594.9666662674313,
                    757.4657929961453,
                    -18.044831294622487
                ],
                [
                    2595.9898368371723,
                    759.1128640283766,
                    -18.044831294622487
                ],
                [
                    2595.9898368371723,
                    759.1128640283766,
                    -19.54083129462249
                ],
                [
                    2594.9666662674313,
                    757.4657929961453,
                    -19.54083129462249
                ]
            ]
        }
    ],
    "points": [
        [
            2566.571924017235,
            746.7817289698219,
            -15.269245470863748,
            76.5,
            127.5,
            89.46617199365393
        ],
        [
            2566.592983606823,
            746.6791987335685,
            -15.275803826279521,
            76.5,
            127.5,
            89.45471117247024
        ],
        [
            2566.616361739416,
            746.4903185513501,
            -15.28628929674075,
            76.5,
            127.5,
            89.41336375503832
        ]
    ],
    "type": "lidar/beta"
}
```

#### NumPy arrays

Using [the same array formats defined above](#numpy-array-formats), you can use `numpy` arrays directly with [the `from_numpy` method](/models/ref/python/#from_numpy) to define a point cloud.

```python theme={null}
run.log({"my_cloud_from_numpy_xyz": wandb.Object3D.from_numpy(
     np.array(  
        [
            [0.4, 1, 1.3], # x, y, z
            [1, 1, 1], 
            [1.2, 1, 1.2]
        ]
    )
)})
```

```python theme={null}
run.log({"my_cloud_from_numpy_cat": wandb.Object3D.from_numpy(
     np.array(  
        [
            [0.4, 1, 1.3, 1], # x, y, z, category 
            [1, 1, 1, 1], 
            [1.2, 1, 1.2, 12], 
            [1.2, 1, 1.3, 12], 
            [1.2, 1, 1.4, 12], 
            [1.2, 1, 1.5, 12], 
            [1.2, 1, 1.6, 11], 
            [1.2, 1, 1.7, 11], 
        ]
    )
)})
```

```python theme={null}
run.log({"my_cloud_from_numpy_rgb": wandb.Object3D.from_numpy(
     np.array(  
        [
            [0.4, 1, 1.3, 255, 0, 0], # x, y, z, r, g, b 
            [1, 1, 1, 0, 255, 0], 
            [1.2, 1, 1.3, 0, 255, 255],
            [1.2, 1, 1.4, 0, 255, 255],
            [1.2, 1, 1.5, 0, 0, 255],
            [1.2, 1, 1.1, 0, 0, 255],
            [1.2, 1, 0.9, 0, 0, 255],
        ]
    )
)})
```

```python theme={null}
run.log({"protein": wandb.Molecule("6lu7.pdb")})
```

Log molecular data in any of 10 file types:`pdb`, `pqr`, `mmcif`, `mcif`, `cif`, `sdf`, `sd`, `gro`, `mol2`, or `mmtf.`

W\&B also supports logging molecular data from SMILES strings, [`rdkit`](https://www.rdkit.org/docs/index.html) `mol` files, and `rdkit.Chem.rdchem.Mol` objects.

```python theme={null}
resveratrol = rdkit.Chem.MolFromSmiles("Oc1ccc(cc1)C=Cc1cc(O)cc(c1)O")

run.log(
    {
        "resveratrol": wandb.Molecule.from_rdkit(resveratrol),
        "green fluorescent protein": wandb.Molecule.from_rdkit("2b3p.mol"),
        "acetaminophen": wandb.Molecule.from_smiles("CC(=O)Nc1ccc(O)cc1"),
    }
)
```

When your run finishes, you'll be able to interact with 3D visualizations of your molecules in the UI.

[See a live example using AlphaFold](https://wandb.me/alphafold-workspace)

<Frame>
  <img alt="Molecule structure" />
</Frame>

### PNG image

[`wandb.Image`](/models/ref/python/data-types/image) converts `numpy` arrays or instances of `PILImage` to PNGs by default.

```python theme={null}
run.log({"example": wandb.Image(...)})
# Or multiple images
run.log({"example": [wandb.Image(...) for img in images]})
```

### Video

Videos are logged using the [`wandb.Video`](/models/ref/python/) data type:

```python theme={null}
run.log({"example": wandb.Video("myvideo.mp4")})
```

Now you can view videos in the media browser. Go to your project workspace, run workspace, or report and click **Add visualization** to add a rich media panel.

## 2D view of a molecule

You can log a 2D view of a molecule using the [`wandb.Image`](/models/ref/python/data-types/image) data type and [`rdkit`](https://www.rdkit.org/docs/index.html):

```python theme={null}
molecule = rdkit.Chem.MolFromSmiles("CC(=O)O")
rdkit.Chem.AllChem.Compute2DCoords(molecule)
rdkit.Chem.AllChem.GenerateDepictionMatching2DStructure(molecule, molecule)
pil_image = rdkit.Chem.Draw.MolToImage(molecule, size=(300, 300))

run.log({"acetic_acid": wandb.Image(pil_image)})
```

## Other media

W\&B also supports logging of a variety of other media types.

### Audio

```python theme={null}
run.log({"whale songs": wandb.Audio(np_array, caption="OooOoo", sample_rate=32)})
```

A maximum of 100 audio clips can be logged per step. For more usage information, see [`audio-file`](/models/ref/query-panel/audio-file).

### Video

```python theme={null}
run.log({"video": wandb.Video(numpy_array_or_path_to_video, fps=4, format="gif")})
```

If a numpy array is supplied we assume the dimensions are, in order: time, channels, width, height. By default we create a 4 fps gif image ([`ffmpeg`](https://www.ffmpeg.org) and the [`moviepy`](https://pypi.org/project/moviepy/) python library are required when passing numpy objects). Supported formats are `"gif"`, `"mp4"`, `"webm"`, and `"ogg"`. If you pass a string to `wandb.Video` we assert the file exists and is a supported format before uploading to wandb. Passing a `BytesIO` object will create a temporary file with the specified format as the extension.

On the W\&B [Run](/models/runs/) and [Project](/models/track/project-page/) Pages, you will see your videos in the Media section.

For more usage information, see [`video-file`](/models/ref/query-panel/video-file).

### Text

Use `wandb.Table` to log text in tables to show up in the UI. By default, the column headers are `["Input", "Output", "Expected"]`. To ensure optimal UI performance, the default maximum number of rows is set to 10,000. However, you can explicitly override the maximum with `wandb.Table.MAX_ROWS = {DESIRED_MAX}`.

```python theme={null}
with wandb.init(project="my_project") as run:
    columns = ["Text", "Predicted Sentiment", "True Sentiment"]
    # Method 1
    data = [["I love my phone", "1", "1"], ["My phone sucks", "0", "-1"]]
    table = wandb.Table(data=data, columns=columns)
    run.log({"examples": table})

    # Method 2
    table = wandb.Table(columns=columns)
    table.add_data("I love my phone", "1", "1")
    table.add_data("My phone sucks", "0", "-1")
    run.log({"examples": table})
```

You can also pass a pandas `DataFrame` object.

```python theme={null}
table = wandb.Table(dataframe=my_dataframe)
```

For more usage information, see [`string`](/models/ref/query-panel/).

### HTML

```python theme={null}
run.log({"custom_file": wandb.Html(open("some.html"))})
run.log({"custom_string": wandb.Html('<a href="https://mysite">Link</a>')})
```

Log custom HTML at any key to expose an HTML panel on the run page. By default, we inject default styles; you can turn off default styles by passing `inject=False`.

```python theme={null}
run.log({"custom_file": wandb.Html(open("some.html"), inject=False)})
```

For more usage information, see [`html-file`](/models/ref/query-panel/html-file).


# Create and track plots from experiments
Source: https://docs.wandb.ai/models/track/log/plots

Create and track plots from machine learning experiments.

In W\&B Models, methods in `wandb.plot` let you track charts with `wandb.Run.log()`, including charts that change over time during training. To learn more about the custom charting framework, see the [custom charts walkthrough](/models/app/features/custom-charts/walkthrough/).

### Basic charts

To create a W\&B chart:

1. Create a `wandb.Table` object and add the data you want to visualize.
2. Generate a plot using one of the W\&B's built-in [helper functions](/models/ref/python/custom-charts)
3. Log the plot with `wandb.Run.log()`.

The following basic charts can be used to construct basic visualizations of metrics and results.

<Tabs>
  <Tab title="Line">
    Log a custom line plot, a list of connected and ordered points on arbitrary axes.

    ```python theme={null}
    import wandb

    with wandb.init() as run:
        data = [[x, y] for (x, y) in zip(x_values, y_values)]
        table = wandb.Table(data=data, columns=["x", "y"])
        run.log(
            {
                "my_custom_plot_id": wandb.plot.line(
                    table, "x", "y", title="Custom Y versus X line plot"
                )
            }
        )
    ```

    You can use this to log curves on any two dimensions. If you're plotting two lists of values against each other, the number of values in the lists must match exactly. For example, each point must have an x and a y.

    <Frame>
      <img alt="Custom line plot" />
    </Frame>

    For more information, see the [Creating Custom Line Plots With W\&B](https://wandb.ai/wandb/plots/reports/Custom-Line-Plots--VmlldzoyNjk5NTA) report.

    [Run the code](https://tiny.cc/custom-charts)
  </Tab>

  <Tab title="Scatter">
    Log a custom scatter plot, a list of points (x, y) on a pair of arbitrary axes x and y.

    ```python theme={null}
    import wandb

    with wandb.init() as run:
        data = [[x, y] for (x, y) in zip(class_x_scores, class_y_scores)]
        table = wandb.Table(data=data, columns=["class_x", "class_y"])
        run.log({"my_custom_id": wandb.plot.scatter(table, "class_x", "class_y")})
    ```

    You can use this to log scatter points on any two dimensions. If you're plotting two lists of values against each other, the number of values in the lists must match exactly. For example, each point must have an x and a y.

    <Frame>
      <img alt="Custom scatter plot" />
    </Frame>

    For more information, see the [Creating Custom Scatter Plots With W\&B](https://wandb.ai/wandb/plots/reports/Custom-Scatter-Plots--VmlldzoyNjk5NDQ) report.

    [Run the code](https://tiny.cc/custom-charts)
  </Tab>

  <Tab title="Bar">
    Log a custom bar chart (a list of labeled values as bars) natively in a few lines:

    ```python theme={null}
    import wandb

    with wandb.init() as run:
        data = [[label, val] for (label, val) in zip(labels, values)]
        table = wandb.Table(data=data, columns=["label", "value"])
        run.log(
            {
                "my_bar_chart_id": wandb.plot.bar(
                    table, "label", "value", title="Custom bar chart"
                )
            }
        )
    ```

    You can use this to log arbitrary bar charts. The number of labels and values in the lists must match exactly. Each data point must have both.

    <Frame>
      <img alt="Custom bar chart" />
    </Frame>

    For more information, see the [Custom Bar Charts](https://wandb.ai/wandb/plots/reports/Custom-Bar-Charts--VmlldzoyNzExNzk) report.

    [Run the code](https://tiny.cc/custom-charts)
  </Tab>

  <Tab title="Histogram">
    Log a custom histogram (sort a list of values into bins by count or frequency of occurrence) natively in a few lines. If you have a list of prediction confidence scores (`scores`), you can visualize the distribution like this:

    ```python theme={null}
    import wandb

    with wandb.init() as run:
        data = [[s] for s in scores]
        table = wandb.Table(data=data, columns=["scores"])
        run.log({"my_histogram": wandb.plot.histogram(table, "scores", title="Histogram")})
    ```

    You can use this to log arbitrary histograms. Note that `data` is a list of lists, intended to support a 2D array of rows and columns.

    <Frame>
      <img alt="Custom histogram" />
    </Frame>

    For more information, see the [Creating Custom Histograms With W\&B](https://wandb.ai/wandb/plots/reports/Custom-Histograms--VmlldzoyNzE0NzM) report.

    [Run the code](https://tiny.cc/custom-charts)
  </Tab>

  <Tab title="Multi-line">
    Plot multiple lines, or multiple different lists of x-y coordinate pairs, on one shared set of x-y axes:

    ```python theme={null}
    import wandb

    with wandb.init() as run:
        run.log(
            {
                "my_custom_id": wandb.plot.line_series(
                    xs=[0, 1, 2, 3, 4],
                    ys=[[10, 20, 30, 40, 50], [0.5, 11, 72, 3, 41]],
                    keys=["metric Y", "metric Z"],
                    title="Two Random Metrics",
                    xname="x units",
                )
            }
        )
    ```

    Note that the number of x and y points must match exactly. You can supply one list of x values to match multiple lists of y values, or a separate list of x values for each list of y values.

    <Frame>
      <img alt="Multi-line plot" />
    </Frame>

    For more information, see the [Custom Multi-Line Plots](https://wandb.ai/wandb/plots/reports/Custom-Multi-Line-Plots--VmlldzozOTMwMjU) report.
  </Tab>
</Tabs>

### Model evaluation charts

These preset charts have built-in `wandb.plot()` methods that make it quick to log charts directly from your script and see the exact information you're looking for in the UI.

<Tabs>
  <Tab title="Precision-recall curves">
    Create a [Precision-Recall curve](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.precision_recall_curve.html#sklearn.metrics.precision_recall_curve) in one line:

    ```python theme={null}
    import wandb
    with wandb.init() as run:
        # ground_truth is a list of true labels, predictions is a list of predicted scores.
        # For example ground_truth = [0, 1, 1, 0], predictions = [0.1, 0.4, 0.35, 0.8]
        ground_truth = [0, 1, 1, 0]
        predictions = [0.1, 0.4, 0.35, 0.8]
        run.log({"pr": wandb.plot.pr_curve(ground_truth, predictions)})
    ```

    You can log this whenever your code has access to:

    * A model's predicted scores (`predictions`) on a set of examples.
    * The corresponding ground truth labels (`ground_truth`) for those examples.
    * (Optional) A list of the labels or class names. For example, `labels=["cat", "dog", "bird"]`, if label index 0 means cat, 1 means dog, 2 means bird.
    * (Optional) A subset (still in list format) of the labels to visualize in the plot.

    <Frame>
      <img alt="Precision-recall curve" />
    </Frame>

    For more information, see the [Plot Precision Recall Curves With W\&B](https://wandb.ai/wandb/plots/reports/Plot-Precision-Recall-Curves--VmlldzoyNjk1ODY) report.

    [Run the code](https://colab.research.google.com/drive/1mS8ogA3LcZWOXchfJoMrboW3opY1A8BY?usp=sharing)
  </Tab>

  <Tab title="ROC curves">
    Create an [ROC curve](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.roc_curve.html#sklearn.metrics.roc_curve) in one line:

    ```python theme={null}
    import wandb

    with wandb.init() as run:
        # ground_truth is a list of true labels, predictions is a list of predicted scores.
        # For example ground_truth = [0, 1, 1, 0], predictions = [0.1, 0.4, 0.35, 0.8]
        ground_truth = [0, 1, 1, 0]
        predictions = [0.1, 0.4, 0.35, 0.8]
        run.log({"roc": wandb.plot.roc_curve(ground_truth, predictions)})
    ```

    You can log this whenever your code has access to:

    * A model's predicted scores (`predictions`) on a set of examples.
    * The corresponding ground truth labels (`ground_truth`) for those examples.
    * (Optional) A list of the labels or class names. For example, `labels=["cat", "dog", "bird"]`, if label index 0 means cat, 1 means dog, 2 means bird.
    * (Optional) A subset (still in list format) of these labels to visualize on the plot.

    <Frame>
      <img alt="ROC curve" />
    </Frame>

    For more information, see the [Plot ROC Curves With W\&B](https://wandb.ai/wandb/plots/reports/Plot-ROC-Curves--VmlldzoyNjk3MDE) report.

    [Run the code](https://colab.research.google.com/github/wandb/examples/blob/master/colabs/wandb-log/Plot_ROC_Curves_with_W%26B.ipynb)
  </Tab>

  <Tab title="Confusion matrix">
    Create a multi-class [confusion matrix](https://scikit-learn.org/stable/auto_examples/model_selection/plot_confusion_matrix.html) in one line:

    ```python theme={null}
    import wandb

    cm = wandb.plot.confusion_matrix(
        y_true=ground_truth, preds=predictions, class_names=class_names
    )

    with wandb.init() as run:
        run.log({"conf_mat": cm})
    ```

    You can log this wherever your code has access to:

    * A model's predicted labels on a set of examples (`preds`) or the normalized probability scores (`probs`). The probabilities must have the shape (number of examples, number of classes). You can supply either probabilities or predictions but not both.
    * The corresponding ground truth labels for those examples (`y_true`).
    * A full list of the labels or class names as strings in `class_names`. For example, `class_names=["cat", "dog", "bird"]`, if index 0 is `cat`, 1 is `dog`, 2 is `bird`.

    <Frame>
      <img alt="Confusion matrix" />
    </Frame>

    For more information, see the [Confusion Matrix: Usage and Examples](https://wandb.ai/wandb/plots/reports/Confusion-Matrix--VmlldzozMDg1NTM) report.

    [Run the code](https://colab.research.google.com/github/wandb/examples/blob/master/colabs/wandb-log/Log_a_Confusion_Matrix_with_W%26B.ipynb)
  </Tab>
</Tabs>

### Interactive custom charts

For full customization, tweak a built-in [Custom Chart preset](/models/app/features/custom-charts/walkthrough/) or create a new preset, then save the chart. Use the chart ID to log data to that custom preset directly from your script.

```python theme={null}
import wandb
# Create a table with the columns to plot.
table = wandb.Table(data=data, columns=["step", "height"])

# Map from the table's columns to the chart's fields.
fields = {"x": "step", "value": "height"}

# Use the table to populate the new custom chart preset.
# To use your own saved chart preset, change the vega_spec_name.
# To edit the title, change the string_fields.
my_custom_chart = wandb.plot_table(
    vega_spec_name="carey/new_chart",
    data_table=table,
    fields=fields,
    string_fields={"title": "Height Histogram"},
)

with wandb.init() as run:
    # Log the custom chart.
    run.log({"my_custom_chart": my_custom_chart})
```

[Run the code](https://tiny.cc/custom-charts)

### Matplotlib and Plotly plots

Instead of using W\&B [Custom Charts](/models/app/features/custom-charts/walkthrough/) with `wandb.plot()`, you can log charts generated with [matplotlib](https://matplotlib.org/) and [Plotly](https://plotly.com/).

```python theme={null}
import wandb
import matplotlib.pyplot as plt

with wandb.init() as run:
    # Create a simple matplotlib plot.
    plt.figure()
    plt.plot([1, 2, 3, 4])
    plt.ylabel("some interesting numbers")

    # Log the plot to W&B.
    run.log({"chart": plt})
```

Pass a `matplotlib` plot or figure object to `wandb.Run.log()`. By default we'll convert the plot into a [Plotly](https://plot.ly/) plot. If you'd rather log the plot as an image, you can pass the plot into `wandb.Image`. We also accept Plotly charts directly.

<Note>
  If you get an error like "You attempted to log an empty plot", store the figure separately from the plot with `fig = plt.figure()` and then log `fig` in your call to `wandb.Run.log()`.
</Note>

### Log custom HTML to W\&B Tables

W\&B supports logging interactive charts from Plotly and Bokeh as HTML and adding them to Tables.

#### Log Plotly figures to Tables as HTML

You can log interactive Plotly charts to W\&B Tables by converting them to HTML.

```python theme={null}
import wandb
import plotly.express as px

# Initialize a new run.
with wandb.init(project="log-plotly-fig-tables", name="plotly_html") as run:

    # Create a table.
    table = wandb.Table(columns=["plotly_figure"])

    # Create path for Plotly figure.
    path_to_plotly_html = "./plotly_figure.html"

    # Example Plotly figure.
    fig = px.scatter(x=[0, 1, 2, 3, 4], y=[0, 1, 4, 9, 16])

    # Write Plotly figure to HTML.
    # Set auto_play to False prevents animated Plotly charts
    # from playing in the table automatically.
    fig.write_html(path_to_plotly_html, auto_play=False)

    # Add Plotly figure as HTML file into Table.
    table.add_data(wandb.Html(path_to_plotly_html))

    # Log table.
    run.log({"test_table": table})
```

#### Log Bokeh figures to Tables as HTML

You can log interactive Bokeh charts to W\&B Tables by converting them to HTML.

```python theme={null}
from scipy.signal import spectrogram
import holoviews as hv
import panel as pn
from scipy.io import wavfile
import numpy as np
from bokeh.resources import INLINE

hv.extension("bokeh", logo=False)
import wandb


def save_audio_with_bokeh_plot_to_html(audio_path, html_file_name):
    sr, wav_data = wavfile.read(audio_path)
    duration = len(wav_data) / sr
    f, t, sxx = spectrogram(wav_data, sr)
    spec_gram = hv.Image((t, f, np.log10(sxx)), ["Time (s)", "Frequency (hz)"]).opts(
        width=500, height=150, labelled=[]
    )
    audio = pn.pane.Audio(wav_data, sample_rate=sr, name="Audio", throttle=500)
    slider = pn.widgets.FloatSlider(end=duration, visible=False)
    line = hv.VLine(0).opts(color="white")
    slider.jslink(audio, value="time", bidirectional=True)
    slider.jslink(line, value="glyph.location")
    combined = pn.Row(audio, spec_gram * line, slider).save(html_file_name)


html_file_name = "audio_with_plot.html"
audio_path = "hello.wav"
save_audio_with_bokeh_plot_to_html(audio_path, html_file_name)

wandb_html = wandb.Html(html_file_name)

with wandb.init(project="audio_test") as run:
    my_table = wandb.Table(columns=["audio_with_plot"], data=[[wandb_html]])
    run.log({"audio_table": my_table})
```


# Track CSV files with experiments
Source: https://docs.wandb.ai/models/track/log/working-with-csv

Import CSV files into W&B as Tables and Artifacts for visualization, comparison, and analysis in dashboards.

Use the W\&B Python Library to log a CSV file and visualize it in a [W\&B Dashboard](/models/track/workspaces/). W\&B Dashboard are the central place to organize and visualize results from your machine learning models. This is particularly useful if you have a [CSV file that contains information of previous machine learning experiments](#import-and-log-your-csv-of-experiments) that are not logged in W\&B or if you have [CSV file that contains a dataset](#import-and-log-your-dataset-csv-file).

## Import and log your dataset CSV file

We suggest you use W\&B Artifacts to make the contents of the CSV file easier to re-use.

1. To get started, first import your CSV file. In the following code snippet, replace the `iris.csv` filename with the name of your CSV filename:

```python theme={null}
import wandb
import pandas as pd

# Read our CSV into a new DataFrame
new_iris_dataframe = pd.read_csv("iris.csv")
```

2. Convert the CSV file to a W\&B Table to utilize [W\&B Dashboards](/models/track/workspaces/).

```python theme={null}
# Convert the DataFrame into a W&B Table
iris_table = wandb.Table(dataframe=new_iris_dataframe)
```

3. Next, create a W\&B Artifact and add the table to the Artifact:

```python theme={null}
# Add the table to an Artifact to increase the row
# limit to 200000 and make it easier to reuse
iris_table_artifact = wandb.Artifact("iris_artifact", type="dataset")
iris_table_artifact.add(iris_table, "iris_table")

# Log the raw csv file within an artifact to preserve our data
iris_table_artifact.add_file("iris.csv")
```

For more information about W\&B Artifacts, see the [Artifacts chapter](/models/artifacts/).

4. Lastly, start a new W\&B Run to track and log to W\&B with `wandb.init()`:

```python theme={null}
# Start a W&B run to log data
with wandb.init(project="tables-walkthrough") as run:

    # Log the table to visualize with a run...
    run.log({"iris": iris_table})

    # and Log as an Artifact to increase the available row limit!
    run.log_artifact(iris_table_artifact)
```

The `wandb.init()` API spawns a new background process to log data to a Run, and it synchronizes data to wandb.ai (by default). View live visualizations on your W\&B Workspace Dashboard. The following image demonstrates the output of the code snippet demonstration.

<Frame>
  <img alt="CSV file imported into W&B Dashboard" />
</Frame>

The full script with the preceding code snippets is found below:

```python theme={null}
import wandb
import pandas as pd

# Read our CSV into a new DataFrame
new_iris_dataframe = pd.read_csv("iris.csv")

# Convert the DataFrame into a W&B Table
iris_table = wandb.Table(dataframe=new_iris_dataframe)

# Add the table to an Artifact to increase the row
# limit to 200000 and make it easier to reuse
iris_table_artifact = wandb.Artifact("iris_artifact", type="dataset")
iris_table_artifact.add(iris_table, "iris_table")

# log the raw csv file within an artifact to preserve our data
iris_table_artifact.add_file("iris.csv")

# Start a W&B run to log data
with wandb.init(project="tables-walkthrough") as run:

    # Log the table to visualize with a run...
    run.log({"iris": iris_table})

    # and Log as an Artifact to increase the available row limit!
    run.log_artifact(iris_table_artifact)
```

## Import and log your CSV of Experiments

In some cases, you might have your experiment details in a CSV file. Common details found in such CSV files include:

* A name for the experiment run
* Initial [notes](/models/runs/#add-a-note-to-a-run)
* [Tags](/models/runs/tags/) to differentiate the experiments
* Configurations needed for your experiment (with the added benefit of being able to utilize our [Sweeps Hyperparameter Tuning](/models/sweeps/)).

| Experiment   | Model Name       | Notes                                            | Tags          | Num Layers | Final Train Acc | Final Val Acc | Training Losses                       |
| ------------ | ---------------- | ------------------------------------------------ | ------------- | ---------- | --------------- | ------------- | ------------------------------------- |
| Experiment 1 | mnist-300-layers | Overfit way too much on training data            | \[latest]     | 300        | 0.99            | 0.90          | \[0.55, 0.45, 0.44, 0.42, 0.40, 0.39] |
| Experiment 2 | mnist-250-layers | Current best model                               | \[prod, best] | 250        | 0.95            | 0.96          | \[0.55, 0.45, 0.44, 0.42, 0.40, 0.39] |
| Experiment 3 | mnist-200-layers | Did worse than the baseline model. Need to debug | \[debug]      | 200        | 0.76            | 0.70          | \[0.55, 0.45, 0.44, 0.42, 0.40, 0.39] |
| ...          | ...              | ...                                              | ...           | ...        | ...             | ...           |                                       |
| Experiment N | mnist-X-layers   | NOTES                                            | ...           | ...        | ...             | ...           | \[..., ...]                           |

W\&B can take CSV files of experiments and convert it into a W\&B Experiment Run. The following code snippets and code script demonstrates how to import and log your CSV file of experiments:

1. To get started, first read in your CSV file and convert it into a Pandas DataFrame. Replace `"experiments.csv"` with the name of your CSV file:

```python theme={null}
import wandb
import pandas as pd

FILENAME = "experiments.csv"
loaded_experiment_df = pd.read_csv(FILENAME)

PROJECT_NAME = "Converted Experiments"

EXPERIMENT_NAME_COL = "Experiment"
NOTES_COL = "Notes"
TAGS_COL = "Tags"
CONFIG_COLS = ["Num Layers"]
SUMMARY_COLS = ["Final Train Acc", "Final Val Acc"]
METRIC_COLS = ["Training Losses"]

# Format Pandas DataFrame to make it easier to work with
for i, row in loaded_experiment_df.iterrows():
    run_name = row[EXPERIMENT_NAME_COL]
    notes = row[NOTES_COL]
    tags = row[TAGS_COL]

    config = {}
    for config_col in CONFIG_COLS:
        config[config_col] = row[config_col]

    metrics = {}
    for metric_col in METRIC_COLS:
        metrics[metric_col] = row[metric_col]

    summaries = {}
    for summary_col in SUMMARY_COLS:
        summaries[summary_col] = row[summary_col]
```

2. Next, start a new W\&B Run to track and log to W\&B with [`wandb.init()`](/models/ref/python/functions/init):

   ```python theme={null}
   with wandb.init(
       project=PROJECT_NAME, name=run_name, tags=tags, notes=notes, config=config
   ) as run:
   ```

As an experiment runs, you might want to log every instance of your metrics so they are available to view, query, and analyze with W\&B. Use the [`run.log()`](/models/ref/python/experiments/run/#method-runlog) command to accomplish this:

```python theme={null}
run.log({key: val})
```

You can optionally log a final summary metric to define the outcome of the run using the [`define_metric`](/models/ref/python/experiments/run#define_metric) API. This example adds the summary metrics to our run with `run.summary.update()`:

```python theme={null}
run.summary.update(summaries)
```

For more information about summary metrics, see [Log Summary Metrics](./log-summary).

Below is the full example script that converts the above sample table into a [W\&B Dashboard](/models/track/workspaces/):

```python theme={null}
FILENAME = "experiments.csv"
loaded_experiment_df = pd.read_csv(FILENAME)

PROJECT_NAME = "Converted Experiments"

EXPERIMENT_NAME_COL = "Experiment"
NOTES_COL = "Notes"
TAGS_COL = "Tags"
CONFIG_COLS = ["Num Layers"]
SUMMARY_COLS = ["Final Train Acc", "Final Val Acc"]
METRIC_COLS = ["Training Losses"]

for i, row in loaded_experiment_df.iterrows():
    run_name = row[EXPERIMENT_NAME_COL]
    notes = row[NOTES_COL]
    tags = row[TAGS_COL]

    config = {}
    for config_col in CONFIG_COLS:
        config[config_col] = row[config_col]

    metrics = {}
    for metric_col in METRIC_COLS:
        metrics[metric_col] = row[metric_col]

    summaries = {}
    for summary_col in SUMMARY_COLS:
        summaries[summary_col] = row[summary_col]

    with  wandb.init(
        project=PROJECT_NAME, name=run_name, tags=tags, notes=notes, config=config
    ) as run:

        for key, val in metrics.items():
            if isinstance(val, list):
                for _val in val:
                    run.log({key: _val})
            else:
                run.log({key: val})

        run.summary.update(summaries)
```


# Projects
Source: https://docs.wandb.ai/models/track/project-page

Compare versions of your model, explore results in a scratch workspace, and export findings to a report to save notes and visualizations

A *project* is a central location where you visualize results, compare experiments, view and download artifacts, create an automation, and more.

<Note>
  Each project has a visibility setting that determines who can access it. For more information about who can access a project, see [Project visibility](/platform/hosting/iam/access-management/restricted-projects).
</Note>

Each project contains the following tabs:

* [Overview](/models/track/project-page/#overview-tab): snapshot of your project
* [Workspace](/models/track/project-page/#workspace-tab): personal visualization sandbox
* [Runs](#runs-tab): A table that lists all the runs in your project
* [Automations](#automations-tab): Automations configured in your project
* [Sweeps](/models/sweeps): automated exploration and optimization
* [Reports](/models/track/project-page/#reports-tab): saved snapshots of notes, runs, and graphs
* [Artifacts](#artifacts-tab): Contains all runs and the artifacts associated with that run

## Overview tab

W\&B creates a project for you when you initialize a run with the name you provide for the project field.

From the **Overview** tab, you can find the project name and manage the project.

* To change the project's name, description, or team, click **Edit** in the upper right corner.
* To undelete recently deleted runs, click the **action (<Icon icon="ellipsis" />)** menu in the upper right corner, then click **Undelete recently deleted runs**.
* To delete the project, click the **action (<Icon icon="ellipsis" />)** menu in the upper right corner, then click **Delete project**. Read the confirmation dialog and follow the instructions. Deleting a project is not reversible.

The rest of the **Overview** page is divided into **Details** and **Project roles** tabs.

### Details

Details about the project include:

* **Project visibility**: The visibility of the project. The visibility setting that determines who can access it. See [Project visibility](/platform/hosting/iam/access-management/restricted-projects) for more information.
* **Last active**: Timestamp of the last time data is logged to this project
* **Contributors**: The number of users that contribute to this project
* **Total runs**: The total number of runs in this project
* **Total compute**: we add up all the run times in your project to get this total

The **Details** tab also includes instructions for accessing and exporting the project's data and metrics, downloading the best model from a sweep, and more.

[View a live example project](https://wandb.ai/example-team/sweep-demo/overview)

<Frame>
  <img alt="Project overview tab" />
</Frame>

To change a project's privacy from the **Overview** tab:

1. In the W\&B App, from any page in the project, click **Overview** in the left navigation.
2. At the top right, click **Edit**.
3. Choose a new value for **Project visibility**:

   * **Team** (default): Only your team can view and edit the project.
   * **Restricted**: Only invited members can access the project, and public access is turned off.
   * **Open**: Anyone can submit runs or create reports, but only your team can edit it. Appropriate only for classroom settings, public benchmark competitions, or other non-durable contexts.
   * **Public**: Anyone can view the project, but only your team can edit it.

     <Note>
       If your W\&B admins have turned off **Public** visibility, you cannot choose it. Instead, you can share a view-only [W\&B Report](/models/reports/collaborate-on-reports#share-a-report), or contact your W\&B organization's admins for assistance.
     </Note>
4. Click **Save**.

If you update a project to a more strict privacy setting, you may need to re-invite individual users to restore their ability to access the project.

### Project roles

The **Project roles** tab is visible only to the project owner and those with the **Admin** role. List and search for users with access to the project or change a member's role.

## Workspace tab

A project's *workspace* gives you a personal sandbox to compare experiments. Use projects to organize models that can be compared, working on the same problem with different architectures, hyperparameters, datasets, preprocessing etc.

**Runs sidebar**: list of all the runs in your project.

* **action (<Icon icon="ellipsis-vertical" />)** menu: Hover over a row in the sidebar to see the menu appear on the left side. Use this menu to rename a run, delete a run, or stop and active run.
* **Visibility icon**: click the eye to turn on and off runs on graphs
* **Color**: change the run color to another one of our presets or a custom color
* **Search**: search runs by name. This also filters visible runs in the plots.
* **Filter**: use the sidebar filter to narrow down the set of runs visible
* **Group**: select a config column to dynamically group your runs, for example by architecture. Grouping makes plots show up with a line along the mean value, and a shaded region for the variance of points on the graph.
* **Sort**: pick a value to sort your runs by, for example runs with the lowest loss or highest accuracy. Sorting will affect which runs show up on the graphs.
* **Expand button**: expand the sidebar into the full table
* **Minimize**: press **Cmd+.** (macOS) or **Ctrl+.** (Windows/Linux) to collapse or restore the Runs selector. See [Keyboard shortcuts](/models/app/keyboard-shortcuts) for details.
* **Run count**: the number in parentheses at the top is the total number of runs in the project. The number (N visualized) is the number of runs that have the eye turned on and are available to be visualized in each plot. In the example below, the graphs are only showing the first 10 of 183 runs. Edit a graph to increase the max number of runs visible.

If you pin, hide, or change the order of columns in the [Runs tab](#runs-tab), the Runs sidebar reflects these customizations.

**Panels layout**: use this scratch space to explore results, add and remove charts, and compare versions of your models based on different metrics

[View a live example](https://wandb.ai/example-team/sweep-demo)

<Frame>
  <img alt="Project workspace" />
</Frame>

### Add a section of panels

Click the section dropdown menu and click **Add section** to create a new section for panels. You can rename sections, drag them to reorganize them, and expand and collapse sections.

Each section has options in the upper right corner:

* **Add section**: Add a section above or below from the dropdown menu, or click the button at the bottom of the page to add a new section.
* **Rename section**: Change the title for your section.
* **Export section to report**: Save this section of panels to a new report.
* **Delete section**: Remove the whole section and all the charts. This can be undone with the undo button at the bottom of the page in the workspace bar.
* **Add panel**: Click the plus button to add a panel to the section.

### Move panels between sections

Drag and drop panels to reorder and organize into sections. You can also click the **Move** button in the upper right corner of a panel to select a section to move the panel to.

<Frame>
  <img alt="Moving panels between sections" />
</Frame>

### Resize panels

All panels maintain the same size, and there are pages of panels.
Resize the section by clicking and dragging the lower right corner of the section, which will display a corner icon when hovering.

<Frame>
  <img alt="Resizing panels" />
</Frame>

### Search for metrics

Use the search box in the workspace to filter down the panels. This search matches the panel titles, which are by default the name of the metrics visualized.

<Frame>
  <img alt="Workspace search" />
</Frame>

## Runs tab

Use the Runs tab to filter, group, and sort your runs.

<Frame>
  <img alt="Runs table" />
</Frame>

The following tabs demonstrate some common actions you can take in the Runs tab.

<Tabs>
  <Tab title="Customize columns">
    The Runs tab shows details about runs in the project. It shows a large number of columns by default.

    <Note>
      When you customize the Runs tab, the customization is also reflected in the **Runs** selector of the [Workspace tab](#workspace-tab).
    </Note>

    * To view all visible columns, scroll the page horizontally.
    * To change the order of the columns, drag a column to the left or right.
    * To pin a column, hover over the column name, click the **action (<Icon icon="ellipsis-vertical" />)** menu that appears, then click **Pin column**. Pinned columns appear near the left of the page, after the **Name** column. To unpin a pinned column, choose **Unpin column**.
    * To hide a column, hover over the column name, click the **action (<Icon icon="ellipsis-vertical" />)** menu that appears, then click **Hide column**. To view all columns that are currently hidden, click **Columns**.
    * To show, hide, pin, and unpin multiple columns at once, click **Columns**.
      * Click the name of a hidden column to unhide it.
      * Click the name of a visible column to hide it.
      * Click the pin icon next to a visible column to pin it.
  </Tab>

  <Tab title="Sort">
    Sort all rows in a Table by the value in a given column using one of these options:

    1. Click the **action (<Icon icon="ellipsis-vertical" />)** menu when hovering over any column header to quickly sort the table by that column's values.
    2. Click the **Sort** button and select the desired columns from the drop-down. You can use this panel to sort by multiple columns in priority order.

    <Frame>
      <img alt="Configuring a run table sort" />
    </Frame>

    The preceding image demonstrates how to view sorting options for a Table column called `val_acc`.
  </Tab>

  <Tab title="Filter">
    Apply filter expressions to narrow down the runs you want to visualize.

    <Frame>
      <img alt="Configuring a run table filter" />
    </Frame>

    See [Create a filter expression](/models/runs/filter-runs#create-a-filter-expression) for more information on how to write filter expressions.
  </Tab>

  <Tab title="Group">
    Click the **Group** button above the table to group all rows by the value in a particular column. For more information on how to group runs, see [Organize runs into groups](/models/runs/grouping).
  </Tab>
</Tabs>

## Automations tab

Automate downstream actions for versioning artifacts. To create an automation, define trigger events and resulting actions. Actions include executing a webhook or launching a W\&B job. For more information, see [Automations](/models/automations).

## Sweeps tab

Start a new [sweep](/models/sweeps) from your project.

<Frame>
  <img alt="Sweeps tab" />
</Frame>

## Reports tab

See all the snapshots of results in one place, and share findings with your team.

<Frame>
  <img alt="Reports tab" />
</Frame>

To view a report, select the **Reports** tab from the project sidebar, then select a report from the list.

## Artifacts tab

View all [artifacts](/models/artifacts) associated with a project, from training datasets and [fine-tuned models](/models/registry) to [tables of metrics and media](/models/tables/tables-walkthrough).

### Metadata panel

<Frame>
  <img alt="Artifact metadata panel" />
</Frame>

The metadata panel provides access to the artifact's metadata. This metadata might include configuration arguments required to reconstruct the artifact, URLs where more information can be found, or metrics produced during the run which logged the artifact.
View the configuration for the run that produced the artifact as well as history metrics.

To view an artifact's metadata:

1. Select the **Artifacts** tab from the project sidebar.
2. Select an artifact from the list to view the details page for the latest version of that artifact.
3. Select the **Metadata** tab to view the metadata associated with that artifact.

### Usage panel

<Frame>
  <img alt="Artifact usage panel" />
</Frame>

The usage panel provides a code snippet for downloading the artifact for use outside of the web app, for example on a local machine. This section also indicates and links to the run which output the artifact and any runs which use the artifact as an input.

To view the artifact usage:

1. Select the **Artifacts** tab from the project sidebar.
2. Select an artifact from the list to view the details page for the latest version of that artifact.
3. Select the **Usage** tab to view the code snippet and related runs.

### Files panel

<Frame>
  <img alt="Artifact files panel" />
</Frame>

The files panel lists the files and folders associated with the artifact. W\&B uploads certain files for a run automatically. For example, `requirements.txt` shows the versions of each library the run used, and `wandb-metadata.json`, and `wandb-summary.json` include information about the run. Other files may be uploaded, such as artifacts or media, depending on the run's configuration.

To view files logged to an artifact:

1. Select the **Artifacts** tab from the project sidebar.
2. Select an artifact from the list to view the details page for the latest version of that artifact.
3. Select the **Files** tab to view all files associated with that artifact.

### Lineage panel

<Frame>
  <img alt="Artifact lineage" />
</Frame>

The lineage panel provides a view of all of the artifacts associated with a project and the runs that connect them to each other. It shows run types as blocks and artifacts as circles, with arrows to indicate when a run of a given type consumes or produces an artifact of a given type. The type of the particular artifact selected in the left-hand column is highlighted.

Click the Explode toggle to view all of the individual artifact versions and the specific runs that connect them.

### Versions tab

<Frame>
  <img alt="Artifact versions tab" />
</Frame>

The versions tab shows all versions of the artifact. Select an artifact to view the details for that specific version.

1. Select the **Artifacts** tab from the project sidebar.
2. Select an artifact from the list to view the details page for the latest version of that artifact.
3. Select the **Versions** tab to view all versions of that artifact.
4. From the dropdown (next to the artifact name), select **All Versions**.

For example, the previous image shows different model artifact versions for a model artifact named `"zoo-wyhak4p0"`.

## Create a project

You can create a project in the W\&B App or programmatically by specifying a project in a call to `wandb.init()`.

<Tabs>
  <Tab title="W&B App">
    In the W\&B App, you can create a project from the **Projects** page or from a team's landing page.

    From the **Projects** page:

    1. Click the global navigation icon in the upper left. The project sidebar opens.
    2. In the **Projects** section of the navigation, click **View all** to open the project overview page.
    3. Click **Create new project**.
    4. Set **Team** to the name of the team that will own the project.
    5. Specify a name for your project using the **Name** field.
    6. Set **Project visibility**, which defaults to **Team**.
    7. Optionally, provide a **Description**.
    8. Click **Create project**.

    From a team's landing page:

    1. Click the global navigation icon in the upper left. The project sidebar opens.
    2. In the **Teams** section of the navigation, click the name of a team to open its landing page.
    3. In the landing page, click **Create new project**.
    4. **Team** is automatically set to the team that owns the landing page you were viewing. If necessary, change the team.
    5. Specify a name for your project using the **Name** field.
    6. Set **Project visibility**, which defaults to **Team**.
    7. Optionally, provide a **Description**.
    8. Click **Create project**.
  </Tab>

  <Tab title="Python SDK">
    To create a project programmatically, specify a `project` when calling `wandb.init()`. If the project does not yet exist, it is created automatically, and is owned by the specified entity. For example:

    ```python theme={null}
    import wandb with wandb.init(entity="<entity>", project="<project_name>") as run: run.log({"accuracy": .95})
    ```

    Refer to the [`wandb.init()` API reference](/models/ref/python/functions/init/#examples).
  </Tab>
</Tabs>

## Organization home and Recent Activity

In the W\&B App, the organization home page summarizes recent work.

The **Recent Activity** viewer shows recent runs. Choose **Your projects** to see your own runs, or **Organization** to see recent runs by your colleagues.

* **Search** to filter the activity list.
* **Sortable columns** let you reorder rows by the column you care about.
* **Notes** and **tags** columns display automatically if a run includes these details.

Below the **Recent Activity** viewer, quickly access your recently created reports.

## Star a project

Add a star to a project to mark that project as important.

Starred projects are grouped at the top of the **Projects** page. Quickly access up to 10 recent projects in the **Projects** list in the top-left **Global navigation menu**, with starred projects listed first.

There are two ways to mark a project as important: within a project's overview tab or within your team's profile page.

<Tabs>
  <Tab title="Project overview">
    1. Navigate to your W\&B project on the W\&B App at `https://wandb.ai/<team>/<project-name>`.
    2. Select the **Overview** tab from the project sidebar.
    3. Click the star icon in the upper right corner next to the **Edit** button.

    <Frame>
      <img alt="Star project from overview" />
    </Frame>
  </Tab>

  <Tab title="Team profile">
    1. Navigate to your team's profile page at `https://wandb.ai/<team>/projects`.
    2. Select the **Projects** tab.
    3. Hover your mouse next to the project you want to star. Click on star icon that appears.

    For example, the following image shows the star icon next to the "Compare\_Zoo\_Models" project.

    <Frame>
      <img alt="Star project from team page" />
    </Frame>
  </Tab>
</Tabs>

Confirm that your project is available.. In the **Project** section of the left navigation, click **View all** to open the **Project** page. Search or filter to find your project by its name, team, or other metadata. Starred projects appear at the top of the**Projects** page, and at the top of the **Projects** section of the left navigation.

## Delete a project

You can delete your project using the **action (<Icon icon="ellipsis" />)** menu on the right of the overview tab.

1. Navigate to your W\&B project
2. Select the **Overview** tab from the project sidebar.
3. Click the **action (<Icon icon="ellipsis" />)** menu in the upper right hand corner.
4. Select **Delete project** from the dropdown menu.

<Frame>
  <img alt="Project overview menu with delete project option in the dropdown" />
</Frame>

<Warning>
  W\&B does not terminate active [sweeps](/models/sweeps) or agents when you delete a project.
</Warning>

## Add notes to a project

Add notes to your project either as a description overview or as a markdown panel within your workspace.

### Add description overview to a project

Descriptions you add to your page appear in the **Overview** tab of your profile.

1. Navigate to your W\&B project
2. Select the **Overview** tab from the project sidebar
3. Click **Edit** in the upper right hand corner
4. Add your notes in the **Description** field
5. Select the **Save** button

<Note>
  **Create reports to create descriptive notes comparing runs**

  You can also create a W\&B Report to add plots and markdown side by side. Use different sections to show different runs, and tell a story about what you worked on.
</Note>

### Add notes to run workspace

1. Navigate to your W\&B project
2. Select the **Workspace** tab from the project sidebar
3. Click the **Add panels** button from the top right corner
4. Select the **TEXT AND CODE** dropdown from the modal that appears
5. Select **Markdown**
6. Add your notes in the markdown panel that appears in your workspace


# Import and export data
Source: https://docs.wandb.ai/models/track/public-api-guide

Import data from MLFlow, export or update data that you have saved to W&B

Export data or import data  with W\&B Public APIs.

<Note>
  This feature requires python>=3.8
</Note>

## Import data from MLFlow

W\&B supports importing data from MLFlow, including experiments, runs, artifacts, metrics, and other metadata.

Install dependencies:

```shell theme={null}
# note: this requires py38+
pip install wandb[importers]
```

Log in to W\&B. Follow the prompts if you have not logged in before.

```shell theme={null}
wandb login
```

Import all runs from an existing MLFlow server:

```py theme={null}
from wandb.apis.importers.mlflow import MlflowImporter

importer = MlflowImporter(mlflow_tracking_uri="...")

runs = importer.collect_runs()
importer.import_runs(runs)
```

By default, `importer.collect_runs()` collects all runs from the MLFlow server. If you prefer to upload a special subset, you can construct your own runs iterable and pass it to the importer.

```py theme={null}
import mlflow
from wandb.apis.importers.mlflow import MlflowRun

client = mlflow.tracking.MlflowClient(mlflow_tracking_uri)

runs: Iterable[MlflowRun] = []
for run in mlflow_client.search_runs(...):
    runs.append(MlflowRun(run, client))

importer.import_runs(runs)
```

<Note>
  You might need to [configure the Databricks CLI first](https://docs.databricks.com/dev-tools/cli/index.html) if you import from Databricks MLFlow.

  Set `mlflow-tracking-uri="databricks"` in the previous step.
</Note>

To skip importing artifacts, you can pass `artifacts=False`:

```py theme={null}
importer.import_runs(runs, artifacts=False)
```

To import to a specific W\&B entity and project, you can pass a `Namespace`:

```py theme={null}
from wandb.apis.importers import Namespace

importer.import_runs(runs, namespace=Namespace(entity, project))
```

## Export data

Use the Public API to export or update data that you have saved to W\&B. Before using this API, log data from your script. Check the [Quickstart](/models/quickstart/) for more details.

**Use Cases for the Public API**

* **Export Data**: Pull down a dataframe for custom analysis in a Jupyter Notebook. Once you have explored the data, you can sync your findings by creating a new analysis run and logging results, for example: `wandb.init(job_type="analysis")`
* **Update Existing Runs**: You can update the data logged in association with a W\&B run. For example, you might want to update the config of a set of runs to include additional information, like the architecture or a hyperparameter that wasn't originally logged.

See the [Generated Reference Docs](/models/ref/python/public-api/) for details on available functions.

### Create an API key

An API key authenticates your machine to W\&B.

To create an API key, select the **Personal API key** or **Service Account API key** tab for details.

<Tabs>
  <Tab title="Personal API key">
    To create a personal API key owned by your user ID:

    1. Log in to W\&B, click your user profile icon, then click **User Settings**.
    2. Click **Create new API key**.
    3. Provide a descriptive name for your API key.
    4. Click **Create**.
    5. Copy the displayed API key immediately and store it securely.
  </Tab>

  <Tab title="Service account API key">
    To create an API key owned by a service account:

    1. Navigate to the **Service Accounts** tab in your team or organization settings.
    2. Find the service account in the list.
    3. Click the **action (<Icon icon="ellipsis" />)** menu, then click **Create API key**.
    4. Provide a name for the API key, then click **Create**.
    5. Copy the displayed API key immediately and store it securely.
    6. Click **Done**.

    You can create multiple API keys for a single service account to support different environments or workflows.
  </Tab>
</Tabs>

<Warning>
  The full API key is only shown once at creation time. After you close the dialog, you cannot view the full API key again. Only the key ID (first part of the key) is visible in your settings. If you lose the full API key, you must create a new API key.
</Warning>

For secure storage options, see [Store API keys securely](/platform/app/settings-page/user-settings/#store-and-handle-api-keys-securely).

## Store and handle API keys securely

API keys provide access to your W\&B account and should be protected like passwords. Follow these best practices:

### Recommended storage methods

* **Secrets manager**: Use a dedicated secrets management system such as [AWS Secrets Manager](https://aws.amazon.com/secrets-manager/), [HashiCorp Vault](https://developer.hashicorp.com/vault), [Azure Key Vault](https://azure.microsoft.com/en-us/products/key-vault), or [Google Secret Manager](https://cloud.google.com/security/products/secret-manager).
* **Password manager**: Use a reputable password manager application.
* **OS-level keychains**: Store keys in macOS Keychain, Windows Credential Manager, or Linux secret service. Not suggested for production.

### What to avoid

* Never commit API keys to version control systems such as Git.
* Do not store API keys in plain text configuration files.
* Do not pass API keys on the command line, because they will be visible in the output of OS commands like `ps`.
* Avoid sharing API keys through email, chat, or other unencrypted channels.
* Do not hard-code API keys in your source code.

If an API key is exposed, delete the API key from your W\&B account immediately and contact [support](mailto:support@wandb.ai) or your AISE.

### Environment variables

When using API keys in your code, pass them through environment variables:

```bash theme={null}
export WANDB_API_KEY="your-api-key-here"
```

This approach keeps keys out of your source code and makes it easier to rotate them when needed.

<Warning>
  Avoid setting the environment variable in line with the command, because it will be visible in the output of OS commands like `ps`:

  ```bash theme={null}
  # Avoid this pattern, which can expose the API key in process managers
  export WANDB_API_KEY="your-api-key-here" ./my-script.sh
  ```
</Warning>

### SDK version compatibility

New API keys are longer than legacy keys. When authenticating with older versions of the `wandb` or `weave` SDKs, you may encounter an API key length error.

**Solution**: Update to a newer SDK version:

* `wandb` SDK v0.22.3+

  ```bash theme={null}
  pip install --upgrade wandb==0.22.3
  ```
* `weave` SDK v0.52.17+
  ```bash theme={null}
  pip install --upgrade weave==0.52.17
  ```

If you cannot upgrade the SDK immediately, set the API key using the `WANDB_API_KEY` environment variable as a workaround.

### Find the run path

To use the Public API, you'll often need the run path which is `<entity>/<project>/<run_id>`. In the app UI, open a run page and click the [Overview tab ](/models/track/public-api-guide/#overview-tab)to get the run path.

### Export run data

Download data from a finished or active run. Common usage includes downloading a dataframe for custom analysis in a Jupyter notebook, or using custom logic in an automated environment.

```python theme={null}
import wandb

api = wandb.Api()
run = api.run("<entity>/<project>/<run_id>")
```

The most commonly used attributes of a run object are:

| Attribute       | Meaning                                                                                                                                                                                                                                                                                                          |
| --------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `run.config`    | A dictionary of the run's configuration information, such as the hyperparameters for a training run or the preprocessing methods for a run that creates a dataset Artifact. Think of these as the run's inputs.                                                                                                  |
| `run.history()` | A list of dictionaries meant to store values that change while the model is training such as loss. The command `run.log()` appends to this object.                                                                                                                                                               |
| `run.summary`   | A dictionary of information that summarizes the run's results. This can be scalars like accuracy and loss, or large files. By default, `run.log()` sets the summary to the final value of a logged time series. The contents of the summary can also be set directly. Think of the summary as the run's outputs. |

You can also modify or update the data of past runs. By default a single instance of an api object will cache all network requests. If your use case requires real time information in a running script, call `api.flush()` to get updated values.

### Understanding different run attributes

The following code snippet shows how to create a run, log some data, and then access the run's attributes:

```python theme={null}
import wandb
import random

with wandb.init(project="public-api-example") as run:
    n_epochs = 5
    config = {"n_epochs": n_epochs}
    run.config.update(config)
    for n in range(run.config.get("n_epochs")):
        run.log(
            {"val": random.randint(0, 1000), "loss": (random.randint(0, 1000) / 1000.00)}
        )
```

The following sections describe the different outputs for the above run object attributes

##### `run.config`

```python theme={null}
{"n_epochs": 5}
```

#### `run.summary`

```python theme={null}
{
    "_step": 4,
    "_timestamp": 1644345412,
    "_wandb": {"runtime": 3},
    "loss": 0.041,
    "val": 525,
}
```

### Sampling

The default history method samples the metrics to a fixed number of samples (the default is 500, you can change this with the `samples` \_\_ argument). If you want to export all of the data on a large run, you can use the `run.scan_history()` method. For more details see the [API Reference](/models/ref/python/public-api).

### Querying multiple runs

<Tabs>
  <Tab title="DataFrame and CSVs">
    This example script finds a project and outputs a CSV of runs with name, configs and summary stats. Replace `<entity>` and `<project>` with your W\&B entity and the name of your project, respectively.

    ```python theme={null}
    import pandas as pd
    import wandb

    api = wandb.Api()
    entity, project = "<entity>", "<project>"
    runs = api.runs(entity + "/" + project)

    summary_list, config_list, name_list = [], [], []
    for run in runs:
        # .summary contains output keys/values for
        # metrics such as accuracy.
        #  We call ._json_dict to omit large files
        summary_list.append(run.summary._json_dict)

        # .config contains the hyperparameters.
        #  We remove special values that start with _.
        config_list.append({k: v for k, v in run.config.items() if not k.startswith("_")})

        # .name is the human-readable name of the run.
        name_list.append(run.name)

    runs_df = pd.DataFrame(
        {"summary": summary_list, "config": config_list, "name": name_list}
    )

    runs_df.to_csv("project.csv")

    run.finish()
    ```
  </Tab>

  <Tab title="MongoDB Style">
    The W\&B API also provides a way for you to query across runs in a project with api.runs(). The most common use case is exporting runs data for custom analysis. The query interface is the same as the one [MongoDB uses](https://www.mongodb.com/docs/manual/reference/mql/query-predicates/).

    ```python theme={null}
    runs = api.runs(
        "username/project",
        {"$or": [{"config.experiment_name": "foo"}, {"config.experiment_name": "bar"}]},
    )
    print(f"Found {len(runs)} runs")
    ```
  </Tab>
</Tabs>

Calling `api.runs` returns a `Runs` object that is iterable and acts like a list. By default the object loads 50 runs at a time in sequence as required, but you can change the number loaded per page with the `per_page` keyword argument.

`api.runs` also accepts an `order` keyword argument. The default order is `-created_at`. To order results ascending, specify `+created_at`. You can also sort by config or summary values. For example, `summary.val_acc` or `config.experiment_name`.

### Error handling

If errors occur while talking to W\&B servers a `wandb.CommError` will be raised. The original exception can be introspected via the `exc` attribute.

### Get the latest git commit through the API

In the UI, click on a run and then click the Overview tab on the run page to see the latest git commit. It's also in the file `wandb-metadata.json` . Using the public API, you can get the git hash with `run.commit`.

### Get a run's name and ID during a run

After calling `wandb.init()` you can access the random run ID or the human readable run name from your script like this:

* Unique run ID (8 character hash): `run.id`
* Random run name (human readable): `run.name`

If you're thinking about ways to set useful identifiers for your runs, here's what we recommend:

* **Run ID**: leave it as the generated hash. This needs to be unique across runs in your project.
* **Run name**: This should be something short, readable, and preferably unique so that you can tell the difference between different lines on your charts.
* **Run notes**: This is a great place to put a quick description of what you're doing in your run. You can set this with `wandb.init(notes="your notes here")`
* **Run tags**: Track things dynamically in run tags, and use filters in the UI to filter your table down to just the runs you care about. You can set tags from your script and then edit them in the UI, both in the runs table and the overview tab of the run page. See the detailed instructions [here](/models/runs/tags/).

## Public API Examples

### Export data to visualize in matplotlib or seaborn

Check out our [API examples](/models/ref/python/public-api/) for some common export patterns. You can also click the download button on a custom plot or on the expanded runs table to download a CSV from your browser.

### Read metrics from a run

This example outputs timestamp and accuracy saved with `run.log({"accuracy": acc})` for a run saved to `"<entity>/<project>/<run_id>"`.

```python theme={null}
import wandb

api = wandb.Api()

run = api.run("<entity>/<project>/<run_id>")
if run.state == "finished":
    for i, row in run.history().iterrows():
        print(row["_timestamp"], row["accuracy"])
```

### Filter runs

You can filter by using the MongoDB Query Language.

#### Date

```python theme={null}
runs = api.runs(
    "<entity>/<project>",
    {"$and": [{"created_at": {"$lt": "YYYY-MM-DDT##", "$gt": "YYYY-MM-DDT##"}}]},
)
```

### Read specific metrics from a run

To pull specific metrics from a run, use the `keys` argument. The default number of samples when using `run.history()` is 500. Logged steps that do not include a specific metric will appear in the output dataframe as `NaN`. The `keys` argument will cause the API to sample steps that include the listed metric keys more frequently.

```python theme={null}
import wandb

api = wandb.Api()

run = api.run("<entity>/<project>/<run_id>")
if run.state == "finished":
    for i, row in run.history(keys=["accuracy"]).iterrows():
        print(row["_timestamp"], row["accuracy"])
```

### Compare two runs

This will output the config parameters that are different between `run1` and `run2`.

```python theme={null}
import pandas as pd
import wandb

api = wandb.Api()

# replace with your <entity>, <project>, and <run_id>
run1 = api.run("<entity>/<project>/<run_id>")
run2 = api.run("<entity>/<project>/<run_id>")


df = pd.DataFrame([run1.config, run2.config]).transpose()

df.columns = [run1.name, run2.name]
print(df[df[run1.name] != df[run2.name]])
```

Outputs:

```
              c_10_sgd_0.025_0.01_long_switch base_adam_4_conv_2fc
batch_size                                 32                   16
n_conv_layers                               5                    4
optimizer                             rmsprop                 adam
```

### Update metrics for a run, after the run has finished

This example sets the accuracy of a previous run to `0.9`. It also modifies the accuracy histogram of a previous run to be the histogram of `numpy_array`.

```python theme={null}
import wandb

api = wandb.Api()

run = api.run("<entity>/<project>/<run_id>")
run.summary["accuracy"] = 0.9
run.summary["accuracy_histogram"] = wandb.Histogram(numpy_array)
run.summary.update()
```

### Rename a metric in a completed run

This example renames a summary column in your tables.

```python theme={null}
import wandb

api = wandb.Api()

run = api.run("<entity>/<project>/<run_id>")
run.summary["new_name"] = run.summary["old_name"]
del run.summary["old_name"]
run.summary.update()
```

<Note>
  Renaming a column only applies to tables. Charts will still refer to metrics by their original names.
</Note>

### Update config for an existing run

This example updates one of your configuration settings.

```python theme={null}
import wandb

api = wandb.Api()

run = api.run("<entity>/<project>/<run_id>")
run.config["key"] = updated_value
run.update()
```

### Export system resource consumptions to a CSV file

The snippet below would find the system resource consumptions and then, save them to a CSV.

```python theme={null}
import wandb

with wandb.Api().run("<entity>/<project>/<run_id>") as run:

    system_metrics = run.history(stream="events")
    system_metrics.to_csv("sys_metrics.csv")
```

### Get unsampled metric data

When you pull data from history, by default it's sampled to 500 points. Get all the logged data points using `run.scan_history()`. Here's an example downloading all the `loss` data points logged in history.

```python theme={null}
import wandb

api = wandb.Api()

run = api.run("<entity>/<project>/<run_id>")
history = run.scan_history()
losses = [row["loss"] for row in history]
```

### Get paginated data from history

If metrics are being fetched slowly on our backend or API requests are timing out, you can try lowering the page size in `scan_history` so that individual requests don't time out. The default page size is 500, so you can experiment with different sizes to see what works best:

```python theme={null}
import wandb

api = wandb.Api()

run = api.run("<entity>/<project>/<run_id>")
run.scan_history(keys=sorted(cols), page_size=100)
```

### Export metrics from all runs in a project to a CSV file

This script pulls down the runs in a project and produces a dataframe and a CSV of runs including their names, configs, and summary stats. Replace `<entity>` and `<project>` with your W\&B entity and the name of your project, respectively.

```python theme={null}
import pandas as pd
import wandb

api = wandb.Api()
entity, project = "<entity>", "<project>"
runs = api.runs(entity + "/" + project)

summary_list, config_list, name_list = [], [], []
for run in runs:
    # .summary contains the output keys/values
    #  for metrics such as accuracy.
    #  We call ._json_dict to omit large files
    summary_list.append(run.summary._json_dict)

    # .config contains the hyperparameters.
    #  We remove special values that start with _.
    config_list.append({k: v for k, v in run.config.items() if not k.startswith("_")})

    # .name is the human-readable name of the run.
    name_list.append(run.name)

runs_df = pd.DataFrame(
    {"summary": summary_list, "config": config_list, "name": name_list}
)

runs_df.to_csv("project.csv")
```

### Get the starting time for a run

This code snippet retrieves the time at which the run was created.

```python theme={null}
import wandb

api = wandb.Api()

run = api.run("entity/project/run_id")
start_time = run.created_at
```

### Upload files to a finished run

The code snippet below uploads a selected file to a finished run.

```python theme={null}
import wandb

api = wandb.Api()

run = api.run("entity/project/run_id")
run.upload_file("file_name.extension")
```

### Download a file from a run

This finds the file "model-best.h5" associated with run ID uxte44z7 in the cifar project and saves it locally.

```python theme={null}
import wandb

api = wandb.Api()

run = api.run("<entity>/<project>/<run_id>")
run.file("model-best.h5").download()
```

### Download all files from a run

This finds all files associated with a run and saves them locally.

```python theme={null}
import wandb

api = wandb.Api()

run = api.run("<entity>/<project>/<run_id>")
for file in run.files():
    file.download()
```

### Get runs from a specific sweep

This snippet downloads all the runs associated with a particular sweep.

```python theme={null}
import wandb

api = wandb.Api()

sweep = api.sweep("<entity>/<project>/<sweep_id>")
sweep_runs = sweep.runs
```

### Get the best run from a sweep

The following snippet gets the best run from a given sweep.

```python theme={null}
import wandb

api = wandb.Api()

sweep = api.sweep("<entity>/<project>/<sweep_id>")
best_run = sweep.best_run()
```

The `best_run` is the run with the best metric as defined by the `metric` parameter in the sweep config.

### Download the best model file from a sweep

This snippet downloads the model file with the highest validation accuracy from a sweep with runs that saved model files to `model.h5`.

```python theme={null}
import wandb

api = wandb.Api()

sweep = api.sweep("<entity>/<project>/<sweep_id>")
runs = sorted(sweep.runs, key=lambda run: run.summary.get("val_acc", 0), reverse=True)
val_acc = runs[0].summary.get("val_acc", 0)
print(f"Best run {runs[0].name} with {val_acc}% val accuracy")

runs[0].file("model.h5").download(replace=True)
print("Best model saved to model-best.h5")
```

### Delete all files with a given extension from a run

This snippet deletes files with a given extension from a run.

```python theme={null}
import wandb

api = wandb.Api()

run = api.run("<entity>/<project>/<run_id>")

extension = ".png"
files = run.files()
for file in files:
    if file.name.endswith(extension):
        file.delete()
```

### Download system metrics data

This snippet produces a dataframe with all the system resource consumption metrics for a run and then saves it to a CSV.

```python theme={null}
import wandb

api = wandb.Api()

run = api.run("<entity>/<project>/<run_id>")
system_metrics = run.history(stream="events")
system_metrics.to_csv("sys_metrics.csv")
```

### Update summary metrics

You can pass a dictionary to update summary metrics.

```python theme={null}
summary.update({"key": val})
```

### Get the command that ran the run

Each run captures the command that launched it on the run overview page. To pull this command down from the API, you can run:

```python theme={null}
import wandb

api = wandb.Api()

run = api.run("<entity>/<project>/<run_id>")

meta = json.load(run.file("wandb-metadata.json").download())
program = ["python"] + [meta["program"]] + meta["args"]
```


# Reproduce experiments
Source: https://docs.wandb.ai/models/track/reproduce_experiments

Reproduce a teammate's W&B experiment by downloading the associated code, dependencies, and configuration from a run.

Reproduce an experiment that a team member creates to verify and validate their results.

Before you reproduce an experiment, you need to make note of the:

* Name of the project the run was logged to
* Name of the run you want to reproduce

To reproduce an experiment:

1. Navigate to the project where the run is logged to.
2. Select the **Workspace** tab from the project sidebar.
3. Click the run that you want to reproduce. The run page opens with the **Overview** tab shown by default.

To continue, download the experiment's code at a given hash or clone the experiment's entire repository.

<Tabs>
  <Tab title="Download Python script or notebook">
    Download the experiment's Python script or notebook:

    1. On the **Overview** tab (shown by default), in the **Command** field, make a note of the name of the script that created the experiment.
    2. Select the **Code** tab on the run page.
    3. Click **Download** next to the file that corresponds to the script or notebook.
  </Tab>

  <Tab title="GitHub">
    Clone the GitHub repository your teammate used when creating the experiment. To do this:

    1. If necessary, gain access to the GitHub repository that your teammate used to create the experiment.
    2. Copy the **Git repository** field, which contains the GitHub repository URL.
    3. Clone the repository:
       ```bash theme={null}
       git clone https://github.com/your-repo.git && cd your-repo
       ```
    4. Copy and paste the **Git state** field into your terminal. The Git state is a set of Git commands that checks out the exact commit that your teammate used to create the experiment. Replace values specified in the following code snippet with your own:
       ```bash theme={null}
       git checkout -b "<run-name>" 0123456789012345678901234567890123456789
       ```
  </Tab>
</Tabs>

5. On the run page, select the **Files** tab.

6. Download the `requirements.txt` file and store it in your working directory. This directory should contain either the cloned GitHub repository or the downloaded Python script or notebook.

7. (Recommended) Create a Python virtual environment.

8. Install the requirements specified in the `requirements.txt` file.
   ```bash theme={null}
   pip install -r requirements.txt
   ```

9. Now that you have the code and dependencies, you can run the script or notebook to reproduce the experiment. If you cloned a repository, you might need to navigate to the directory where the script or notebook is located. Otherwise, you can run the script or notebook from your working directory.

<Tabs>
  <Tab title="Python notebook">
    If you downloaded a Python notebook, navigate to the directory where you downloaded the notebook and run the following command in your terminal:

    ```bash theme={null}
    jupyter notebook
    ```
  </Tab>

  <Tab title="Python script">
    If you downloaded a Python script, navigate to the directory where you downloaded the script and run the following command in your terminal; Replace values enclosed in `<>` with your own:

    ```bash theme={null}
    python <your-script-name>.py
    ```
  </Tab>
</Tabs>


# View experiments results
Source: https://docs.wandb.ai/models/track/workspaces

A playground for exploring run data with interactive visualizations

W\&B workspace is your personal sandbox to customize charts and explore model results. A W\&B workspace consists of *Tables* and *Panel sections*:

* **Tables**: All runs logged to your project are listed in the project's table. Turn on and off runs, change colors, and expand the table to see notes, config, and summary metrics for each run.
* **Panel sections**: A section that contains one or more [panels](/models/app/features/panels/). Create new panels, organize them, and export to reports to save snapshots of your workspace.

<Frame>
  <img alt="Workspace table and panels" />
</Frame>

## Workspace types

There are two main workspace categories: **Personal workspaces** and **Saved views**.

* **Personal workspaces:**  A customizable workspace for in-depth analysis of models and data visualizations. Only the owner of the workspace can edit and save changes. Teammates can view a personal workspace but teammates can not make changes to someone else's personal workspace.
* **Saved views:** Saved views are collaborative snapshots of a workspace. Anyone on your team can view, edit, and save changes to saved workspace views. Use saved workspace views for reviewing and discussing experiments, runs, and more.

The following image shows multiple personal workspaces created by Cécile-parker's teammates. In this project, there are no saved views:

<Frame>
  <img alt="No saved views" />
</Frame>

## Saved workspace views

Improve team collaboration with tailored workspace views. Create Saved Views to organize your preferred setup of charts and data.

### Create a new saved workspace view

1. Navigate to a personal workspace or a saved view.
2. Make edits to the workspace.
3. Click the **action (<Icon icon="ellipsis" />)** menu at the top right corner of your workspace. Click **Save as a new view**.

New saved views appear in the workspace navigation menu.

<Frame>
  <img alt="Saved views menu" />
</Frame>

### Update a saved workspace view

Saved changes overwrite the previous state of the saved view. Unsaved changes are not retained. To update a saved workspace view in W\&B:

1. Navigate to a saved view.
2. Make the desired changes to your charts and data within the workspace.
3. Click the **Save** button to confirm your changes.

<Note>
  A confirmation dialog appears when you save your updates to a workspace view. If you prefer not to see this prompt in the future, select the option **Do not show this modal next time** before confirming your save.
</Note>

### Delete a saved workspace view

Remove saved views that are no longer needed.

1. Navigate to the saved view you want to remove.
2. Select the **menu (<Icon icon="bars" />)** button at the top right of the view.
3. Choose **Delete view**.
4. Confirm the deletion to remove the view from your workspace menu.

### Share a workspace view

Share your customized workspace with your team by sharing the workspace URL directly. All users with access to the workspace project can see the saved Views of that workspace.

## Workspace templates

<Note>
  This feature requires an [Enterprise](https://wandb.ai/site/pricing/) license.
</Note>

Use *workspace templates* to create workspaces using the same settings as an existing workspace instead of the [default settings for new workspaces](#default-workspace-settings).

### Default workspace settings

W\&B uses the following default settings for workspaces:

#### Hide empty sections during search

By default, W\&B does not show empty sections in the workspace. You can change this setting to show empty sections in the workspace.

#### Sort panels alphabetically

By default, W\&B does not sort panels by plot title alphabetically. You can change this setting to sort panels alphabetically by their plot title.

#### Section organization

By default, W\&B organizes panels into sections based on the first prefix of the metric name. For example, if your workspace includes the metrics `a/b/c/d` and `a/e/f`, W\&B organizes these metrics into a section called `a`. If your workspace includes the metrics `a/b/c/d` and `e/f/g`, W\&B organizes these metrics into sections called `a` and `e`.

You can change the default section organization to group by the last prefix. For example, if your workspace includes the metrics `a/b/c/d` and `a/e/f`, W\&B organizes these metrics into sections called `d` and `f`. If your workspace includes the metrics `a/b/c/d` and `e/f/g`, W\&B organizes these metrics into sections called `d` and `g`.

#### Line plot settings

By default, new workspaces use these default settings for line plots:

| X axis    | Default |
| --------- | ------- |
| Value     | Step    |
| Log scale | false   |

| Y axis    | Default |
| --------- | ------- |
| Log scale | false   |

| Smoothing                     | Default         |
| ----------------------------- | --------------- |
| Type                          | Time weight EMA |
| Weight                        | 0               |
| Show original after smoothing | Off             |

| Max number of runs | Default |
| ------------------ | ------- |
| Max runs           | 10      |

| Data              | Default       |
| ----------------- | ------------- |
| Point aggregation | Full fidelity |

| Grouping               | Default |
| ---------------------- | ------- |
| Use grouping in charts | On      |
| Group aggregation      | Mean    |

| Display preferences       | Default |
| ------------------------- | ------- |
| Color run names           | On      |
| Display full run name     | Off     |
| Show X range in tooltip   | Off     |
| Tooltip runs              | Default |
| Sync zoom across charts   | Off     |
| Show highlighted run only | Off     |

### Configure your workspace template

1. Open any workspace or create a new one.
2. Click on the **Settings** button at the top right corner of the workspace.
3. Select **Workspace layout** from the panel.
4. Configure the workspace's settings according to your preferences.

### Save your workspace template

1. At the top of the workspace, click the **action (<Icon icon="ellipsis" />)** menu near the **Undo** and **Redo** arrow icons.
2. Click **Save personal workspace template**.
3. Review the settings for the template, then click **Save**.

New workspaces will use these settings instead of the defaults.

### View your workspace template

To view your workspace template's current configuration:

1. From any page, select your user icon on the top right corner. From the dropdown, choose **User Settings**.
2. Navigate to the **Personal workspace template** section. If you are using a workspace template, its configuration displays. Otherwise, the section includes no details.

### Update your workspace template

To update your workspace template:

1. Open any workspace.
2. Modify the workspace's settings. For example, set the number of runs to include to `11`.
3. To save the changes to the template, click the **action (<Icon icon="ellipsis" />)** menu near the **Undo** and **Redo** arrow icons, then click **Update personal workspace template**.
4. Verify the settings, then click **Update**. The template is updated, and reapplied to all workspaces that use it.

### Delete your workspace template

To delete your workspace template and go back to the default settings:

1. From any page, select your user icon on the top right corner. From the dropdown, choose **User Settings**.
2. Navigate to the **Personal workspace template** section. Your workspace template's configuration displays.
3. Click the trash icon next to **Settings**.

<Note>
  For Dedicated Cloud and Self-Managed, deleting your workspace template is supported on v0.70 and above. On older Server versions, update your workspace template to use the [default settings](#default-workspace-settings) instead.
</Note>

## Programmatically create workspaces

<Tip>
  For an end-to-end example, see [Programmatic Workspaces](https://colab.research.google.com/github/wandb/wandb-workspaces/blob/Update-wandb-workspaces-tuturial/Workspace_tutorial.ipynb) notebook.
</Tip>

[`wandb-workspaces`](https://github.com/wandb/wandb-workspaces/tree/main) is a Python library for programmatically working with [W\&B](https://wandb.ai/) workspaces and reports.

Define a workspace programmatically with [`wandb-workspaces`](https://github.com/wandb/wandb-workspaces/tree/main). [`wandb-workspaces`](https://github.com/wandb/wandb-workspaces/tree/main) is a Python library for programmatically working with [W\&B](https://wandb.ai/) workspaces and reports.

You can define the workspace's properties, such as:

* Set panel layouts, colors, and section orders.
* Configure workspace settings like default x-axis, section order, and collapse states.
* Add and customize panels within sections to organize workspace views.
* Load and modify existing workspaces using a URL.
* Save changes to existing workspaces or save as new views.
* Filter, group, and sort runs programmatically using simple expressions.
* Customize run appearance with settings like colors and visibility.
* Copy views from one workspace to another for integration and reuse.

### Install Workspace API

In addition to `wandb`, ensure that you install `wandb-workspaces`:

```bash theme={null}
pip install wandb wandb-workspaces
```

### Define and save a workspace view programmatically

```python theme={null}
import wandb_workspaces.reports.v2 as ws

workspace = ws.Workspace(entity="your-entity", project="your-project", views=[...])
workspace.save()
```

### Edit an existing view

```python theme={null}
existing_workspace = ws.Workspace.from_url("workspace-url")
existing_workspace.views[0] = ws.View(name="my-new-view", sections=[...])
existing_workspace.save()
```

### Copy a workspace `saved view` to another workspace

```python theme={null}
old_workspace = ws.Workspace.from_url("old-workspace-url")
old_workspace_view = old_workspace.views[0]
new_workspace = ws.Workspace(entity="new-entity", project="new-project", views=[old_workspace_view])

new_workspace.save()
```

See [`wandb-workspace examples`](https://github.com/wandb/wandb-workspaces/tree/main/examples/workspaces) for comprehensive workspace API examples.


# Settings
Source: https://docs.wandb.ai/platform/app/settings-page

Use the W&B Settings page to customize your individual user profile or team settings.

Use the W\&B Settings page to customize your individual user profile and manage settings for the teams you belong to. The following sections summarize what you can configure at each level, with links to the detailed pages.

The user settings page supports the following actions:

* [Edit your profile](/platform/app/settings-page/user-settings#profile), including your profile picture, display name, geographic location, biography information, and email addresses associated with your account.
* [Configure API keys](/platform/app/settings-page/user-settings#api-keys).
* [Configure alerts](/platform/app/settings-page/user-settings#alerts) for runs.
* [Link your GitHub repository](/platform/app/settings-page/user-settings#personal-github-integration).
* [Delete your account](/platform/app/settings-page/user-settings#delete-your-account).

For more information, see [User settings](/platform/app/settings-page/user-settings/).

If you belong to one or more teams, the team settings page supports the following actions:

* [Invite or remove members](/platform/app/settings-page/teams#members) from a team.
* [Manage alerts](/platform/app/settings-page/teams#alerts) for team runs.
* [Change privacy settings](/platform/app/settings-page/teams#privacy).
* [View and manage team storage](/platform/app/settings-page/teams#usage) usage.

For more information about team settings, see [Team settings](/platform/app/settings-page/teams/).


# Manage billing settings
Source: https://docs.wandb.ai/platform/app/settings-page/billing-settings

View plan details, monitor usage, and configure usage and spending alerts for your W&B organization's billing.

The **Billing** settings page lets organization and billing admins review the current plan, monitor usage, configure usage and spending alerts, manage payment methods, and access invoices. Use this page to keep your organization within its usage limits and to stay informed about upcoming charges.

To open billing settings:

1. In the top right corner, click your user icon.
2. From the dropdown, choose **Billing**, or choose **Settings** and then select the **Billing** tab.

## Plan details

The **Plan details** section summarizes your organization's current plan, charges, limits, and usage. From here, you can compare plans or talk to Sales.

From the **Plan details** section, you can take the following actions:

* For details and a list of users, click **Manage users**.
* For details about usage, click **View usage**.
* To view the amount of storage your organization uses, both free and paid, see the storage summary. From here, you can purchase additional storage and manage storage that is in use. Learn more about [storage settings](/platform/app/settings-page/storage/).

## Plan usage

The **Plan usage** section visually summarizes current usage and displays upcoming usage charges. For detailed insights into usage by month, click **View usage** on an individual tile. To export usage by calendar month, team, or project, click **Export CSV**.

For questions about usage or billing, contact your AISE or [Support](mailto:support@wandb.com).

### Usage alerts

Usage alerts notify organization and billing admins by email when your organization reaches 85% and 100% of its usage limit for the current billing period in a given category. Configure usage alerts to monitor consumption and avoid unexpected overages. The following categories are tracked:

| Category      | Description                                                                                                                                                                                           |
| ------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Storage       | Total accumulated volume of your organization's stored artifacts and run data, measured in GB.                                                                                                        |
| Tracked hours | Wall-clock training time during the billing period, measured in hours.                                                                                                                                |
| Weave         | Total volume of data imported during the billing period, measured in MB. Shown only if Weave is active in the deployment.                                                                             |
| Inference     | Total input and output tokens used by Inference. See [Inference Pricing](https://wandb.ai/site/pricing/inference/). Shown only if Inference is active in the deployment.                              |
| Training      | Token-based inference and compute, measured in GPU/hour. See [Serverless RL Pricing](https://wandb.ai/site/pricing/reinforcement-learning/). Shown only if Serverless RL is active in the deployment. |

For organizations on the **Enterprise** plan, all **Organization admins** and **Billing admins** receive usage alerts. On the [Pro plan](https://wandb.ai/site/pricing/), only the **Billing admin** receives usage alerts.

Messages are sent from `support@wandb.com`. The thresholds are not configurable, but you can selectively turn off one or both alerts per category.

To configure usage alerts for your organization, you must be an **Organization admin** or **Billing admin**:

1. In the top right corner, click your user profile icon.
2. In the **Account** section, click **Usage & Alerts**. Alternatively, click **Settings**, then click the **Usage & Alerts** tab.
3. From the sidebar, select a category.
4. Click the **Usage** tab to view your organization's usage and alert settings for that category.
5. Turn on or off the **85% usage alert** or **100% usage alert** for that category. <Warning>If you turn off both usage alerts, no alerts are sent for that category of usage.</Warning>

### Spending alerts

In a given billing period, spending alerts notify organization and billing admins by email when your organization reaches one or more customizable spending thresholds expressed in USD. Use spending alerts to track costs against a budget you define. You can configure up to five unique spending alerts per category:

| Category      | Description                                                                                                                                                                                                                                  |
| ------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Storage       | Charges for your organization's storage [usage](#usage-alerts) during the billing period. See [Pricing](https://wandb.ai/site/pricing/).                                                                                                     |
| Tracked hours | Charges for your organization's training [usage](#usage-alerts) during the billing period. See [Pricing](https://wandb.ai/site/pricing/).                                                                                                    |
| Weave         | Charges for your organization's Weave [usage](#usage-alerts) during the billing period. See [Pricing](https://wandb.ai/site/pricing/). Shown only if Weave is active in the deployment.                                                      |
| Inference     | Charges for your organization's Inference [usage](#usage-alerts) during the billing period. See [Inference Pricing](https://wandb.ai/site/pricing/inference/). Shown only if Inference is active in the deployment.                          |
| Training      | Charges for your organization's Serverless RL [usage](#usage-alerts) during the billing period. See [Serverless RL Pricing](https://wandb.ai/site/pricing/reinforcement-learning/). Shown only if Serverless RL is active in the deployment. |

Messages are sent from `support@wandb.com`.

For organizations on the **Enterprise** plan:

* All **Organization admins** and **Billing admins** receive spending alerts.
* No spending alerts are configured by default.

On the Pro plan:

* Only the **Billing admin** receives spending alerts.
* By default, spending alerts are configured at $200, $450, $700, and $1,000.

<Note>
  You aren't billed for tracked hours, which are unlimited for all plans. Tracked hours are displayed on the **Usage** page for monitoring purposes.
</Note>

#### View spending alerts

To view your organization's spending alerts:

1. In the top right corner, click your user profile icon.
2. In the **Account** section, click **Usage & Alerts**. Alternatively, click **Settings**, then click the **Usage & Alerts** tab.
3. From the sidebar, select a category.
4. Initially, the **Usage** tab displays. To view spending alerts, click **Spend alerts**. All configured spending alerts for the category display.

#### Add or delete a spending alert

To manage spending alerts:

1. In the top right corner, click your user profile icon.
2. In the **Account** section, click **Usage & Alerts**. Alternatively, click **Settings**, then click the **Usage & Alerts** tab.
3. From the sidebar, select a category, such as **Storage**.
4. To add an alert, click **Add alert**.
   a. Enter a threshold amount in USD.
   a. Click **Create alert**.
5. To delete a spending alert, click the trash icon next to it.

## Payment methods

The **Payment methods** section shows the payment methods on file for your organization. If you haven't added a payment method, you're prompted to do so when you upgrade your plan or add paid storage.

## Billing admin

The **Billing admin** section shows the current billing admin. The billing admin is an organization admin, receives all billing-related emails, and can view and manage payment methods.

<Note>
  In W\&B Dedicated Cloud, multiple users can be billing admins. In W\&B Multi-tenant Cloud, only one user at a time can be the billing admin.
</Note>

To change the billing admin or assign the role to additional users:

1. Click **Manage roles**.
2. Search for a user.
3. Click the **Billing admin** field in that user's row.
4. Read the summary, then click **Change billing user**.

## Invoices

If you pay using a credit card, the **Invoices** section lets you view monthly invoices. The following exceptions apply:

* For Enterprise accounts that pay through wire transfer, this section is blank. For questions, contact your account team.
* If your organization incurs no charges, no invoice is generated.


# Manage email settings
Source: https://docs.wandb.ai/platform/app/settings-page/emails

Add, delete, and manage email addresses and login methods in your W&B profile settings page.

In your W\&B **Settings** page, the Emails dashboard lets you add, delete, and manage email types and primary email addresses.

To open the Emails dashboard, follow these steps:

1. In the W\&B dashboard, select your profile icon in the upper-right corner.
2. From the dropdown, select **Settings**.
3. Scroll down to the Emails dashboard.

<Frame>
  <img alt="Email management dashboard" />
</Frame>

## Manage primary email

A smiling-face-with-sunglasses icon marks your primary email. Your primary email defaults to the address you provided when you created your W\&B account.

To change the primary email associated with your W\&B account, select the **action (<Icon icon="ellipsis-vertical" />)** menu.

<Note>
  You can only set verified emails as primary.
</Note>

<Frame>
  <img alt="Primary email dropdown" />
</Frame>

## Add emails

To add an email, select **+ Add Email**. The Auth0 page opens, where you can enter credentials for the new email or connect using single sign-on (SSO).

## Delete emails

To delete an email registered to your W\&B account, select the **action (<Icon icon="ellipsis-vertical" />)** menu and choose **Delete Emails**.

<Note>
  You can't delete a primary email. Set a different email as primary before deleting.
</Note>

## Login methods

The **Log in Methods** column shows the login methods associated with your account.

When you create a W\&B account, W\&B sends a verification email to your address. Your email account stays unverified until you verify the address, and unverified emails appear in red.

If you no longer have the original verification email, attempt to log in with your email address again to receive a second one.

For account login issues, contact [support@wandb.com](mailto:support@wandb.com).


# Manage storage
Source: https://docs.wandb.ai/platform/app/settings-page/storage

Manage W&B data storage consumption using reference artifacts, external buckets, and TTL deletion policies.

This page describes how to manage your W\&B data storage so you can stay within your storage limit. If you're approaching or exceeding your storage limit, you have multiple ways to manage your data. The best option depends on your account type and your current project setup.

## Manage storage consumption

To reduce how much storage your account uses, W\&B offers different methods of optimizing your storage consumption:

* Use [reference artifacts](/models/artifacts/track-external-files/) to track files saved outside the W\&B system instead of uploading them to W\&B storage.
* Use an [external cloud storage bucket](/platform/app/settings-page/teams/) for storage (Enterprise only).

## Delete data

If optimizing consumption isn't sufficient, you can also choose to delete data to remain under your storage limit. You have several ways to do this:

* Delete data interactively with the app UI.
* [Set a TTL policy](/models/artifacts/ttl/) on artifacts so W\&B deletes them automatically.


# Team settings
Source: https://docs.wandb.ai/platform/app/settings-page/teams

Collaborate with your colleagues, share results, and track all the experiments across your team. Manage team settings including members, alerts, and privacy.

Use W\&B Teams as a central workspace for your ML team. This page explains how organization admins and team admins can create a team, configure its members and privacy, and assign roles so that collaborators can share experiments and reports while keeping sensitive work appropriately scoped.

With W\&B Teams, you can do the following:

* **Track all the experiments** your team has tried so you never duplicate work.
* **Save and reproduce** previously trained models.
* **Share progress** and results with your boss and collaborators.
* **Catch regressions** and immediately get alerted when performance drops.
* **Benchmark model performance** and compare model versions.

<Frame>
  <img alt="Teams workspace overview" />
</Frame>

<Note>
  Only Administration account types can change team settings or remove a member from a team.
</Note>

## Create a collaborative team

Follow these steps to create a new team and invite your first collaborators. After the team exists, you can refine membership, alerts, and privacy in [Team configuration](#team-configuration).

1. [Sign up or log in](https://app.wandb.ai/login?signup=true) to your free W\&B account.
2. Click **Invite Team** in the navigation bar.
3. Create your team and invite collaborators.
4. Configure your team using the settings described in the following sections.

<Note>
  Only the admin of an organization can create a new team.
</Note>

## Team configuration

You can configure settings for your team, including its membership, alerts, and privacy settings. The following sections describe each settings category.

To manage your team's settings, navigate to the **Teams** section in the left menu and click the team you want to change settings for.

### Members

The Members section shows a list of all pending invitations and the members who have accepted the invitation to join the team. The list displays each member's name, username, email, team role, and access privileges to Models and W\&B Weave, which are inherited from the organization. You can choose from the standard team roles **Admin**, **Member**, and **View-Only**. If your organization has created [custom roles](/platform/hosting/iam/access-management/manage-organization#add-and-manage-custom-roles), you can assign a custom role instead.

See [Add and Manage teams](/platform/hosting/iam/access-management/manage-organization#add-and-manage-teams) for information on how to create a team, manage teams, and manage team membership and roles. To configure who can invite new members and configure other privacy settings for the team, refer to [Privacy](#privacy).

To remove a team member, admins can open the team settings page and click the delete button next to the departing member's name. Any runs logged to the team remain after a user leaves.

### Avatar

A team avatar helps members and visitors recognize the team across the W\&B app. To set an avatar for your team:

1. Navigate to the **Teams** section in the left menu and click the team you want to add an avatar for. This opens the team's overview page.
2. Hover over the team's default avatar image in the upper-left corner of the page and click the **Upload photo** button. This opens a file prompt.
3. From the file prompt, select the image you want to use and then click **Open**. This uploads the photo to your team and sets it as your team's avatar.

### Alerts

Alerts keep your team informed about run status without requiring anyone to watch the dashboard. You can set up alerts to notify your team when runs crash and finish. W\&B sends alerts through email or Slack, and you can customize them to meet your needs.

In the **Team alerts** section, toggle the switch next to the event type you want to receive alerts from. W\&B provides the following event type options:

* **Runs finished**: A W\&B run successfully finishes.
* **Run crashed**: A run fails to finish.
* **wandb.alert()**: A custom scriptable alert. See [Send alerts with `wandb.Run.alert()`](/models/runs/alert) for more information.

### Slack notifications

Configure Slack destinations where your team's [automations](/models/automations/) can send notifications when an event occurs in a registry or a project, such as when a new artifact is created or when a run metric meets a defined threshold. Refer to [Create a Slack automation](/models/automations/create-automations/slack).

<Info>
  This feature is available for all [Enterprise](https://wandb.ai/site/pricing/) licenses.
</Info>

### Webhooks

Configure webhooks that your team's [automations](/models/automations/) can run when an event occurs in a registry or a project, such as when a new artifact is created or when a run metric meets a defined threshold. Refer to [Create a webhook automation](/models/automations/create-automations/webhook).

<Info>
  This feature is available for all [Enterprise](https://wandb.ai/site/pricing/) licenses.
</Info>

### Privacy

Team privacy settings control visibility, invitations, report sharing, and defaults such as code saving for runs in the team. Team admins can change these options on the team **Settings** page unless an organization admin has [enforced the same category of policy](/platform/hosting/privacy-settings#enforce-privacy-settings-for-all-teams) for all teams. Organization admins can still open the team **Settings** page to review configuration.

You can change the following privacy settings:

* Hide this team from all non-members.
* Make all future team projects private (public sharing not allowed).
* Allow any team member to invite other team members (not only admins).
* Disable public sharing to outside of team for reports in private projects. This disables existing magic links.
* Automatically recommend new users with matching email domains join this team upon signup.
* Enable code saving by default.

For step-by-step navigation and how organization enforcement interacts with team toggles, see [Configure privacy settings](/platform/hosting/privacy-settings).

To change these settings, open the team dashboard at `https://wandb.ai/[TEAM-NAME]`, select **Team settings** in the left navigation, then open the **Privacy** section. Replace `[TEAM-NAME]` with your team name.

### Usage

The **Usage** section describes the total memory usage the team has consumed on the W\&B servers. The default storage plan is 100 GB. For more information about storage and pricing, see the [Pricing](https://wandb.ai/site/pricing) page.

### Storage

The **Storage** section describes the cloud storage bucket configuration that the team's data uses. For more information, see [Secure Storage Connector](#secure-storage-connector) or check out our [W\&B Server](/platform/hosting/data-security/secure-storage-connector) docs if you are self-hosting.

## Create a team profile

A team profile is a public-facing page that helps your team attract collaborators and highlight its work. You can customize your team's profile page to show an introduction and showcase reports and projects that are visible to the public or team members. Present reports, projects, and external links.

Use a team profile to do the following:

* **Highlight your research** to visitors by showcasing your best public reports.
* **Showcase the most active projects** to make it easier for teammates to find them.
* **Find collaborators** by adding external links to your company or research lab's website and any papers you've published.

## Team roles and permissions

Team roles determine what each member can see and do within the team, from viewing reports to managing other members. Select a team role when you invite colleagues to join a team. The following team role options are available:

* **Admin**: Team admins can add and remove other admins or team members. They have permissions to modify all projects and full deletion permissions. This includes, but is not limited to, deleting runs, projects, artifacts, and sweeps.
* **Member**: A regular member of the team. By default, only an admin can invite a team member. To change this behavior, refer to [Privacy settings](#privacy).
* **View-Only (Enterprise-only feature)**: View-Only members can view assets within the team such as runs, reports, and workspaces. They can follow and comment on reports, but they can't create, edit, or delete project overview, reports, or runs.
* **Custom roles (Enterprise-only feature)**: Custom roles let organization admins compose new roles based on either of the **View-Only** or **Member** roles, together with additional permissions to achieve fine-grained access control. Team admins can then assign any of those custom roles to users in their respective teams. Refer to [Introducing Custom Roles for W\&B Teams](https://wandb.ai/wandb_fc/announcements/reports/Introducing-Custom-Roles-for-W-B-Teams--Vmlldzo2MTMxMjQ3) for details.

A team member can delete only runs they created. Suppose you have two members A and B. Member B moves a run from team B's project to a different project owned by Member A. Member A cannot delete the run Member B moved to Member A's project. An admin can manage runs and sweep runs created by any team member.

### Service accounts

In addition to user roles, teams can also use **service accounts** for automation. Service accounts aren't users, but rather non-human identities used for automated workflows. Refer to [Use service accounts to automate workflows](/platform/hosting/iam/service-accounts) for detailed information.

<Note>
  W\&B recommends assigning more than one admin in a team to ensure that admin operations can continue when the primary admin isn't available.
</Note>

### Team settings

The following table summarizes which roles can manage team membership and team-wide settings. Team settings let you manage the settings for your team and its members. With these privileges, you can effectively oversee and organize your team within W\&B.

| Permissions          | View-Only | Team Member | Team Admin |
| -------------------- | --------- | ----------- | ---------- |
| Add team members     |           |             | X          |
| Remove team members  |           |             | X          |
| Manage team settings |           |             | X          |

### Reports

Report permissions grant access to create, view, and edit reports. The following table lists permissions that apply to all reports across a given team.

| Permissions    | View-Only | Team Member                                      | Team Admin |
| -------------- | --------- | ------------------------------------------------ | ---------- |
| View reports   | X         | X                                                | X          |
| Create reports |           | X                                                | X          |
| Edit reports   |           | X (team members can only edit their own reports) | X          |
| Delete reports |           | X (team members can only edit their own reports) | X          |

### Experiments

The following table lists permissions that apply to all experiments across a given team.

| Permissions                                                                          | View-Only | Team Member                                               | Team Admin |
| ------------------------------------------------------------------------------------ | --------- | --------------------------------------------------------- | ---------- |
| View experiment metadata (includes history metrics, system metrics, files, and logs) | X         | X                                                         | X          |
| Edit experiment panels and workspaces                                                |           | X                                                         | X          |
| Log experiments                                                                      |           | X                                                         | X          |
| Delete experiments                                                                   |           | X (team members can only delete experiments they created) | X          |
| Stop experiments                                                                     |           | X (team members can only stop experiments they created)   | X          |

### Artifacts

The following table lists permissions that apply to all artifacts across a given team.

| Permissions        | View-Only | Team Member | Team Admin |
| ------------------ | --------- | ----------- | ---------- |
| View artifacts     | X         | X           | X          |
| Download artifacts | X         | X           | X          |
| Create artifacts   |           | X           | X          |
| Delete artifacts   |           | X           | X          |
| Edit metadata      |           | X           | X          |
| Edit aliases       |           | X           | X          |
| Delete aliases     |           | X           | X          |

### System settings (W\&B Server only)

This section applies only to self-hosted W\&B Server deployments. Use system permissions to create and manage teams and their members and to adjust system settings. These privileges enable you to effectively administer and maintain the W\&B instance.

| Permissions               | View-Only | Team Member | Team Admin | System Admin |
| ------------------------- | --------- | ----------- | ---------- | ------------ |
| Configure system settings |           |             |            | X            |
| Create or delete teams    |           |             |            | X            |

### Team service account behavior

* When you configure a team in your training environment, you can use a service account from that team to log runs in either of private or public projects within that team. Additionally, you can attribute those runs to a user if `WANDB_USERNAME` or `WANDB_USER_EMAIL` variable exists in your environment and the referenced user is part of that team.
* When you do not configure a team in your training environment and use a service account, the runs log to the named project within that service account's parent team. In this case as well, you can attribute the runs to a user if `WANDB_USERNAME` or `WANDB_USER_EMAIL` variable exists in your environment and the referenced user is part of the service account's parent team.
* A service account can't log runs to a private project in a team different from its parent team. A service account can log runs to a project only if the project visibility is set to `Open`.

## Team trials

See the [pricing page](https://wandb.ai/site/pricing) for more information on W\&B plans. You can download all your data at any time, either using the dashboard UI or the [Export API](/models/ref/python/public-api).

## Advanced configuration

The following advanced options are available to teams with additional data residency or compliance requirements.

### Secure storage connector

The team-level secure storage connector lets teams use their own cloud storage bucket with W\&B. This provides greater data access control and data isolation for teams with highly sensitive data or strict compliance requirements. Refer to [Secure Storage Connector](/platform/hosting/data-security/secure-storage-connector) for more information.


# Manage user settings
Source: https://docs.wandb.ai/platform/app/settings-page/user-settings

Manage your profile information, account defaults, alerts, participation in beta products, GitHub integration, storage usage, account activation, and create teams in your user settings.

Your user settings let you manage your profile, default team, API keys, alerts, integrations, and other account-level options. This page describes each section of the user settings and how to use it.

To open your user settings, navigate to your user profile page and select your user icon on the top right corner. From the dropdown, choose **Settings**.

## Profile

In the **Profile** section, you can manage and modify your account name and institution. You can optionally add a biography, location, link to a personal or your institution's website, and upload a profile image.

### Edit your intro

Your intro appears at the top of your profile and is a good place to introduce yourself, link to projects, or share related accounts.

To edit your intro, click **Edit** at the top of your profile. The WYSIWYG editor that opens supports Markdown.

1. To edit a line, click it. To save time, type `/` and choose Markdown from the list.
2. To move an item, use its drag handles.
3. To delete a block, click the drag handle, then click **Delete**.
4. To save your changes, click **Save**.

Your updated intro is now visible on your profile.

#### Add social badges

To add a follow badge for the `@weights_biases` account on X, add a Markdown-style link with an HTML `<img>` tag that points to the badge image:

```markdown theme={null}
[![X: @weights_biases](https://img.shields.io/twitter/follow/weights_biases?style=social)](https://x.com/intent/follow?screen_name=weights_biases)
```

In an `<img>` tag, you can specify `width`, `height`, or both. If you specify only one, the image keeps its proportions.

## Default team

If you're a member of more than one team, the **Default team** section lets you configure the default team to use when a run or a Weave trace doesn't specify a team. If you're a member of only one team, that team is the default and this section doesn't appear.

The steps to set your default team differ depending on your deployment. Select the tab that matches your environment.

<Tabs>
  <Tab title="Multi-tenant Cloud">
    Next to **Default location to create new projects in**, click the dropdown, then select your default team.
  </Tab>

  <Tab title="Dedicated Cloud / Self-Managed">
    1. Next to **Default location to create new projects in**, click the dropdown, then select your default team or your personal entity.
    2. **Optional:** If an admin has turned on public projects in **Account** > **Settings** > **Privacy**, configure the default visibility for your new projects. Click the button next to **Default project privacy in your personal account**, then select **Private** (the default) or **Public**.
    3. **Optional:** If an admin has turned on [default saving and diffing code](/models/app/features/panels/code/) in **Account** > **Settings** > **Privacy**, click **Enable code saving in your personal account** to turn it on for your runs.
  </Tab>
</Tabs>

<Note>
  To specify the default team when you're running a script in an automated environment, specify the default location with the `WANDB_ENTITY` [environment variable](https://docs.wandb.ai/models/track/environment-variables).
</Note>

## Teams

The **Teams** section lists all of your teams.

* Click a team name to go to the team page.
* If you have permission to join additional teams, click **View teams** next to **We found teams for you to join**.
* **Optional:** Turn on **Hide teams in public profile**.

<Note>
  To create or manage a team, see [Manage teams](/platform/app/settings-page/teams/).
</Note>

## API keys

The **API Keys** section lets you manage your personal API keys for authenticating with W\&B services. From this section, you can review existing keys, create new ones, and revoke keys you no longer need.

### View your API keys

The API keys table displays:

* **Key ID**: The first part of each API key, used for identification.
* **Name**: A descriptive name you provided when you created the key.
* **Created**: When the key was created.
* **Last used**: The most recent usage timestamp.

<Note>
  For security, the table shows only the key ID (the first part of the key). The full secret API key appears only once, when you create it.
</Note>

To filter the list of API keys, enter a partial key name or ID.

### Create a new API key

To create an API key, select the **Personal API key** or **Service Account API key** tab for details.

<Tabs>
  <Tab title="Personal API key">
    To create a personal API key owned by your user ID:

    1. Log in to W\&B, click your user profile icon, then click **User Settings**.
    2. Click **Create new API key**.
    3. Provide a descriptive name for your API key.
    4. Click **Create**.
    5. Copy the displayed API key immediately and store it securely.
  </Tab>

  <Tab title="Service account API key">
    To create an API key owned by a service account:

    1. Navigate to the **Service Accounts** tab in your team or organization settings.
    2. Find the service account in the list.
    3. Click the **action (<Icon icon="ellipsis" />)** menu, then click **Create API key**.
    4. Provide a name for the API key, then click **Create**.
    5. Copy the displayed API key immediately and store it securely.
    6. Click **Done**.

    You can create multiple API keys for a single service account to support different environments or workflows.
  </Tab>
</Tabs>

<Warning>
  The full API key is only shown once at creation time. After you close the dialog, you cannot view the full API key again. Only the key ID (first part of the key) is visible in your settings. If you lose the full API key, you must create a new API key.
</Warning>

For secure storage options, see [Store API keys securely](/platform/app/settings-page/user-settings/#store-and-handle-api-keys-securely).

### Delete an API key

Delete an API key when you no longer need it or when it may have been exposed. To revoke access by deleting an API key:

1. Find the key you want to delete in the API keys table.
2. Click the delete button next to the key.
3. Confirm the deletion.

W\&B removes the key from the table, and you can no longer use it to authenticate.

<Warning>
  Deleting an API key immediately revokes access for any scripts or services that use that key. Before you delete the old key, make sure you've updated all systems to use a new key.
</Warning>

## Store and handle API keys securely

API keys provide access to your W\&B account and should be protected like passwords. Follow these best practices:

### Recommended storage methods

* **Secrets manager**: Use a dedicated secrets management system such as [AWS Secrets Manager](https://aws.amazon.com/secrets-manager/), [HashiCorp Vault](https://developer.hashicorp.com/vault), [Azure Key Vault](https://azure.microsoft.com/en-us/products/key-vault), or [Google Secret Manager](https://cloud.google.com/security/products/secret-manager).
* **Password manager**: Use a reputable password manager application.
* **OS-level keychains**: Store keys in macOS Keychain, Windows Credential Manager, or Linux secret service. Not suggested for production.

### What to avoid

* Never commit API keys to version control systems such as Git.
* Do not store API keys in plain text configuration files.
* Do not pass API keys on the command line, because they will be visible in the output of OS commands like `ps`.
* Avoid sharing API keys through email, chat, or other unencrypted channels.
* Do not hard-code API keys in your source code.

If an API key is exposed, delete the API key from your W\&B account immediately and contact [support](mailto:support@wandb.ai) or your AISE.

### Environment variables

When using API keys in your code, pass them through environment variables:

```bash theme={null}
export WANDB_API_KEY="your-api-key-here"
```

This approach keeps keys out of your source code and makes it easier to rotate them when needed.

<Warning>
  Avoid setting the environment variable in line with the command, because it will be visible in the output of OS commands like `ps`:

  ```bash theme={null}
  # Avoid this pattern, which can expose the API key in process managers
  export WANDB_API_KEY="your-api-key-here" ./my-script.sh
  ```
</Warning>

### SDK version compatibility

New API keys are longer than legacy keys. When authenticating with older versions of the `wandb` or `weave` SDKs, you may encounter an API key length error.

**Solution**: Update to a newer SDK version:

* `wandb` SDK v0.22.3+

  ```bash theme={null}
  pip install --upgrade wandb==0.22.3
  ```
* `weave` SDK v0.52.17+
  ```bash theme={null}
  pip install --upgrade weave==0.52.17
  ```

If you cannot upgrade the SDK immediately, set the API key using the `WANDB_API_KEY` environment variable as a workaround.

## Beta features

In the **Beta Features** section, you can optionally enable add-ons and previews of products in development. Select the toggle switch next to the beta feature you want to enable.

## Alerts

Get notified when your runs crash or finish, or set custom alerts with [`wandb.Run.alert()`](/models/runs/alert/). You receive notifications through email or Slack. Toggle the switch next to the event type you want to receive alerts for:

* **Runs finished**: Notify you when a W\&B run successfully finishes.
* **Run crashed**: Notify you when a run fails to finish.

For more information about how to set up and manage alerts, see [Send alerts with `wandb.Run.alert()`](/models/runs/alert/).

## Personal GitHub integration

Connect a personal GitHub account so that W\&B can access GitHub resources on your behalf. To connect a GitHub account:

1. Select the **Connect Github** button. W\&B redirects you to an open authorization (OAuth) page.
2. In the **Organization access** section, select the organization to grant access.
3. Select **Authorize wandb**.

Your personal GitHub account is now linked to your W\&B account.

## Delete your account

Select the **Delete Account** button to delete your account.

<Warning>
  You can't reverse account deletion.
</Warning>

## Storage

The **Storage** section describes the total memory your account has consumed on the W\&B servers. The default storage plan is 100 GB. For more information about storage and pricing, see the [Pricing](https://wandb.ai/site/pricing) page.


# Deployment options overview
Source: https://docs.wandb.ai/platform/hosting

Compare W&B deployment options including Multi-tenant Cloud, Dedicated Cloud, and Self-Managed installations.

You can deploy W\&B in various ways to meet your organization's infrastructure and security requirements. This page helps administrators and platform decision-makers compare the available deployment options, so you can choose the one that best fits your security, scale, and operational needs.

<Tip title="Getting started">
  To get started quickly, W\&B recommends [Multi-tenant Cloud](https://wandb.ai/home).

  Some features and functionality require an [Enterprise](https://wandb.ai/site/pricing/) license. To try these features, sign up for an [Enterprise trial](https://wandb.ai/site/enterprise-trial/) and choose your deployment type.
</Tip>

<CardGroup>
  <a aria-label="W&B Multi-tenant Cloud" />

  <Card title="W&B Multi-tenant Cloud">
    A fully managed service deployed in W\&B's cloud infrastructure, where you can access the W\&B products at the desired scale, with cost-efficient pricing options, and with continuous updates for the latest features and functionality. W\&B recommends Multi-tenant Cloud for your product trial, or to manage your production AI workflows if you don't need the security of a private deployment, self-service onboarding is important, and cost efficiency is critical.

    For more information, see [W\&B Multi-tenant Cloud](/platform/hosting/hosting-options/multi_tenant_cloud).
  </Card>

  <a aria-label="W&B Dedicated Cloud" />

  <Card title="W&B Dedicated Cloud">
    A fully managed service with dedicated, isolated infrastructure in W\&B's cloud. Choose this option if your organization requires conformance to strict governance controls including data residency, needs advanced security capabilities, and wants to optimize AI operating costs by not having to build and manage the required infrastructure with security, scale, and performance characteristics.

    For more information, see [W\&B Dedicated Cloud](/platform/hosting/hosting-options/dedicated-cloud).
  </Card>

  <a aria-label="W&B Self-Managed" />

  <Card title="W&B Self-Managed">
    Deploy and manage W\&B Server on your own managed infrastructure. W\&B Server is a self-contained packaged mechanism to run the W\&B Platform and its supported W\&B products. W\&B recommends this option if all your existing infrastructure is on-premises, or your organization has strict regulatory needs that W\&B Dedicated Cloud doesn't satisfy. With this option, you're fully responsible for managing the provisioning, and continuous maintenance and upgrades of the infrastructure required to support W\&B Server.

    For more information, see [W\&B Self-Managed](/platform/hosting/hosting-options/self-managed).
  </Card>
</CardGroup>

## Key differences

After reviewing the preceding deployment options, use the following table to compare how each option handles infrastructure, security, and operational responsibilities.

|                                       | Multi-tenant Cloud                                                                                                                                                                                     | Dedicated Cloud                                                                                                                                                                                                    | Self-Managed                                     |
| ------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------ |
| MySQL / DB management                 | Fully hosted and managed by W\&B                                                                                                                                                                       | Fully hosted and managed by W\&B on cloud or region of customer choice                                                                                                                                             | Fully hosted and managed by customer             |
| Object storage (S3/GCS/Blob storage)  | **Option 1**: Fully hosted by W\&B<br />**Option 2**: Customer can configure their own bucket per team, using the [Secure Storage Connector](/platform/hosting/data-security/secure-storage-connector) | **Option 1**: Fully hosted by W\&B<br />**Option 2**: Customer can configure their own bucket per instance or team, using the [Secure Storage Connector](/platform/hosting/data-security/secure-storage-connector) | Fully hosted and managed by customer             |
| SSO support                           | W\&B managed via Auth0                                                                                                                                                                                 | **Option 1**: Customer managed<br />**Option 2**: Managed by W\&B via Auth0                                                                                                                                        | Fully managed by customer                        |
| W\&B Service (App)                    | Fully managed by W\&B                                                                                                                                                                                  | Fully managed by W\&B                                                                                                                                                                                              | Fully managed by customer                        |
| App security                          | Fully managed by W\&B                                                                                                                                                                                  | Shared responsibility of W\&B and customer                                                                                                                                                                         | Fully managed by customer                        |
| Maintenance (upgrades, backups, etc.) | Managed by W\&B                                                                                                                                                                                        | Managed by W\&B                                                                                                                                                                                                    | Managed by customer                              |
| Support                               | Support SLA                                                                                                                                                                                            | Support SLA                                                                                                                                                                                                        | Support SLA                                      |
| Supported cloud infrastructure        | Google Cloud                                                                                                                                                                                           | AWS, Google Cloud, Azure                                                                                                                                                                                           | AWS, Google Cloud, Azure, on-premises bare-metal |


# Data encryption in Dedicated Cloud
Source: https://docs.wandb.ai/platform/hosting/data-security/data-encryption

Learn how W&B encrypts data in Dedicated Cloud using cloud-native keys and the customer-managed encryption key policy.

This page describes how W\&B encrypts the W\&B-managed database and object storage in [Dedicated Cloud](/platform/hosting/hosting-options/dedicated-cloud), and explains W\&B's policy on customer-managed encryption keys. This page is intended for security and compliance teams evaluating Dedicated Cloud for use with sensitive AI workloads.

W\&B uses a W\&B-managed cloud-native key to encrypt the W\&B-managed database and object storage in every Dedicated Cloud instance, using the customer-managed encryption key (CMEK) capability in each cloud. In this case, W\&B acts as a customer of the cloud provider while providing the W\&B platform as a service to you. Using a W\&B-managed key means that W\&B controls the keys that encrypt the data in each cloud, reinforcing its commitment to provide a secure platform to its customers.

W\&B uses a unique key to encrypt the data in each customer instance, providing another layer of isolation between Dedicated Cloud tenants. The capability is available on AWS, Azure, and Google Cloud.

<Note>
  Dedicated Cloud instances on AWS have used the W\&B-managed cloud-native key for encryption since before August 2024.

  On Google Cloud and Azure, Dedicated Cloud instances that W\&B created in August 2024 or later use the W\&B-managed cloud-native key to encrypt the W\&B-managed database and object storage. Instances that W\&B provisioned before August 2024 use the default cloud provider managed key.
</Note>

W\&B doesn't generally allow customers to bring their own cloud-native key to encrypt the W\&B-managed database and object storage in their Dedicated Cloud instance. Multiple teams in an organization often have access to its cloud infrastructure, and some teams might not know that W\&B is a critical component in the organization's technology stack. They might remove the cloud-native key or revoke W\&B's access to it, which could corrupt all data in the organization's W\&B instance and leave it in an unrecoverable state.

If your organization needs to use its own cloud-native key to encrypt the W\&B-managed database and object storage as a condition for adopting Dedicated Cloud, W\&B can review the request on an exception basis. If approved, use of your cloud-native key for encryption conforms to the shared responsibility model of W\&B Dedicated Cloud.

<Warning>
  If any user in your organization removes your key or revokes W\&B's access to it at any point when your Dedicated Cloud instance is live, W\&B isn't liable for any resulting data loss or corruption and isn't responsible for recovery of the data.
</Warning>


# Configure IP allowlisting for Dedicated Cloud
Source: https://docs.wandb.ai/platform/hosting/data-security/ip-allowlisting

Restrict access to a W&B Dedicated Cloud instance to authorized IP addresses using IP allowlisting.

This page explains how IP allowlisting works on W\&B [Dedicated Cloud](/platform/hosting/hosting-options/dedicated-cloud) and how to request it for your instance. Use IP allowlisting to restrict access to your Dedicated Cloud instance to an authorized list of IP addresses, so that only traffic from approved locations can reach the W\&B APIs and the W\&B app UI.

IP allowlisting applies to access from your AI workloads to the W\&B APIs and from your user browsers to the W\&B app UI. After you set up IP allowlisting for your Dedicated Cloud instance, W\&B denies any requests from other unauthorized locations.

IP allowlisting is available on Dedicated Cloud instances on AWS, Google Cloud, and Azure.

To configure IP allowlisting for your Dedicated Cloud instance, contact your W\&B team.

You can use IP allowlisting with [secure private connectivity](./private-connectivity). If you use both, W\&B recommends that you use secure private connectivity for all traffic from your AI workloads and most of the traffic from your user browsers, and use IP allowlisting for instance administration from privileged locations.

<Warning>
  W\&B recommends that you use [CIDR blocks](https://en.wikipedia.org/wiki/Classless_Inter-Domain_Routing) assigned to your corporate or business egress gateways rather than individual `/32` IP addresses. Using individual IP addresses doesn't scale and has strict limits per cloud.
</Warning>


# Access BYOB using pre-signed URLs
Source: https://docs.wandb.ai/platform/hosting/data-security/presigned-urls

Understand how W&B uses pre-signed URLs for blob storage access, including team-level access control and audit logging.

W\&B uses pre-signed URLs to simplify access to blob storage from your AI workloads or user browsers. This page explains how pre-signed URLs work in W\&B. It also outlines the access controls, network restrictions, and audit logging that administrators should configure to secure blob storage access.

For background on pre-signed URLs, refer to the cloud provider's documentation:

* [Pre-signed URLs for AWS S3](https://docs.aws.amazon.com/AmazonS3/latest/userguide/using-presigned-url.html), which also applies to S3-compatible storage like [CoreWeave AI Object Storage](https://docs.coreweave.com/docs/products/storage/object-storage).
* [Signed URLs for Google Cloud Storage](https://cloud.google.com/storage/docs/access-control/signed-urls).
* [Shared Access Signature for Azure Blob Storage](https://learn.microsoft.com/azure/storage/common/storage-sas-overview).

Pre-signed URLs work as follows:

1. When needed, AI workloads or user browser clients within your network request pre-signed URLs from W\&B.
2. W\&B responds to the request by accessing the blob storage to generate the pre-signed URL with the required permissions.
3. W\&B returns the pre-signed URL to the client.
4. The client uses the pre-signed URL to read from or write to the blob storage.

A pre-signed URL expires after the following durations:

* **Read operations**: 1 hour.
* **Write operations**: 24 hours, to allow more time to upload large objects in chunks.

## Team-level access control

Each pre-signed URL is restricted to specific buckets based on [team-level access control](/platform/hosting/iam/access-management/manage-organization#add-and-manage-teams) in the W\&B platform. Consider a user who belongs to only one team, and that team is mapped to a storage bucket using the [secure storage connector](./secure-storage-connector). In this case, the pre-signed URLs generated for their requests can't access storage buckets mapped to other teams.

<Note>
  W\&B recommends adding users only to the teams they need to belong to.
</Note>

## Network restriction

W\&B recommends using IAM policies to restrict the networks that can use pre-signed URLs to access external storage. This helps ensure that only networks running your AI workloads, or gateway IP addresses that map to your user machines, can access your W\&B-specific buckets.

Consult your cloud provider's documentation for guidance on configuring these IAM policies:

* For CoreWeave AI Object Storage, refer to [Bucket policy reference](https://docs.coreweave.com/docs/products/storage/object-storage/reference/bucket-policy#condition) in the CoreWeave documentation.
* For AWS S3 or S3-compatible storage like MinIO hosted on your premises, refer to the [Amazon S3 User Guide](https://docs.aws.amazon.com/AmazonS3/latest/userguide/using-presigned-url.html#PresignedUrlUploadObject-LimitCapabilities), the [MinIO documentation](https://github.com/minio/minio), or the documentation for your S3-compatible storage provider.

## Audit logs

W\&B recommends using [W\&B audit logs](../monitoring-usage/audit-logging) together with blob-storage-specific audit logs. For blob storage audit logs, refer to the documentation for each cloud provider:

* [CoreWeave audit logs](https://docs.coreweave.com/docs/products/storage/object-storage/concepts/audit-logging#audit-logging-policies).
* [AWS S3 access logs](https://docs.aws.amazon.com/AmazonS3/latest/userguide/ServerLogs.html).
* [Google Cloud Storage audit logs](https://cloud.google.com/storage/docs/audit-logging).
* [Monitor Azure Blob Storage](https://learn.microsoft.com/azure/storage/blobs/monitor-blob-storage).

Admin and security teams can use audit logs to track what each user does in W\&B and take action if they need to limit certain operations for specific users.

<Note>
  Pre-signed URLs are the only supported blob storage access mechanism in W\&B. W\&B recommends configuring some or all of the preceding security controls to fit your organization's needs.
</Note>

## Determine the user that requested a pre-signed URL

To correlate pre-signed URL activity with specific W\&B users when reviewing audit logs, inspect the query parameter that W\&B appends to each URL. When W\&B returns a pre-signed URL, a query parameter in the URL contains the requester's username:

| Storage provider            | Signed URL query parameter |
| --------------------------- | -------------------------- |
| CoreWeave AI Object Storage | `X-User`                   |
| AWS S3                      | `X-User`                   |
| Google Cloud Storage        | `X-User`                   |
| Azure Blob Storage          | `scid`                     |


# Configure private connectivity to Dedicated Cloud
Source: https://docs.wandb.ai/platform/hosting/data-security/private-connectivity

Connect to a W&B Dedicated Cloud instance over a private network using AWS PrivateLink, GCP, or Azure Private Link.

This document explains how to connect to a W\&B Dedicated Cloud instance over a cloud provider's secure private network, so that traffic between your environment and W\&B avoids the public internet. It's intended for cloud and security administrators who manage Dedicated Cloud instances and want to harden network access for AI workloads and user traffic.

You can connect to your [Dedicated Cloud](/platform/hosting/hosting-options/dedicated-cloud) instance over the cloud provider's secure private network. This applies to access from your AI workloads to the W\&B APIs and optionally from your user browsers to the W\&B app UI. When you use private connectivity, the relevant requests and responses don't transit through the public network or internet. This reduces exposure to the public internet and helps satisfy network isolation requirements for sensitive workloads.

<Note>
  Secure private connectivity is offered as an advanced security option with Dedicated Cloud.
</Note>

Secure private connectivity is available on Dedicated Cloud instances on AWS, Google Cloud, and Azure:

* Use [AWS PrivateLink](https://aws.amazon.com/privatelink/) on AWS.
* Use [Google Cloud Private Service Connect](https://cloud.google.com/vpc/docs/private-service-connect) on Google Cloud.
* Use [Azure Private Link](https://azure.microsoft.com/products/private-link) on Azure.

After W\&B enables the feature, W\&B creates a private endpoint service for your instance and provides you the relevant DNS URI to connect to. With that, you can create private endpoints in your cloud accounts that route the relevant traffic to the private endpoint service. You can set up private endpoints for your AI training workloads that run within your cloud VPC or VNet. To use the same mechanism for traffic from your user browsers to the W\&B app UI, you must configure appropriate DNS-based routing from your corporate network to the private endpoints in your cloud accounts. The DNS configuration ensures browser traffic resolves to your private endpoint instead of the public W\&B endpoint.

<Note>
  To use this feature, contact your W\&B team.
</Note>

You can use secure private connectivity with [IP allowlisting](./ip-allowlisting) to combine network isolation with location-based access controls. If you combine the two, W\&B recommends that you use secure private connectivity for all traffic from your AI workloads and for browser traffic from your users where possible, and use IP allowlisting for instance administration from privileged locations.


# Bring your own bucket (BYOB)
Source: https://docs.wandb.ai/platform/hosting/data-security/secure-storage-connector

Store W&B artifacts and data in your own cloud storage buckets using the Bring Your Own Bucket (BYOB) feature.

<Note>
  **This guide applies to all W\&B deployment types:**

  * **Multi-tenant Cloud**: Team-level BYOB
  * **Dedicated Cloud**: Instance and team-level BYOB
  * **Self-Managed**: Instance and team-level BYOB

  The bucket provisioning instructions in this guide are the same regardless of your deployment type.
</Note>

## Overview

Bring your own bucket (BYOB) lets you store W\&B artifacts and other sensitive data in your own cloud or on-premises infrastructure. For [Dedicated Cloud](/platform/hosting/hosting-options/dedicated-cloud) or [Multi-tenant Cloud](/platform/hosting/hosting-options/multi_tenant_cloud), W\&B doesn't copy the data you store in your bucket to the W\&B managed infrastructure. This page is for W\&B administrators and platform engineers who need to retain ownership of artifact storage to meet data governance, residency, or compliance requirements.

<Note>
  * Communication between W\&B SDK / CLI / UI and your buckets occurs using [pre-signed URLs](./presigned-urls).
  * W\&B uses garbage collection and related processes to remove deleted **artifacts** and **run data** from your bucket over time. For artifact deletion, see [Delete an artifact](/models/artifacts/delete-artifacts). Deleted run data on Dedicated Cloud and Self-Managed deployments also depends on `GORILLA_DATA_RETENTION_PERIOD` as described in [Configure environment variables](/platform/hosting/env-vars). W\&B doesn't guarantee cleanup timing. For a single overview of bucket usage and costs, see [Manage bucket storage and costs](/platform/hosting/managing-bucket-storage).
  * You can specify a sub-path when you configure a bucket, to ensure that W\&B doesn't store any files in a folder at the root of the bucket. This helps you better conform to your organization's bucket governance policy.
</Note>

### Data stored in the central database vs buckets

When you use BYOB functionality, W\&B stores certain types of data in the W\&B central database, and other types in your bucket. Use the following lists to understand which data remains in W\&B-managed infrastructure and which data W\&B writes to your own storage.

#### Database

The W\&B central database stores the following data:

* Metadata for users, teams, artifacts, experiments, and projects.
* Reports.
* Experiment logs.
* System metrics.
* Console logs.

#### Buckets

Your storage bucket stores the following data:

* Experiment files and metrics.
* Artifact files.
* Media files.
* Run files.
* Exported history metrics and system events in Parquet format.

### Bucket scopes

You can configure your storage bucket to one of two scopes:

| Scope          | Description                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |
| -------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Instance level | In [Dedicated Cloud](/platform/hosting/hosting-options/dedicated-cloud) and [Self-Managed](/platform/hosting/hosting-options/self-managed), any user with the required permissions within your organization or instance can access files stored in your instance's storage bucket. Not applicable to [Multi-tenant Cloud](/platform/hosting/hosting-options/multi_tenant_cloud).                                                                                                                                                                                                                                            |
| Team level     | If you configure a W\&B team to use a team level storage bucket, team members can access files stored in it. Team level storage buckets allow greater data access control and data isolation for teams with sensitive data or strict compliance requirements.<br /><br />Team level storage helps different business units or departments that share an instance to efficiently use the infrastructure and administrative resources. It also lets separate project teams manage AI workflows for separate customer engagements. Available for all deployment types. You configure team level BYOB when you set up the team. |

This design supports different storage topologies, depending on your organization's needs. For example:

* The same bucket can serve the instance and one or more teams.
* Each team can use a separate bucket, some teams can choose to write to the instance bucket, or multiple teams can share a bucket by writing to subpaths.
* Buckets for different teams can reside in different cloud infrastructure environments or regions, and different storage admin teams can manage them.

For example, suppose you have a team called Kappa in your organization. Your organization (and team Kappa) uses the instance level storage bucket by default. Next, you create a team called Omega. When you create team Omega, you configure a team level storage bucket for that team. Team Kappa can't access files that team Omega generates. However, team Omega can access files that team Kappa creates. To isolate data for team Kappa, you must also configure a team level storage bucket for them.

### Availability matrix

Before you begin, confirm that BYOB is available for your deployment type and storage provider. W\&B can connect to the following storage providers:

* [CoreWeave AI Object Storage](https://docs.coreweave.com/products/storage/object-storage): High-performance, S3-compatible object storage service optimized for AI workloads.
* [Amazon S3](https://aws.amazon.com/s3/): Object storage service offering scalability, data availability, security, and performance.
* [Google Cloud Storage](https://cloud.google.com/storage): Managed service for storing unstructured data at scale.
* [Azure Blob Storage](https://azure.microsoft.com/en-us/products/storage/blobs): Cloud-based object storage solution for storing massive amounts of unstructured data like text, binary data, images, videos, and logs.
* S3-compatible storage such as [MinIO Enterprise (AIStor)](https://min.io/product/aistor) or other enterprise-grade solutions hosted in your cloud or on-premises infrastructure.

The following table shows the availability of BYOB at each scope for each W\&B deployment type.

| W\&B deployment type | Instance level | Team level                              | Additional information                                                                                                                                                                                                                                                                          |
| -------------------- | -------------- | --------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Dedicated Cloud      | ✓              | ✓                                       | Instance and team level BYOB are supported for CoreWeave AI Object Storage, Amazon S3, Google Cloud Storage, Microsoft Azure Blob Storage, and S3-compatible storage such as [MinIO Enterprise (AIStor)](https://www.min.io/product/aistor) hosted in your cloud or on-premises infrastructure. |
| Multi-tenant Cloud   | Not applicable | ✓<sup><a href="#footnote_1">1</a></sup> | Team level BYOB is supported for CoreWeave AI Object Storage, Amazon S3, and Google Cloud Storage.                                                                                                                                                                                              |
| Self-Managed         | ✓              | ✓                                       | Instance and team level BYOB are supported for CoreWeave AI Object Storage, Amazon S3, Google Cloud Storage, Microsoft Azure Blob Storage, and S3-compatible storage such as [MinIO Enterprise (AIStor)](https://www.min.io/product/aistor) hosted in your cloud or on-premises infrastructure. |

<sup><a aria-label="Footnote 1">1</a>.</sup>Azure Blob Storage is not supported for team level BYOB on Multi-tenant Cloud.

The following sections guide you through the process of setting up BYOB.

## Provision your bucket

After you [verify availability](#availability-matrix), you're ready to provision your storage bucket, including its access policy and CORS. Provisioning creates the bucket that W\&B writes to and grants the W\&B platform the permissions it needs to generate pre-signed URLs on your behalf. Select a tab to continue.

<Tabs>
  <Tab title="CoreWeave">
    <a aria-label="CoreWeave requirements" />**Requirements**:

    * **Multi-tenant Cloud**, or
    * **Dedicated Cloud** v0.73.0 or later, or
    * **Self-Managed** v0.73.0 or later deployed with v0.33.14+ of the Helm chart
    * A CoreWeave account with AI Object Storage enabled and with permission to create buckets, API access keys, and secret keys.
    * Your W\&B instance must be able to connect to CoreWeave network endpoints.

    For details, see [Create a CoreWeave AI Object Storage bucket](https://docs.coreweave.com/docs/products/storage/object-storage/buckets/create-bucket) in the CoreWeave documentation.

    1. <a />**Multi-tenant Cloud**: Obtain your organization ID, which is required for your bucket policy.
       1. Log in to the [W\&B App](https://wandb.ai/site).
       2. In the left navigation, click **Create a new team**.
       3. In the drawer that opens, copy the W\&B organization ID, which appears above **Invite team members**.
       4. Leave this page open. You use it to [configure W\&B](#configure-byob).

    2. <a aria-label="CoreWeave customer namespace" />**Dedicated Cloud** / **Self-Managed**: Obtain your customer namespace, which is required for your bucket policy.
       1. In the W\&B App, click your user profile icon, then click **System Console**.
       2. Click the **Authentication** tab.
       3. At the bottom of the page, copy the value for **Customer Namespace**. Keep this value for configuring the bucket policy.
       4. You can close the System Console.

    3. In CoreWeave, create the bucket with a name of your choice in your preferred CoreWeave availability zone. Optionally, create a folder for W\&B to use as a sub-path for all W\&B files. Make a note of the bucket name, availability zone, API access key, secret key, and sub-path.

    4. Set the following cross-origin resource sharing (CORS) policy for the bucket:
       ```json theme={null}
       [
         {
           "AllowedHeaders": [
             "*"
           ],
           "AllowedMethods": [
             "GET",
             "HEAD",
             "PUT"
           ],
           "AllowedOrigins": [
             "*"
           ],
           "ExposeHeaders": [
             "ETag"
           ],
           "MaxAgeSeconds": 3000
         }
       ]
       ```
       CoreWeave storage is S3-compatible. For details about CORS, see [Configuring cross-origin resource sharing (CORS)](https://docs.aws.amazon.com/AmazonS3/latest/userguide/enabling-cors-examples.html) in the AWS documentation.

    5. Configure a bucket policy that grants the required permissions for your W\&B deployment to access the bucket and generate [pre-signed URLs](./presigned-urls) that AI workloads in your cloud infrastructure or user browsers use to access the bucket. See [Bucket Policy Reference](https://docs.coreweave.com/docs/products/storage/object-storage/auth-access/bucket-access/bucket-policies) in the CoreWeave documentation.

       ```json theme={null}
       {
         "Version": "2012-10-17",
         "Statement": [
         {
           "Sid": "AllowWandbUser",
           "Action": [
             "s3:GetObject*",
             "s3:GetEncryptionConfiguration",
             "s3:ListBucket",
             "s3:ListBucketMultipartUploads",
             "s3:ListBucketVersions",
             "s3:AbortMultipartUpload",
             "s3:DeleteObject",
             "s3:PutObject",
             "s3:GetBucketCORS",
             "s3:GetBucketLocation",
             "s3:GetBucketVersioning"
           ],
           "Effect": "Allow",
           "Resource": [
             "arn:aws:s3:::<cw-bucket>/*",
             "arn:aws:s3:::<cw-bucket>"
           ],
           "Principal": {
             "CW": "arn:aws:iam::wandb:static/<wb-cw-principal>"
           },
           "Condition": {
             "StringLike": {
               "wandb:OrgID": [
                 "<wb-org-id>"
               ]
             }
           }
         },
         {
           "Sid": "AllowUsersInOrg",
           "Action": "s3:*",
           "Effect": "Allow",
           "Resource": [
             "arn:aws:s3:::<cw-bucket>",
             "arn:aws:s3:::<cw-bucket>/*"
           ],
           "Principal": {
             "CW": "arn:aws:iam::<cw-storage-org-id>:*"
           }
         }]
       }
       ```

       The clause beginning with `"Sid": "AllowUsersInOrg"` grants users in your organization direct access to the bucket. If you don't need this ability, you can omit the clause from your policy.

    6. In the bucket policy, replace placeholders:
       * `<cw-bucket>`: your bucket name.
       * `<cw-wandb-principal>`:
         * **Multi-tenant Cloud**: `arn:aws:iam::wandb:static/wandb-integration-public`
         * **Dedicated Cloud** or **Self-Managed**: `arn:aws:iam::wandb:static/wandb-integration`
       * `<wb-org-id>`:
         * **Multi-tenant Cloud**: The organization ID from [Provision your bucket](#coreweave-org-id).
         * **Dedicated Cloud** or **Self-Managed**: The customer namespace from [Provision your bucket](#coreweave-customer-namespace).

    7. **Dedicated Cloud**: Contact [support](mailto:support@wandb.ai) to complete additional steps.

    8. <a aria-label="Set environment variable" />**Self-Managed**: Update your W\&B deployment to set the environment variable `GORILLA_SUPPORTED_FILE_STORES` to the exact string `cw://` and restart W\&B. Otherwise, CoreWeave doesn't appear as an option when you configure team storage.

    Next, [configure W\&B](#configure-byob).
  </Tab>

  <Tab title="AWS">
    For details, see [Create an S3 bucket](https://docs.aws.amazon.com/AmazonS3/latest/userguide/create-bucket-overview.html) in the AWS documentation.

    1. Provision the KMS key.

       W\&B requires you to provision a KMS key to encrypt and decrypt the data on the S3 bucket. The key usage type must be `ENCRYPT_DECRYPT`. Assign the following policy to the key:

       ```json theme={null}
       {
         "Version": "2012-10-17",
         "Statement": [
           {
             "Sid" : "Internal",
             "Effect" : "Allow",
             "Principal" : { "AWS" : "<Your_Account_Id>" },
             "Action" : "kms:*",
             "Resource" : "<aws_kms_key.key.arn>"
           },
           {
             "Sid" : "External",
             "Effect" : "Allow",
             "Principal" : { "AWS" : "<aws_principal_and_role_arn>" },
             "Action" : [
               "kms:Decrypt",
               "kms:Describe*",
               "kms:Encrypt",
               "kms:ReEncrypt*",
               "kms:GenerateDataKey*"
             ],
             "Resource" : "<aws_kms_key.key.arn>"
           }
         ]
       }
       ```

       Replace `<Your_Account_Id>` and `<aws_kms_key.key.arn>` accordingly.

       If you use [Multi-tenant Cloud](/platform/hosting/hosting-options#w%26b-multi-tenant-cloud) or [Dedicated Cloud](/platform/hosting/hosting-options#w%26b-dedicated-cloud), replace `<aws_principal_and_role_arn>` with the corresponding value:

       * **Multi-tenant Cloud**: `arn:aws:iam::725579432336:role/WandbIntegration`
       * **Dedicated Cloud**: `arn:aws:iam::830241207209:root`

       This policy grants your AWS account full access to the key and also assigns the required permissions to the AWS account that hosts the W\&B platform. Keep a record of the KMS key ARN.

    2. Provision the S3 bucket.

       Follow these steps to provision the S3 bucket in your AWS account:

       1. Create the S3 bucket with a name of your choice. Optionally, create a folder that you can configure as a sub-path to store all W\&B files.
       2. Enable server-side encryption, using the KMS key from the previous step.
       3. Configure CORS with the following policy:

          ```json theme={null}
          [
            {
                "AllowedHeaders": [
                    "*"
                ],
                "AllowedMethods": [
                    "GET",
                    "HEAD",
                    "PUT"
                ],
                "AllowedOrigins": [
                    "*"
                ],
                "ExposeHeaders": [
                    "ETag"
                ],
                "MaxAgeSeconds": 3000
            }
          ]
          ```

          <Note>
            If data in your bucket expires because of an [object lifecycle management policy](https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-lifecycle-mgmt.html), you may lose the ability to read the history of some runs.
          </Note>
       4. Grant the required S3 permissions to the AWS account that hosts the W\&B platform, which requires these permissions to generate [pre-signed URLs](./presigned-urls) that AI workloads in your cloud infrastructure or user browsers use to access the bucket.

          ```json theme={null}
          {
            "Version": "2012-10-17",
            "Id": "WandBAccess",
            "Statement": [
              {
                "Sid": "WAndBAccountAccess",
                "Effect": "Allow",
                "Principal": { "AWS": "<aws_principal_and_role_arn>" },
                  "Action" : [
                    "s3:GetObject*",
                    "s3:GetEncryptionConfiguration",
                    "s3:ListBucket",
                    "s3:ListBucketMultipartUploads",
                    "s3:ListBucketVersions",
                    "s3:AbortMultipartUpload",
                    "s3:DeleteObject",
                    "s3:PutObject",
                    "s3:GetBucketCORS",
                    "s3:GetBucketLocation",
                    "s3:GetBucketVersioning"
                  ],
                "Resource": [
                  "arn:aws:s3:::<wandb_bucket>",
                  "arn:aws:s3:::<wandb_bucket>/*"
                ]
              }
            ]
          }
          ```

          Replace `<wandb_bucket>` accordingly and keep a record of the bucket name. Next, [configure W\&B](#configure-byob).

          If you use [Multi-tenant Cloud](/platform/hosting/hosting-options/multi_tenant_cloud) or [Dedicated Cloud](/platform/hosting/hosting-options/dedicated-cloud), replace `<aws_principal_and_role_arn>` with the corresponding value.

          * For [Multi-tenant Cloud](/platform/hosting/hosting-options/multi_tenant_cloud): `arn:aws:iam::725579432336:role/WandbIntegration`
          * For [Dedicated Cloud](/platform/hosting/hosting-options/dedicated-cloud): `arn:aws:iam::830241207209:root`

    For more details, see the [AWS Self-Managed hosting guide](/platform/hosting/hosting-options).
  </Tab>

  <Tab title="Google Cloud">
    For details, see [Create a bucket](https://docs.cloud.google.com/storage/docs/creating-buckets) in the Google Cloud documentation.

    1. Provision the GCS bucket.

       Follow these steps to provision the GCS bucket in your Google Cloud project:

       1. Create the GCS bucket with a name of your choice. Optionally, create a folder that you can configure as a sub-path to store all W\&B files.

       2. Set encryption type to `Google-managed`.

       3. Turn on soft deletion. See [Edit a bucket's soft delete policy](https://docs.cloud.google.com/storage/docs/use-soft-delete).

       4. Set the CORS policy with `gsutil`. This is not possible in the UI.

          1. Create a file named `cors-policy.json` locally.
          2. Copy the following CORS policy into the file and save it.

             ```json theme={null}
             [
               {
                 "origin": ["*"],
                 "responseHeader": ["Content-Type"],
                 "exposeHeaders": ["ETag"],
                 "method": ["GET", "HEAD", "PUT"],
                 "maxAgeSeconds": 3000
               }
             ]
             ```

             <Note>
               If data in your bucket expires because of an [object lifecycle management policy](https://cloud.google.com/storage/docs/lifecycle), you may lose the ability to read the history of some runs.
             </Note>

       5. Replace `<bucket_name>` with the correct bucket name and run `gsutil`.

          ```bash theme={null}
          gsutil cors set cors-policy.json gs://<bucket_name>
          ```

       6. Verify the bucket's policy. Replace `<bucket_name>` with the correct bucket name.

          ```bash theme={null}
          gsutil cors get gs://<bucket_name>
          ```

    2. If you use [Multi-tenant Cloud](/platform/hosting/hosting-options/multi_tenant_cloud) or [Dedicated Cloud](/platform/hosting/hosting-options/dedicated-cloud), grant the `storage.admin` role to the Google Cloud service account linked to the W\&B platform. W\&B requires this role to check the bucket's CORS configuration and attributes, such as whether object versioning is enabled. If the service account doesn't have the `storage.admin` role, these checks result in an HTTP 403 error.

       * For [Multi-tenant Cloud](/platform/hosting/hosting-options/multi_tenant_cloud), the account is: `wandb-integration@wandb-production.iam.gserviceaccount.com`
       * For [Dedicated Cloud](/platform/hosting/hosting-options/dedicated-cloud), the account is: `deploy@wandb-production.iam.gserviceaccount.com`

       Keep a record of the bucket name. Next, [configure W\&B for BYOB](#configure-byob).
  </Tab>

  <Tab title="Azure">
    For details, see [Create a blob storage container](https://learn.microsoft.com/en-us/azure/storage/blobs/blob-containers-portal) in the Azure documentation.

    **Instance level BYOB**:

    1. Provision the Azure Blob Storage container.

       For Self-Managed deployments, and for Dedicated Cloud deployments that don't use [this Terraform module](https://github.com/wandb/terraform-azurerm-wandb/tree/main/examples/byob), follow these steps to provision an Azure Blob Storage container in your Azure subscription:

       1. Create a container with a name of your choice. Optionally, create a folder that you can configure as a sub-path to store all W\&B files.
       2. Configure the CORS policy on the container.

          To set the CORS policy through the UI, go to the blob storage, scroll down to `Settings/Resource Sharing (CORS)`, and then set the following:

          | Parameter       | Value                |
          | --------------- | -------------------- |
          | Allowed Origins | `*`                  |
          | Allowed Methods | `GET`, `HEAD`, `PUT` |
          | Allowed Headers | `*`                  |
          | Exposed Headers | `*`                  |
          | Max Age         | `3000`               |

          <Note>
            If data in your bucket expires because of an [object lifecycle management policy](https://learn.microsoft.com/en-us/azure/storage/blobs/lifecycle-management-policy-configure?tabs=azure-portal), you may lose the ability to read the history of some runs.
          </Note>

    2. Generate a storage account access key and make a note of its name and the storage account name. If you use [Dedicated Cloud](/platform/hosting/hosting-options/dedicated-cloud), share the storage account name and access key with your W\&B team using a secure sharing mechanism.

    **Team level BYOB**:

    For Dedicated Cloud deployments, W\&B recommends that you use [Terraform](https://github.com/wandb/terraform-azurerm-wandb/tree/main/examples/secure-storage-connector) to provision the Azure Blob Storage container with the necessary access mechanism and permissions. For Dedicated Cloud deployments that don't use Terraform, or for Self-Managed deployments, provision the bucket by following the steps for provisioning instance-level storage. Provide the OIDC issuer URL for your instance. Make a note of the following details:

    * Storage account name
    * Storage container name
    * Managed identity client ID
    * Azure tenant ID
  </Tab>

  <Tab title="S3-compatible">
    Create your S3-compatible bucket. Make a note of:

    * Access key
    * Secret access key
    * URL endpoint
    * Bucket name
    * Folder path, if applicable
    * Region
  </Tab>
</Tabs>

Next, [determine the storage address](#determine-the-storage-address).

## Determine the storage address

After you provision the bucket, you need a storage address that W\&B uses to locate and authenticate to it. The following sections describe the syntax to use to connect a W\&B team to a BYOB storage bucket. In the examples, replace placeholder values between angle brackets (`<>`) with your bucket's details. Select a tab for detailed instructions.

<Tabs>
  <Tab title="CoreWeave">
    This section is relevant only for team level BYOB on **Dedicated Cloud** or **Self-Managed**. For instance level BYOB or for Multi-tenant Cloud, you're ready to [Configure W\&B](#configure-byob).

    Determine the full bucket path using the following format. Replace placeholders between angle brackets (`<>`) with the bucket's values.

    **Bucket format**:

    ```text theme={null}
    cw://<accessKey>:<secretAccessKey>@cwobject.com/<bucketName>?tls=true
    ```

    W\&B supports the `cwobject.com` HTTPS endpoint. TLS 1.3 is required. Contact [support](mailto:support@wandb.com) to express interest in other CoreWeave endpoints.
  </Tab>

  <Tab title="AWS">
    **Bucket format**:

    ```text theme={null}
    s3://<accessKey>:<secretAccessKey>@<s3_regional_url_endpoint>/<bucketName>?region=<region>
    ```

    In the address, the `region` parameter is mandatory unless both your W\&B instance and your storage bucket are deployed on AWS, and the W\&B instance's `AWS_REGION` matches the bucket's AWS S3 region.
  </Tab>

  <Tab title="Google Cloud">
    **Bucket format**:

    ```text theme={null}
    gs://<serviceAccountEmail>:<urlEncodedPrivateKey>@<bucketName>
    ```
  </Tab>

  <Tab title="Azure">
    **Bucket format**:

    ```text theme={null}
    az://:<urlEncodedAccessKey>@<storageAccountName>/<containerName>
    ```
  </Tab>

  <Tab title="S3-compatible">
    **Bucket format**:

    ```text theme={null}
    s3://<accessKey>:<secretAccessKey>@<url_endpoint>/<bucketName>?region=<region>&tls=true
    ```

    In the address, the `region` parameter is mandatory.

    <Note>
      This section is for S3-compatible storage buckets that aren't hosted in S3, such as [MinIO Enterprise (AIStor)](https://www.min.io/product/aistor) or other enterprise-grade S3-compatible solutions hosted on your premises. For storage buckets hosted in AWS S3, see the **AWS** tab instead.

      MinIO Open Source is in [maintenance mode](https://github.com/minio/minio) with no active development or pre-compiled binaries. For production deployments, use enterprise-grade S3-compatible solutions.

      For cloud-native storage buckets with an optional S3-compatible mode, use the cloud-native protocol specifier when possible. For example, use `cw://` for a CoreWeave bucket, rather than `s3://`.
    </Note>
  </Tab>
</Tabs>

After you determine the storage address, you're ready to [configure team level BYOB](#configure-team-level-byob).

## Configure W\&B

After you [provision your bucket](#provision-your-bucket) and [determine its address](#determine-the-storage-address), you're ready to configure BYOB at the [instance level](#instance-level-byob) or [team level](#team-level-byob). This final step tells W\&B to route storage of artifacts, run files, and other large objects to your bucket.

<Warning>
  Plan your storage bucket layout carefully. After you configure a storage bucket for W\&B, migrating its data to another bucket is complex and requires assistance from W\&B. This applies to storage for Dedicated Cloud and Self-Managed, as well as team-level storage for Multi-tenant Cloud. For questions, contact [support](mailto:support@wandb.com).
</Warning>

### Instance level BYOB

<Note>
  For CoreWeave AI Object Storage at the instance level, contact [W\&B support](mailto:support@wandb.com) instead of following these instructions. Self-service configuration isn't yet supported.
</Note>

For **Dedicated Cloud**: Share the bucket details with your W\&B team, who configures your Dedicated Cloud instance.

For **Self-Managed**, you can configure instance level BYOB using the W\&B App:

1. Log in to W\&B as a user with the `admin` role.
2. Click the user icon at the top, then click **System Console**.
3. Navigate to **Settings** > **System Connections**.
4. In the **Bucket Storage** section, ensure the identity in the **Identity** field has access to the new bucket.
5. Select the **Provider**.
6. Enter the **Bucket Name**.
7. Optional: Enter the **Path** to use in the new bucket.
8. Click **Save**.

After you save, W\&B uses the configured bucket as the default storage destination for new artifacts and run files at the instance level.

### Team level BYOB

You can configure team level BYOB when you create a team in the W\&B App or using the [SCIM API](/platform/hosting/iam/scim#create-team) (POST Groups with optional `storageBucket`). You have two options:

* **Use an existing bucket**: You must [determine the storage location](#determine-the-storage-address) for your bucket first.
* **Create a new bucket** (Multi-tenant Cloud only): W\&B can automatically create a bucket in your cloud provider when you create the team. W\&B supports this for CoreWeave, AWS, and Google Cloud.

<Note>
  - After you create a team, you can't change its storage.
  - For instance level BYOB, see [Instance level BYOB](#instance-level-byob) instead.
  - If you plan to configure CoreWeave storage for the team, review the [CoreWeave requirements](#coreweave-requirements) and contact [support](mailto:support@wandb.com) to verify that your bucket is configured correctly in CoreWeave and to validate your team's configuration, since you can't change the storage details after you create the team.
</Note>

Select your deployment type to continue.

<Tabs>
  <Tab title="Dedicated Cloud / Self-Hosted">
    1. **Dedicated Cloud**: You **must** provide the bucket path to your account team so that they can add it to your instance's supported file stores before you follow the rest of these steps to use the storage bucket for a team.

    2. **Self-Managed**: You **must** add the bucket path to the `GORILLA_SUPPORTED_FILE_STORES` environment variable and then restart W\&B before you follow the rest of these steps to use the storage bucket for a team.

    3. Log in to W\&B as a user with the `admin` role, click the icon at the top left to open the left navigation, then click **Create a team to collaborate**.

    4. Provide a name for the team.

    5. Set **Storage Type** to **External storage**.

       <Note>
         To use the instance level storage for team storage (regardless of whether it's internal or external), leave **Storage Type** set to **Internal**, even if the instance level bucket is configured for BYOB. To use separate external storage for the team, set **Storage Type** for the team to **External** and configure the bucket details in the next step.
       </Note>

    6. Click **Bucket location**.

    7. To use an existing bucket, select it from the list. To add a new bucket, click **Add bucket** at the bottom, then provide the bucket's details.

       Click **Cloud provider** and select **CoreWeave**, **AWS**, **Google Cloud**, or **Azure**.

       If the cloud provider isn't listed, ensure that you've followed the instructions in [Provision your bucket](#set-environment-variable) to add the bucket path to the supported file stores for your instance. If the storage provider is still not listed, [contact support](mailto:support@wandb.ai) for assistance.

    8. Specify the bucket details.
       * For **CoreWeave**, provide only the bucket name.
       * For Amazon S3, Google Cloud, or S3-compatible storage, provide the full bucket path you [determined earlier](#determine-the-storage-address).
       * For Azure on W\&B Dedicated or Self-Managed, set **Account name** to the Azure account and **Container name** to the Azure Blob Storage container.
       * Optionally, provide additional connection settings:
         * If applicable, set **Path** to the bucket sub-path.
         * **CoreWeave**: No additional connection settings required.
         * **AWS**: Set **KMS key ARN** to the ARN of your KMS encryption key.
         * **Google Cloud**: No additional connection settings required.
         * **Azure**: Specify values for **Tenant ID** and **Managed Identity Client ID**. These fields are mandatory unless you configured the connection string with `GORILLA_SUPPORTED_FILE_STORES`.

    9. Click **Create team**.

    If W\&B encounters errors accessing the bucket or detects invalid settings, an error or warning displays at the bottom of the page. Otherwise, W\&B creates the team.
  </Tab>

  <Tab title="Multi-tenant Cloud">
    1. Switch to the browser window where you previously began to create the new team to find the W\&B organization ID. Otherwise, log in to W\&B as a user with the `admin` role, click the icon at the top left to open the left navigation, then click **Create a team to collaborate**.

    2. Provide a name for the team.

    3. Set **Storage Type** to **External storage**.

    4. Click **Bucket location**.

    5. To use an existing bucket, select it from the list.

    6. To create a new bucket, click **Add bucket** at the bottom, then:

       1. Click **Cloud provider** and select **CoreWeave**, **AWS**, or **Google Cloud**.
       2. Enter the bucket details:
          * **Name**: Enter the bucket name.
          * **Path** (optional): Enter a sub-path to use within the bucket.
       3. Provide additional connection settings for the chosen cloud provider:
          * CoreWeave: No additional settings required.
          * AWS: Optionally provide a **KMS key ARN** for encryption.
          * Google Cloud: No additional settings required.

       <Note>
         When you click **Create team**, W\&B automatically creates the bucket in your cloud provider with the specified configuration.
       </Note>

    7. Invite members to the team. In **Invite team members**, specify a comma-separated list of email addresses. Otherwise, you can invite members to the team after you create it.

    8. Click **Create team**.

    If W\&B encounters errors accessing the bucket or detects invalid settings, an error or warning displays at the bottom of the page. Otherwise, W\&B creates the team.
  </Tab>
</Tabs>

## Troubleshooting

If W\&B reports errors when validating or connecting to your bucket, use the following sections to diagnose the most common causes by storage provider.

### CoreWeave

This section helps troubleshoot problems connecting to CoreWeave AI Object Storage.

* **Connection errors**
  * Verify that your W\&B instance can connect to CoreWeave network endpoints.
  * CoreWeave uses virtual-hosted style paths, where the bucket name is a subdomain at the beginning of the path. For example, `cw://bucket-name.cwobject.com` is correct, while `cw://cwobject.com/bucket-name/` isn't.
  * Bucket names must not contain underscores (`_`) or other characters incompatible with DNS rules.
  * Bucket names must be globally unique among CoreWeave locations.
  * Bucket names must not begin with `cw-` or `vip-`, which are reserved prefixes.
* **CORS validation failures**
  * A CORS policy is required. CoreWeave is S3-compatible. For details about CORS, see [Configuring cross-origin resource sharing (CORS)](https://docs.aws.amazon.com/AmazonS3/latest/userguide/enabling-cors-examples.html) in the AWS documentation.
  * `AllowedMethods` must include methods `GET`, `PUT`, and `HEAD`.
  * `ExposeHeaders` must include `ETag`.
  * The CORS policy's `AllowedOrigins` must include W\&B front-end domains. The example CORS policies provided on this page include all domains using `*`.
* **LOTA endpoint issues**
  * W\&B doesn't yet support connections to LOTA endpoints. To express interest, [contact support](mailto:support@wandb.com).
* **Access key and permission errors**
  * Verify that your CoreWeave API access key isn't expired.
  * Verify that your CoreWeave API access key and secret key have sufficient permissions `GetObject`, `PutObject`, `DeleteObject`, `ListBucket`. The examples on this page meet this requirement. See [Create and Manage Access Keys](https://docs.coreweave.com/docs/products/storage/object-storage/auth-access/manage-access-keys/about) in the CoreWeave documentation.

### Google Cloud

This section helps troubleshoot problems connecting to Google Cloud Storage.

* `Bucket does not have soft deletion enabled`
  Ensure that soft deletion is turned on for your Google Cloud Storage bucket. See [Edit a bucket's soft delete policy](https://docs.cloud.google.com/storage/docs/use-soft-delete).


# Configure environment variables
Source: https://docs.wandb.ai/platform/hosting/env-vars

Configure a self-managed W&B Server installation using environment variables for database, storage, Redis, and IAM settings.

In addition to configuring instance-level settings through the System Settings admin UI, W\&B also provides a way to configure these values in code using environment variables. This page lists the environment variables you can set to control database, storage, Redis, identity provider, and other instance-level behavior for a self-managed W\&B Server deployment. You can use these variables to manage configuration as code instead of through the admin UI. For IAM-specific variables, see [advanced configuration for IAM](./iam/advanced_env_vars).

## Environment variable reference

The following table describes each environment variable, the behavior it controls, and any constraints on its value.

| Environment variable                    | Description                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
| --------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `LICENSE`                               | Your wandb/local license                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
| `MYSQL`                                 | The MySQL connection string                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
| `BUCKET`                                | The S3 / GCS bucket for storing data                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |
| `BUCKET_QUEUE`                          | The SQS / Google PubSub queue for object creation events                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
| `NOTIFICATIONS_QUEUE`                   | The SQS queue on which to publish run events                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
| `AWS_REGION`                            | The AWS Region where your bucket lives                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
| `HOST`                                  | The FQD of your instance, for example `https://my.domain.net`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
| `OIDC_ISSUER`                           | A URL to your Open ID Connect identity provider, for example `https://cognito-idp.us-east-1.amazonaws.com/us-east-1_uiIFNdacd`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| `OIDC_CLIENT_ID`                        | The Client ID of application in your identity provider                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
| `OIDC_AUTH_METHOD`                      | Implicit (default) or pkce. For more context, see the following sections.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
| `SLACK_CLIENT_ID`                       | The client ID of the Slack application you want to use for alerts                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |
| `SLACK_SECRET`                          | The secret of the Slack application you want to use for alerts                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| `LOCAL_RESTORE`                         | If you can't access your instance, you can temporarily set this to true. Check the logs from the container for temporary credentials.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           |
| `REDIS`                                 | Can be used to set up an external REDIS instance with W\&B.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
| `LOGGING_ENABLED`                       | When set to true, access logs are streamed to stdout. You can also mount a sidecar container and tail `/var/log/gorilla.log` without setting this variable.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
| `GORILLA_ALLOW_USER_TEAM_CREATION`      | When set to true, lets non-admin users create a new team. False by default.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
| `GORILLA_CUSTOMER_SECRET_STORE_SOURCE`  | Sets the secret manager for storing team secrets used by W\&B Weave. These secret managers are supported: <ul><li><b>Internal secret manager</b> (default): <code>k8s-secretmanager://wandb-secret</code></li><li><b>AWS Secret Manager</b>: <code>aws-secretmanager</code></li><li><b>Google Cloud Secret Manager</b>: <code>gcp-secretmanager</code></li><li><b>Azure</b>: <code>az-secretmanger</code></li></ul>                                                                                                                                                                                                                                                                                                                             |
| `GORILLA_DATA_RETENTION_PERIOD`         | How long to retain deleted data from runs in hours. Deleted run data is unrecoverable. Append an `h` to the input value. For example, `"24h"`.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| `GORILLA_DISABLE_PERSONAL_ENTITY`       | When set to true, turns off [personal entities](/support/models/articles/what-is-the-difference-between-team-and-). Prevents creation of new personal projects in their personal entities and prevents writing to existing personal projects.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
| `GORILLA_GRAPHQL_DISABLE_INTROSPECTION` | When set to true, disables GraphQL introspection: `__type` and `__schema` queries return no schema data while the request still succeeds. On **Self-Managed**, setting the Gorilla configuration field `graphql-disable-introspection` has the same effect. Set this variable under `spec.values.global.extraEnv` in your `WeightsAndBiases` custom resource (see the [`global.extraEnv` example](/platform/hosting/self-managed/operator#ldap) in the Operator guide). **Client applications require [W\&B SDK v0.26.0](/release-notes/sdk-releases#0-26-0) or later** against deployments with introspection already turned off.                                                                                                              |
| `GRAPHQL_REJECT_UNAUTHED_REQUESTS`      | When set to `true` on the **API** service, rejects GraphQL requests that don't have an authenticated user. Unauthenticated requests receive HTTP 401. **Self-Managed** and **Dedicated Cloud** v0.80.0+ only; not available on Multi-tenant Cloud. This feature is opt-in: if the environment variable is unset or not `true`, behavior is unchanged. Set on the API component only (for example, `api.env` in Helm values). Before activating, confirm that workflows that rely on anonymous GraphQL access (such as viewing shared reports without signing in, or open projects) still meet your requirements. On **Self-Managed**, setting the Gorilla configuration field `graphql-reject-unauthed-requests` to `true` has the same effect. |
| `GORILLA_ARTIFACT_GC_ENABLED`           | When set to true, enables garbage collection for deleted artifacts. Required for self-managed deployments. See [Delete an artifact](/models/artifacts/delete-artifacts) for more information.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
| `WANDB_ARTIFACT_DIR`                    | Where to store all downloaded artifacts. If unset, defaults to the `artifacts` directory relative to your training script. Make sure this directory exists and the running user has permission to write to it. This does not control the location of generated metadata files, which you can set using the `WANDB_DIR` environment variable.                                                                                                                                                                                                                                                                                                                                                                                                    |
| `WANDB_DATA_DIR`                        | Where to upload staging artifacts. The default location depends on your platform, because it uses the value of `user_data_dir` from the `platformdirs` Python package. Make sure this directory exists and the running user has permission to write to it.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |
| `WANDB_DIR`                             | Where to store all generated files. If unset, defaults to the `wandb` directory relative to your training script. Make sure this directory exists and the running user has permission to write to it. This does not control the location of downloaded artifacts, which you can set using the `WANDB_ARTIFACT_DIR` environment variable.                                                                                                                                                                                                                                                                                                                                                                                                        |
| `WANDB_IDENTITY_TOKEN_FILE`             | For [identity federation](/platform/hosting/iam/identity_federation/), the absolute path to the local directory where Java Web Tokens (JWTs) are stored.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |

<Note>
  * Use the `GORILLA_DATA_RETENTION_PERIOD` environment variable cautiously. It applies to **deleted run data** (including run-associated files such as media after deletion flows). It does **not** delete artifacts; use artifact deletion and `GORILLA_ARTIFACT_GC_ENABLED` as described in [Delete an artifact](/models/artifacts/delete-artifacts). For how deleting runs and files relates to storage and this setting, see [When deleted run data is removed from storage](/models/runs/delete-runs#when-deleted-run-data-is-removed-from-storage) in **Delete runs**. Data is removed according to the retention window once the variable is set. Back up both the database and the storage bucket before you enable or change this value.

    Background removal of objects from your bucket is **approximate** and not guaranteed to finish within a specific time. For expectations, troubleshooting, and how this relates to storage costs, see [Manage bucket storage and costs](/platform/hosting/managing-bucket-storage).
  * To enable `GRAPHQL_REJECT_UNAUTHED_REQUESTS` with the [Kubernetes Operator](/platform/hosting/self-managed/operator), set it on the API component only:

    ```yaml theme={null}
    api:
    env:
        GRAPHQL_REJECT_UNAUTHED_REQUESTS: "true"
    ```

    Apply your changes and wait for the API pods to roll out before you verify the setting. You can disable the behavior by removing the variable or setting it to another value.
</Note>

## Advanced reliability settings

The following section describes optional configuration you can apply to improve the reliability and performance of your W\&B Server deployment.

### Redis

An external Redis server is optional but recommended for production systems. Redis helps improve the reliability of the service and enables caching to decrease load times, especially in large projects. Use a managed Redis service such as ElastiCache with high availability (HA) and the following specifications:

* Minimum 4 GB of memory, suggested 8 GB
* Redis version 6.x
* In transit encryption
* Authentication enabled

To configure the Redis instance with W\&B, go to the W\&B settings page at `http(s)://YOUR-W&B-SERVER-HOST/system-admin`. Enable the **Use an external Redis instance** option, and fill in the Redis connection string in the following format:

<Frame>
  <img alt="Configuring REDIS in W&B" />
</Frame>

You can also configure Redis using the environment variable `REDIS` on the container or in your Kubernetes deployment. Alternatively, you can set up `REDIS` as a Kubernetes secret.

This page assumes the Redis instance is running at the default port of `6379`. If you configure a different port, set up authentication, and want TLS enabled on the `redis` instance, the connection string format is: `redis://$USER:$PASSWORD@$HOST:$PORT?tls=true`


# Dedicated Cloud
Source: https://docs.wandb.ai/platform/hosting/hosting-options/dedicated-cloud

Learn about W&B Dedicated Cloud deployment features including compliance, data security, IAM, and maintenance policies.

W\&B Dedicated Cloud is a fully managed platform with dedicated, isolated infrastructure, deployed in W\&B's AWS, Google Cloud, or Azure cloud accounts. Each Dedicated Cloud instance has its own isolated network, compute, and storage from other W\&B Dedicated Cloud instances. W\&B stores your W\&B-specific metadata and data in isolated cloud storage and processes it using isolated cloud compute services.

Use Dedicated Cloud when you need the benefits of a managed W\&B platform along with isolation that helps you meet security, compliance, and data residency requirements. This page describes the compliance, data security, IAM, monitoring, and maintenance features available with Dedicated Cloud, and links to deeper documentation for each topic.

W\&B Dedicated Cloud is available in [multiple global regions for each cloud provider](./dedicated-cloud/regions).

## Rate limits

W\&B applies default rate limits on Dedicated Cloud to maintain instance stability. See [Rate limits](/platform/hosting/hosting-options/dedicated-cloud/rate-limits) for default values, how limits are enforced, and how to request higher limits when you scale up training.

## Compliance

W\&B Dedicated Cloud supports the following compliance frameworks:

* **SOC 2**: W\&B Dedicated Cloud's hosting platform meets the requirements of the [Service and Organization Controls (SOC) 2 Type 2](https://www.aicpa-cima.com/topic/audit-assurance/audit-and-assurance-greater-than-soc-2), published by the [Auditing Standards Board of the American Institute of Certified Public Accountants (AICPA)](https://www.aicpa-cima.com/home). A SOC 2 report evaluates a service organization's controls for security, availability, processing integrity, confidentiality, and privacy. W\&B Dedicated Cloud is subject to periodic internal and external audits to verify continued compliance. Refer to the [W\&B Security Portal](https://security.wandb.ai/) to request the SOC 2 report and other security and compliance documents.
* **HIPAA**: When configured appropriately, W\&B Dedicated Cloud meets the requirements of the [Health Insurance Portability and Accountability Act of 1996 (HIPAA)](https://www.hhs.gov/hipaa/for-professionals/index.html). Compliance with HIPAA is a shared responsibility that involves W\&B, the customer, and any third-party services involved in the deployment. Organizations subject to HIPAA must have a **Business Associate Agreement** on file with W\&B. Refer to the [W\&B Security Portal](https://security.wandb.ai/) to request more information.

## Data security

Dedicated Cloud provides several mechanisms for controlling where your data is stored and how it can be accessed. The following options let you bring your own storage, restrict network access, and connect privately to your instance.

You can bring your own bucket (BYOB) using the [secure storage connector](/platform/hosting/data-security/secure-storage-connector) at the [instance and team levels](/platform/hosting/data-security/secure-storage-connector#configuration-options) to store your files such as models, datasets, and more.

Similar to W\&B Multi-tenant Cloud, you can configure a single bucket for multiple teams or you can use separate buckets for different teams. If you don't configure secure storage connector for a team, W\&B stores that data in the instance-level bucket.

<Frame>
  <img alt="Dedicated Cloud architecture diagram" />
</Frame>

In addition to BYOB with secure storage connector, you can use [IP allowlisting](/platform/hosting/data-security/ip-allowlisting) to restrict access to your Dedicated Cloud instance from only trusted network locations.

You can connect privately to your Dedicated Cloud instance using [cloud provider's secure connectivity solution](/platform/hosting/data-security/private-connectivity).

You're responsible for ensuring that your deployment complies with your organization's policies and [Security Technical Implementation Guidelines (STIG)](https://en.wikipedia.org/wiki/Security_Technical_Implementation_Guide), if applicable.

## Identity and access management (IAM)

Use the identity and access management capabilities for secure authentication and effective authorization in your W\&B Organization. The following features are available for IAM in Dedicated Cloud instances:

* Authenticate with [SSO using OpenID Connect (OIDC)](/platform/hosting/iam/sso) or with [LDAP](/platform/hosting/iam/ldap).
* [Configure appropriate user roles](/platform/hosting/iam/access-management/manage-organization#assign-or-update-a-users-role) at the scope of the organization and within a team.
* Define the scope of a W\&B project to limit who can view, edit, and submit W\&B runs to it with [restricted projects](/platform/hosting/iam/access-management/restricted-projects).
* Use JSON Web Tokens with [identity federation](/platform/hosting/iam/identity_federation) to access W\&B APIs.

## Monitor

Use [Audit logs](/platform/hosting/monitoring-usage/audit-logging) to track user activity within your teams and to conform to your enterprise governance requirements. Also, you can view organization usage in your Dedicated Cloud instance with [W\&B Organization Dashboard](/platform/hosting/monitoring-usage/org_dashboard).

## Maintenance

Similar to W\&B Multi-tenant Cloud, you don't incur the overhead and costs of provisioning and maintaining the W\&B platform with Dedicated Cloud.

To understand how W\&B manages updates on Dedicated Cloud, refer to the [server release process](/platform/hosting/server-upgrade-process).

## Data retention policy

By default, a Dedicated Cloud instance retains the following items for 7 days after deletion:

* Runs and history
* Non-artifact run files, such as media, configuration files, and log files
* Artifacts and artifact references

Until this period elapses, you can restore these items. Contact [support](mailto:support@wandb.ai) or your AISE for assistance.

To meet your data retention requirements, you can change the data retention period for your Dedicated Cloud instance. Depending on your use case, select the **Environment variable** or **Helm** tab for details.

<Tabs>
  <Tab title="Environment variable">
    To change the data retention policy, set the environment variable `GORILLA_DATA_RETENTION_PERIOD` to a value in hours. For example, to retain deleted data for 14 days (336 hours):

    ```bash theme={null}
    export GORILLA_DATA_RETENTION_PERIOD="336h"
    ```
  </Tab>

  <Tab title="Helm">
    To change the data retention policy, set the Helm value `env.dataRetentionPeriod` to a value in hours. For example, to retain deleted data for 14 days (336 hours):

    ```yaml theme={null}
    env:
      dataRetentionPeriod: "336h"
    ```
  </Tab>
</Tabs>

## Migration options

W\&B supports migration to Dedicated Cloud from a [Self-Managed instance](/platform/hosting/hosting-options/self-managed) or [Multi-tenant Cloud](/platform/hosting/hosting-options/multi_tenant_cloud), subject to specific limits and migration-related constraints.

## Next steps

If you're interested in using Dedicated Cloud, submit the [Dedicated SaaS trial request form](https://wandb.ai/site/for-enterprise/dedicated-saas-trial) to start the process with W\&B.


# Export data from Dedicated Cloud
Source: https://docs.wandb.ai/platform/hosting/hosting-options/dedicated-cloud/export-data

Export runs, metrics, artifacts, and reports from a W&B Dedicated Cloud instance using the Python SDK API.

Export the data managed in your W\&B Dedicated Cloud instance, such as runs, metrics, artifacts, and reports, when you need a portable copy for backup, migration, or external analysis. To extract this data, use the W\&B SDK API with the [Import and Export API](/models/ref/python/public-api/). The following table covers some key export use cases.

| Purpose                  | Documentation                                                                                                                                                                                                 |
| ------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Export project metadata  | [Projects API](/models/ref/python/public-api/projects)                                                                                                                                                        |
| Export runs in a project | [Runs API](/models/ref/python/public-api/runs)                                                                                                                                                                |
| Export reports           | [Report and Workspace API](/models/reports/clone-and-export-reports/)                                                                                                                                         |
| Export artifacts         | [Explore artifact graphs](/models/artifacts/explore-and-traverse-an-artifact-graph/), [Download and use artifacts](/models/artifacts/download-and-use-an-artifact/#download-and-use-an-artifact-stored-on-wb) |

If you manage artifacts stored in the Dedicated Cloud with [Secure Storage Connector](/platform/app/settings-page/teams/#secure-storage-connector), you might not need to export the artifacts using the W\&B SDK API.

<Note>
  Using the W\&B SDK API to export all of your data can be slow if you have many runs, artifacts, and similar resources. W\&B recommends running the export process in appropriately sized batches to avoid overwhelming your Dedicated Cloud instance.
</Note>


# Rate limits
Source: https://docs.wandb.ai/platform/hosting/hosting-options/dedicated-cloud/rate-limits

Default rate limits on Dedicated Cloud and how to request changes

W\&B Dedicated Cloud applies rate limits to maintain instance stability. W\&B can adjust limits when you scale up training or need higher throughput. Use this page to understand the default limits that apply to your Dedicated Cloud instance and how to request an increase if your workloads need more capacity.

## Default limits and notification

The following section lists the default rate limits and describes how the W\&B SDK behaves when you exceed a limit.

The following default limits help maintain platform stability:

| Limit                            | Default | Scope   |
| -------------------------------- | ------- | ------- |
| Filestream requests per second   | 500     | Project |
| Filestream ingestion per second  | 120 MB  | Project |
| Filestream requests per second   | 2       | Run     |
| Run creation requests per second | 80      | Project |

When a limit is exceeded, the W\&B SDK returns HTTP response `429`, and the message `HTTP 429: rate limited exceeded` appears in the SDK logs.

* Filesystem rate limits never cause logging to crash or fail. When the SDK receives a `429` response on a filestream request, it will back off and retry the rate-limited request as-is, while subsequent updates accumulate.

* Run creation rate limits block further training.

```
HTTP 429: rate limit exceeded
```

W\&B recommends these defaults for typical production workloads. If your workloads consistently exceed these limits, contact [W\&B support](mailto:support@wandb.com) or your Account Solutions Engineer (AISE) to request higher limits. Provide details of your experimental setup and usage patterns so that W\&B can scope an appropriate adjustment.


# Supported Dedicated Cloud regions
Source: https://docs.wandb.ai/platform/hosting/hosting-options/dedicated-cloud/regions

View all supported AWS, Google Cloud, and Azure regions available for W&B Dedicated Cloud instances.

AWS, Google Cloud, and Azure support cloud computing services in multiple locations worldwide. Global regions help ensure that you satisfy requirements such as data residency, compliance, latency, and cost efficiency. W\&B supports many of the available global regions for Dedicated Cloud. The following sections list the supported regions for each cloud provider.

<Note>
  Reach out to W\&B Support if your preferred AWS, Google Cloud, or Azure region isn't listed. W\&B can validate whether the relevant region has all the services that Dedicated Cloud needs and prioritize support depending on the outcome of the evaluation.
</Note>

## Supported AWS regions

The following table lists [AWS regions](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.RegionsAndAvailabilityZones.html) that W\&B supports for Dedicated Cloud instances.

| Region location          | Region name      |
| ------------------------ | ---------------- |
| US East (Ohio)           | `us-east-2`      |
| US East (N. Virginia)    | `us-east-1`      |
| US West (N. California)  | `us-west-1`      |
| US West (Oregon)         | `us-west-2`      |
| Canada (Central)         | `ca-central-1`   |
| Europe (Frankfurt)       | `eu-central-1`   |
| Europe (Ireland)         | `eu-west-1`      |
| Europe (London)          | `eu-west-2`      |
| Europe (Milan)           | `eu-south-1`     |
| Europe (Stockholm)       | `eu-north-1`     |
| Asia Pacific (Mumbai)    | `ap-south-1`     |
| Asia Pacific (Singapore) | `ap-southeast-1` |
| Asia Pacific (Sydney)    | `ap-southeast-2` |
| Asia Pacific (Tokyo)     | `ap-northeast-1` |
| Asia Pacific (Seoul)     | `ap-northeast-2` |

For more information about AWS regions, see [Regions, Availability Zones, and Local Zones](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.RegionsAndAvailabilityZones.html) in the AWS Documentation.

See [What to Consider when Selecting a Region for your Workloads](https://aws.amazon.com/blogs/architecture/what-to-consider-when-selecting-a-region-for-your-workloads/) for an overview of factors to consider when choosing an AWS region.

## Supported Google Cloud regions

The following table lists [Google Cloud regions](https://cloud.google.com/compute/docs/regions-zones) that W\&B supports for Dedicated Cloud instances.

| Region location | Region name               |
| --------------- | ------------------------- |
| South Carolina  | `us-east1`                |
| N. Virginia     | `us-east4`                |
| Iowa            | `us-central1`             |
| Oregon          | `us-west1`                |
| Los Angeles     | `us-west2`                |
| Las Vegas       | `us-west4`                |
| Toronto         | `northamerica-northeast2` |
| Belgium         | `europe-west1`            |
| London          | `europe-west2`            |
| Frankfurt       | `europe-west3`            |
| Netherlands     | `europe-west4`            |
| Sydney          | `australia-southeast1`    |
| Tokyo           | `asia-northeast1`         |
| Seoul           | `asia-northeast3`         |

For more information about Google Cloud regions, see [Regions and zones](https://cloud.google.com/compute/docs/regions-zones) in the Google Cloud Documentation.

## Supported Azure regions

The following table lists [Azure regions](https://azure.microsoft.com/explore/global-infrastructure/geographies/#geographies) that W\&B supports for Dedicated Cloud instances.

| Region location | Region name     |
| --------------- | --------------- |
| Virginia        | `eastus`        |
| Iowa            | `centralus`     |
| Washington      | `westus2`       |
| California      | `westus`        |
| Canada Central  | `canadacentral` |
| France Central  | `francecentral` |
| Netherlands     | `westeurope`    |
| Tokyo, Saitama  | `japaneast`     |
| Seoul           | `koreacentral`  |

For more information about Azure regions, see [Azure geographies](https://azure.microsoft.com/explore/global-infrastructure/geographies/#overview) in the Azure Documentation.


# Multi-tenant Cloud
Source: https://docs.wandb.ai/platform/hosting/hosting-options/multi_tenant_cloud

Learn about W&B Multi-tenant Cloud, a fully managed deployment on Google Cloud with built-in compliance and data security.

This page describes W\&B Multi-tenant Cloud, including its architecture, compliance posture, data security options, and identity and access management capabilities, so you can decide whether it fits your organization's hosting requirements.

W\&B Multi-tenant Cloud is a fully managed platform deployed in W\&B's Google Cloud account in [Google Cloud's North America regions](https://cloud.google.com/compute/docs/regions-zones). W\&B Multi-tenant Cloud uses autoscaling in Google Cloud to scale the platform based on changes in traffic.

<Frame>
  <img alt="Multi-tenant Cloud architecture diagram" />
</Frame>

W\&B Multi-tenant Cloud scales to meet your organization's needs and supports logging up to 250,000 metrics per project with up to 1 million data points per metric. For larger deployments, contact [support](mailto:support@wandb.com).

## Compliance

W\&B Multi-tenant Cloud's hosting platform meets the requirements of the [Service and Organization Controls (SOC) 2 Type 2](https://www.aicpa-cima.com/topic/audit-assurance/audit-and-assurance-greater-than-soc-2), published by the [Auditing Standards Board of the American Institute of Certified Public Accountants (AICPA)](https://www.aicpa-cima.com/home). A SOC 2 report evaluates a service organization's controls for security, availability, processing integrity, confidentiality, and privacy. W\&B Multi-tenant Cloud is subject to periodic internal and external audits to verify continued compliance. Refer to the [W\&B Security Portal](https://security.wandb.ai/) to request the SOC 2 report and other security and compliance documents.

<Tip>W\&B Multi-tenant Cloud doesn't meet the requirements of the [Health Insurance Portability and Accountability Act of 1996 (HIPAA)](https://www.hhs.gov/hipaa/for-professionals/index.html). If your organization is subject to HIPAA, consider [W\&B Dedicated Cloud](/platform/hosting/hosting-options/dedicated-cloud) instead.</Tip>

## Data security

For users on Free or Pro plans, W\&B stores all data only in the shared cloud storage and processes it with shared cloud compute services. Depending on your pricing plan, you may be subject to storage limits.

Users on an Enterprise plan can [bring their own bucket (BYOB) using the secure storage connector](/platform/hosting/data-security/secure-storage-connector) at the [team level](/platform/hosting/data-security/secure-storage-connector#configuration-options) to store their files such as models, datasets, and more. You can configure a single bucket for multiple teams or you can use separate buckets for different W\&B Teams. If you don't configure BYOB for a team, W\&B stores the team's data in the shared cloud storage.

You are responsible for ensuring that your deployment complies with your organization's policies and [Security Technical Implementation Guidelines (STIG)](https://en.wikipedia.org/wiki/Security_Technical_Implementation_Guide), if applicable.

## Data retention policy

Multi-tenant Cloud retains the following items for 7 days after deletion:

* Runs and history.
* Non-artifact run files, such as media, configuration files, and log files.
* Artifacts and artifact references.

Until this period elapses, W\&B can restore these items. Contact [support](mailto:support@wandb.ai) or your AISE for assistance.

## Identity and access management (IAM)

If you're on an Enterprise plan, identity and access management features support secure authentication and authorization for your W\&B deployment:

* Authenticate users with SSO using OIDC or SAML. Contact your W\&B team or support if you'd like to configure SSO for your organization.
* [Configure user roles](/platform/hosting/iam/access-management/manage-organization#assign-or-update-a-users-role) at the scope of the organization and within a team.
* Define the scope of a W\&B project to limit who can view, edit, and submit W\&B runs to it with [restricted projects](/platform/hosting/iam/access-management/restricted-projects).

## Billing and usage

Organization admins can manage usage and billing for their account from the **Billing** tab in their account view. If using the shared cloud storage on Multi-tenant Cloud, an admin can optimize storage usage across different teams in their organization.

## Maintenance

W\&B Multi-tenant Cloud is a multi-tenant, fully managed platform. Because W\&B manages Multi-tenant Cloud, you don't incur the overhead and costs of provisioning and maintaining the W\&B platform.

## Next steps

Access [Multi-tenant Cloud directly](https://wandb.ai) to get started with most features for free. To try data security and IAM features, [request an Enterprise trial](https://wandb.ai/site/for-enterprise/multi-tenant-saas-trial).


# W&B Self-Managed deployment overview
Source: https://docs.wandb.ai/platform/hosting/hosting-options/self-managed

Deploy W&B Self-Managed on cloud or on-premises infrastructure

<Tip>
  W\&B recommends fully managed deployment options such as [W\&B Multi-tenant Cloud](/platform/hosting/hosting-options/multi_tenant_cloud) or [W\&B Dedicated Cloud](/platform/hosting/hosting-options/dedicated-cloud). W\&B fully managed services are secure to use, with minimal configuration required.
</Tip>

W\&B Self-Managed lets you run W\&B Server in infrastructure you control, so you can meet internal policies for data residency, network isolation, and compliance. This overview is for IT, DevOps, and MLOps teams who plan, deploy, and operate W\&B Server on their own cloud account or on-premises infrastructure.

Deploy W\&B Server on your [AWS, Google Cloud, or Azure cloud account](#deploy-wb-server-within-Self-Managed-cloud-accounts) or within your [on-premises infrastructure](#deploy-wb-server-in-on-prem-infrastructure).

Your IT, DevOps, or MLOps team has the following responsibilities:

* Provision your deployment.
* Secure your infrastructure in accordance with your organization's policies, [Security Technical Implementation Guidelines (STIG)](https://en.wikipedia.org/wiki/Security_Technical_Implementation_Guide), if applicable.
* Maintain compliance with your organization's regulatory requirements, such as [Service and Organization Controls (SOC) 2 Type 2](https://www.aicpa-cima.com/topic/audit-assurance/audit-and-assurance-greater-than-soc-2) and [Health Insurance Portability and Accountability Act of 1996 (HIPAA)](https://www.hhs.gov/hipaa/for-professionals/index.html).
* Manage upgrades and apply patches.
* Maintain your W\&B Self-Managed deployment on an ongoing basis.

<Info>
  If your organization is subject to regulatory requirements, consider deploying on [W\&B Dedicated Cloud](/platform/hosting/hosting-options/dedicated-cloud), which is maintained by W\&B.

  * W\&B Dedicated Cloud's hosting platform meets the requirements of SOC 2 Type 2.
  * When configured appropriately, a W\&B Dedicated Cloud deployment complies with HIPAA.

  Refer to the [W\&B Security Portal](https://security.wandb.ai/) to request more information.
</Info>

## About the W\&B Kubernetes Operator

W\&B Self-Managed installations are delivered and managed through a Kubernetes operator, which handles the day-to-day complexity of running W\&B Server.

Use the W\&B Kubernetes Operator to deploy W\&B Self-Managed. The operator simplifies deploying, administering, troubleshooting, and scaling W\&B.

The operator connects to a central [deploy.wandb.ai](https://deploy.wandb.ai) server to request the latest specification changes for a given release channel and apply them. The operator receives updates as long as the license is valid. The operator uses [Helm](https://helm.sh/) both to install itself and to manage the W\&B Kubernetes stack. The deployment consists of multiple pods, one per service, and each pod name is prefixed with `wandb-`.

Configuration follows a hierarchy: **Release Channel Values** (defaults from W\&B), **User Input Values** (overrides via the System Console), and **Custom Resource Values** (your spec overrides both).

Use the following resources based on your deployment target:

* To deploy W\&B Self-Managed in public cloud or on-premises infrastructure, see [Deploy W\&B with Kubernetes Operator](/platform/hosting/self-managed/operator).
* To deploy W\&B Self-Managed to a custom cloud platform that isn't AWS, the requirements are similar to the requirements to deploy in [on-premises infrastructure](#deploy-wb-server-in-on-prem-infrastructure).
* To deploy W\&B Self-Managed in air-gapped environments, see [Deploy on Air-Gapped Kubernetes](/platform/hosting/self-managed/on-premises-deployments/kubernetes-airgapped).

You can optionally configure rate limits on your instance to maintain stability. For more information, see [Rate limits](/platform/hosting/self-managed/rate-limits).

## Required infrastructure

Before you deploy, make sure the following infrastructure is available. All W\&B Self-Managed deployment options depend on these components:

* Kubernetes
* MySQL 8 database
* Amazon S3-compatible object storage
* Redis cache

W\&B can provide recommendations for the different components and provide guidance through the installation process. For more information, see [Requirements](/platform/hosting/self-managed/requirements).

## Obtain your W\&B Server license

A W\&B Server license authorizes your deployment. You must obtain a license before installation. For step-by-step instructions, see [License](/platform/hosting/self-managed/requirements#license) on the Requirements page.


# Access management
Source: https://docs.wandb.ai/platform/hosting/iam/access-management-intro

Manage users, teams, roles, and project visibility as an organization or team admin in W&B.

This page introduces how access management works in W\&B, including how to administer users and teams, maintain admin access to your organization, and control who can view individual projects. It's intended for organization admins, team admins, and project owners.

## Manage users and teams within an organization

This section explains how admin roles are assigned and what each role can do to manage users and teams.

W\&B assigns the *instance admin role* to the first user who signs up to W\&B with a unique organization domain. The organization admin assigns specific users team admin roles.

<Note>
  W\&B recommends that you have more than one instance admin in an organization. This best practice ensures that admin operations can continue when the primary admin isn't available.
</Note>

A *team admin* is a user in an organization who has administrative permissions within a team.

Organization admins can access and use an organization's account settings at `https://wandb.ai/account-settings/` to invite users, assign or update a user's role, create teams, remove users from your organization, and assign the billing admin. See [Add and manage users](/platform/hosting/iam/access-management/manage-organization/#add-and-manage-users) for more information.

After an organization admin creates a team, the instance admin or a team admin can:

* Invite users to the team or remove users from the team. By default, only an admin can perform this action. To change this behavior, see [Team settings](/platform/app/settings-page/teams/#privacy).
* Assign or update a team member's role.
* Automatically add new users to a team when they join your organization.

Both the organization admin and the team admin use team dashboards at `https://wandb.ai/[YOUR-TEAM-NAME]` to manage teams. For more information, and to configure a team's default privacy settings, see [Add and manage teams](/platform/hosting/iam/access-management/manage-organization/#add-and-manage-teams).

## Maintain admin access

This section describes why you must always preserve admin access and how automated user deprovisioning can put that access at risk.

You must ensure that at least one admin user always exists in your instance or organization. Otherwise, no user can configure or maintain your organization's W\&B account.

If you manage users interactively, deleting a user requires admin access, including when deleting another admin user. This helps reduce the risk of removing the sole admin user.

However, if an organization uses automated processes to deprovision users from W\&B, a deprovisioning operation could inadvertently remove the last remaining admin from the instance or organization.

For assistance with developing operational procedures, or to restore admin access, contact [support](mailto:support@wandb.com).

## Limit visibility to specific projects

Define the scope of a W\&B project to limit who can view, edit, and submit W\&B runs to it. Limiting who can view a project is particularly useful if a team works with sensitive or confidential data.

An organization admin, team admin, or the owner of a project can set and edit a project's visibility.

For more information, see [Project visibility](/platform/hosting/iam/access-management/restricted-projects/).


# Manage your organization
Source: https://docs.wandb.ai/platform/hosting/iam/access-management/manage-organization

Manage users, teams, roles, seats, and billing within a W&B organization as an organization admin.

This page describes how to administer a W\&B organization, including how to invite users, manage teams, assign roles and seats, and configure billing. Use these procedures to keep your organization's membership, access, and team structure aligned with how your company uses W\&B.

As an admin of an organization, you can [manage individual users](#add-and-manage-users) within your organization and [manage teams](#add-and-manage-teams).

As a team admin, you can [manage teams](#add-and-manage-teams).

<Note>
  The following workflow applies to users with instance admin roles. Contact an admin in your organization if you believe you should have instance admin permissions.
</Note>

To simplify user management in your organization, see [Automate user and team management](../automate_iam).

## Change the name of your organization

<Note>
  The following workflow only applies to W\&B Multi-tenant Cloud.
</Note>

1. Navigate to [https://wandb.ai/home](https://wandb.ai/home).
2. In the upper right corner of the page, select the **User menu** dropdown. Within the **Account** section of the dropdown, select **Settings**.
3. Within the **Settings** tab, select **General**.
4. Select the **Change name** button.
5. Within the modal that appears, provide a new name for your organization and select the **Save name** button.

## Add and manage users

As an admin, use your organization's dashboard to:

* Invite or remove users.
* Assign or update a user's organization role, and create custom roles.
* Assign the billing admin.

An organization admin can add users to an organization in several ways:

* Member-by-invite
* Auto provisioning with SSO
* Domain capture

The following sections describe each method.

### Seats and pricing

The following table summarizes how seats work for Models and Weave:

| Product | Seats       | Cost based on                                                                                                                                                                                            |
| ------- | ----------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Models  | Pay per set | How many Models paid seats you have and how much usage you've accrued determines your overall subscription cost. You can assign each user one of three available seat types: Full, Viewer, or No-Access. |
| Weave   | Free        | Usage based                                                                                                                                                                                              |

### Invite a user

Admins can invite users to their organization, as well as to specific teams within the organization.

<Tabs>
  <Tab title="Multi-tenant Cloud">
    1. Navigate to [https://wandb.ai/home](https://wandb.ai/home).
    2. In the upper right corner of the page, select the **User menu** dropdown. Within the **Account** section of the dropdown, select **Users**.
    3. Select **Invite new user**.
    4. In the modal that appears, provide the email or username of the user in the **Email or username** field.
    5. Optional: Add the user to a team from the **Choose teams** dropdown menu.
    6. From the **Select role** dropdown, select the role to assign to the user. You can change the user's role later. See the table listed in [Assign a role](#assign-or-update-a-team-members-role) for more information about possible roles.
    7. Click the **Send invite** button.

    After you select the **Send invite** button, W\&B sends an invite link to the user's email using a third-party email server. A user can access your organization once they accept the invite.
  </Tab>

  <Tab title="Dedicated Cloud or Self-Managed">
    1. Navigate to `https://<org-name>.io/console/settings/`. Replace `<org-name>` with your organization name.
    2. Select the **Add user** button.
    3. Within the modal that appears, provide the email of the new user in the **Email** field.
    4. Select a role to assign to the user from the **Role** dropdown. You can change the user's role later. See the table listed in [Assign a role](#assign-or-update-a-team-members-role) for more information about possible roles.
    5. To have W\&B send an invite link to the user's email using a third-party email server, check the **Send invite email to user** box.
    6. Select the **Add new user** button.
  </Tab>
</Tabs>

### Auto provision users

Auto-provisioning streamlines onboarding by letting users join your organization automatically through SSO, so admins don't need to send individual invitations.

If you configure SSO and your SSO provider permits it, a W\&B user with a matching email domain can log in to your W\&B organization with Single Sign-On (SSO). SSO is available for all Enterprise licenses.

<Note>
  **Enable SSO for authentication**

  W\&B recommends that users authenticate using Single Sign-On (SSO). Contact your W\&B team to enable SSO for your organization.

  For more information about how to set up SSO with Dedicated Cloud or Self-Managed instances, see [SSO with OIDC](/platform/hosting/iam/sso/) or [SSO with LDAP](/platform/hosting/iam/ldap/).
</Note>

W\&B assigns auto-provisioning users "Member" roles by default. You can change the role of auto-provisioned users at any time.

Auto-provisioning users with SSO is on by default for Dedicated Cloud instances and Self-Managed deployments. You can turn off auto provisioning. Turning auto provisioning off lets you selectively add specific users to your W\&B organization.

The following tabs describe how to turn off SSO based on deployment type:

<Tabs>
  <Tab title="Multi-tenant Cloud">
    Auto provisioning with SSO isn't configurable for Multi-tenant Cloud. Contact your W\&B team for assistance.
  </Tab>

  <Tab title="Dedicated Cloud or Self-Managed">
    For Dedicated Cloud instances, contact your W\&B team if you want to turn off auto provisioning with SSO.

    For Self-Managed deployments, use the W\&B Console to turn off auto provisioning with SSO:

    1. Navigate to `https://<org-name>.io/console/settings/`. Replace `<org-name>` with your organization name.
    2. Select **Security**.
    3. Select the **Disable SSO Provisioning** to turn off auto provisioning with SSO.
  </Tab>
</Tabs>

<Note>
  Auto provisioning with SSO is useful for adding users to an organization at scale because organization admins don't need to generate individual user invitations.
</Note>

### Domain capture

Domain capture helps your employees join your company's organization to ensure new users don't create assets outside your company's jurisdiction.

<Note>
  **Domains must be unique**

  Domains are unique identifiers. This means that you can't use a domain that's already in use by another organization.
</Note>

<Tabs>
  <Tab title="Multi-tenant Cloud">
    Domain capture lets you automatically add people with a company email address, such as `@example.com`, to your W\&B Multi-tenant Cloud organization. This helps all your employees join the right organization and ensures that new users don't create assets outside your company jurisdiction.

    This table summarizes the behavior of new and existing users with and without domain capture enabled:

    |                | With domain capture                                                                                                                                                                                                                                                             | Without domain capture                                                                                                                               |
    | -------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------- |
    | New users      | Users who sign up for W\&B from verified domains are automatically added as members to your organization's default team. They can choose additional teams to join at sign up, if you enable team joining. They can still join other organizations and teams with an invitation. | Users can create W\&B accounts without knowing a centralized organization is available.                                                              |
    | Invited users  | Invited users automatically join your organization when accepting your invite. Invited users aren't automatically added as members to your organization's default team. They can still join other organizations and teams with an invitation.                                   | Invited users automatically join your organization when accepting your invite. They can still join other organizations and teams with an invitation. |
    | Existing users | Existing users with verified email addresses from your domains can join your organization's teams within the W\&B App. All data that existing users create before joining your organization remains. W\&B doesn't migrate the existing user's data.                             | Existing W\&B users may be spread across multiple organizations and teams.                                                                           |

    To automatically assign non-invited new users to a default team when they join your organization:

    1. Navigate to [https://wandb.ai/home](https://wandb.ai/home).
    2. In the upper right corner of the page, select the **User menu** dropdown. From the dropdown, choose **Settings**.
    3. Within the **Settings** tab, select **General**.
    4. Click the **Claim domain** button within **Domain capture**.
    5. Select the team that you want new users to automatically join from the **Default team** dropdown. If no teams are available, update team settings. See the instructions in [Add and manage teams](#add-and-manage-teams).
    6. Click the **Claim email domain** button.

    You must enable domain matching within a team's settings before you can automatically assign non-invited new users to that team.

    1. Navigate to the team's dashboard at `https://wandb.ai/<team-name>`, where `<team-name>` is the name of the team for which you want to enable domain matching.
    2. Select **Team settings** in the global navigation on the left side of the team's dashboard.
    3. Within the **Privacy** section, toggle the **Recommend new users with matching email domains join this team upon signing up** option.
  </Tab>

  <Tab title="Dedicated Cloud or Self-Managed">
    If you use Dedicated Cloud or Self-Managed deployment type, contact your W\&B Account Team to configure domain capture. Once configured, your W\&B Multi-tenant Cloud instance automatically prompts users who create a W\&B account with your company email address to contact your admin to request access to your Dedicated Cloud or Self-Managed instance.

    |                | With domain capture                                                                                                                                                                                                                          | Without domain capture                                                                                                      |
    | -------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------- |
    | New users      | Users who sign up for W\&B on Multi-tenant Cloud from verified domains are automatically prompted to contact an admin with an email address you customize. They can still create an organization on Multi-tenant Cloud to trial the product. | Users can create W\&B Multi-tenant Cloud accounts without learning that their company has a centralized dedicated instance. |
    | Existing users | Existing W\&B users may be spread across multiple organizations and teams.                                                                                                                                                                   | Existing W\&B users may be spread across multiple organizations and teams.                                                  |
  </Tab>
</Tabs>

### Assign or update a user's role

A user's organization role controls what they can do across the organization, such as inviting other users, managing teams, or viewing content. Assign or update roles to give each user the appropriate level of administrative authority.

Every member in an organization has an organization role and seat for both W\&B Models and Weave. The type of seat they have determines both their billing status and the actions they can take in each product line.

You initially assign an organization role to a user when you invite them to your organization. You can change any user's role later.

A user within an organization can have one of the following roles:

| Role                                   | Descriptions                                                                                                                                                                                                                                                                                                                                                                 |
| -------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Admin                                  | An instance admin who can add or remove other users to the organization, change user roles, manage custom roles, add teams, and more. W\&B recommends having more than one admin in case your admin is unavailable.                                                                                                                                                          |
| Member                                 | A regular user of the organization, invited by an instance admin. An organization member can't invite other users or manage existing users in the organization.                                                                                                                                                                                                              |
| Viewer (Enterprise-only feature)       | A view-only user of your organization, invited by an instance admin. A viewer only has read access to the organization and the underlying teams that they are a member of.                                                                                                                                                                                                   |
| Custom Roles (Enterprise-only feature) | Custom roles let organization admins compose new roles by inheriting from the preceding View-Only or Member roles and adding additional permissions to achieve fine-grained access control. Team admins can then assign any of those custom roles to users in their respective teams. For more information, see [Add and manage custom roles](#add-and-manage-custom-roles). |

To change a user's role:

1. Navigate to [https://wandb.ai/home](https://wandb.ai/home).
2. In the upper right corner of the page, select the **User menu** dropdown. From the dropdown, choose **Users**.
3. Provide the name or email of the user in the search bar.
4. Select a role from the **TEAM ROLE** dropdown next to the name of the user.

### Assign or update a user's access

While the organization role controls administrative actions, the seat type controls what a user can do within Models and Weave. Use this procedure when you need to change a user's product-level permissions independent of their organization role.

A user within an organization has one of the following Model seat or Weave access types: full, viewer, or no access.

| Seat type | Description                                                                                                                                                                         |
| --------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Full      | Users with this role type have full permissions to write, read, and export data for Models or Weave.                                                                                |
| Viewer    | A view-only user of your organization. A viewer only has read access to the organization and the underlying teams that they are a part of, and view-only access to Models or Weave. |
| No access | Users with this role have no access to the Models or Weave products.                                                                                                                |

Model seat type and Weave access type are defined at the organization level and inherited by the team. To change a user's seat type, navigate to the organization settings and follow these steps:

1. For Multi-tenant Cloud users, navigate to your organization's settings at `https://wandb.ai/account-settings/<organization>/settings`. Replace the values enclosed in angle brackets (`<>`) with your organization name. For Dedicated Cloud and Self-Managed deployments, navigate to `https://<your-instance>.wandb.io/org/dashboard`.
2. Select the **Users** tab.
3. From the **Role** dropdown, select the seat type you want to assign to the user.

<Note>
  The organization role and subscription type determine which seat types are available within your organization.
</Note>

### Remove a user

1. Navigate to [https://wandb.ai/home](https://wandb.ai/home).
2. In the upper right corner of the page, select the **User menu** dropdown. From the dropdown, choose **Users**.
3. Provide the name or email of the user in the search bar.
4. Select the **action (<Icon icon="ellipsis" />)** menu when it appears.
5. From the dropdown, choose **Remove member**.

### Assign the billing admin

1. Navigate to [https://wandb.ai/home](https://wandb.ai/home).
2. In the upper right corner of the page, select the **User menu** dropdown. From the dropdown, choose **Users**.
3. Provide the name or email of the user in the search bar.
4. Under the **Billing admin** column, choose the user you want to assign as the billing admin.

## Add and manage teams

Teams group related users together so they can collaborate on projects and share resources within the organization. The following sections describe how to create teams, invite users to them, and manage team membership and roles.

Use your organization's dashboard to create and manage teams within your organization. An organization admin or a team admin can:

* Invite users to a team or remove users from a team.
* Manage a team member's roles.
* Automate the addition of users to a team when they join your organization.
* Manage team storage with the team's dashboard at `https://wandb.ai/<team-name>`.

### Create a team

Use your organization's dashboard to create a team:

1. Navigate to [https://wandb.ai/home](https://wandb.ai/home).
2. Select **Create a team to collaborate** on the left navigation panel underneath **Teams**.

<Frame>
  <img alt="Create new team" />
</Frame>

3. Provide a name for your team in the **Team name** field in the modal that appears.
4. Choose a storage type.
5. Select the **Create team** button.

After you select the **Create team** button, W\&B redirects you to a new team page at `https://wandb.ai/<team-name>`, where `<team-name>` consists of the name you provide when you create a team.

Once you have a team, you can add users to that team.

### Invite users to a team

Invite users to a team in your organization. Use the team's dashboard to invite users using their email address or W\&B username if they already have a W\&B account.

1. Navigate to `https://wandb.ai/<team-name>`.
2. Select **Team settings** in the global navigation on the left side of the dashboard.

<Frame>
  <img alt="Team settings" />
</Frame>

3. Select the **Users** tab.
4. Click **Invite a new user**.
5. Within the modal that appears, provide the email of the user in the **Email or username** field and select the role to assign to that user from the **Select a team** role dropdown. For more information about roles a user can have in a team, see [Assign or update a team member's role](#assign-or-update-a-team-members-role).
6. Click the **Send invite** button.

By default, only a team or instance admin can invite members to a team. To change this behavior, see [Team settings](/platform/app/settings-page/teams#privacy).

Besides inviting users manually with email invites, you can automatically add new users to a team if the new user's [email matches the domain of your organization](#domain-capture).

### Match members to a team organization during sign up

Allow new users within your organization to discover teams within your organization when they sign up. New users must have a verified email domain that matches your organization's verified email domain. Verified new users can view a list of verified teams that belong to an organization when they sign up for a W\&B account.

An organization admin must enable domain claiming. To enable domain capture, see the steps described in [Domain capture](#domain-capture).

### Assign or update a team member's role

A team member's role determines what they can do within the team, such as managing other members, contributing content, or viewing data only. Update a team member's role when their responsibilities within the team change.

1. Select the account type icon next to the name of the team member.
2. From the dropdown, choose the account type you want that team member to possess.

This table lists the roles you can assign to a member of a team:

| Role                                   | Definition                                                                                                                                                                                                                                                                                                                                                                                                                                                 |
| -------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Admin                                  | A user who can add and remove other users in the team, change user roles, and configure team settings.                                                                                                                                                                                                                                                                                                                                                     |
| Member                                 | A regular user of a team, invited by email or their organization-level username by the team admin. A member user can't invite other users to the team.                                                                                                                                                                                                                                                                                                     |
| View-Only (Enterprise-only feature)    | A view-only user of a team, invited by email or their organization-level username by the team admin. A view-only user only has read access to the team and its contents.                                                                                                                                                                                                                                                                                   |
| Custom Roles (Enterprise-only feature) | Custom roles let organization admins compose new roles by inheriting from the preceding View-Only or Member roles and adding additional permissions to achieve fine-grained access control. Team admins can then assign any of those custom roles to users in their respective teams. For more information, see the [custom roles announcement](https://wandb.ai/wandb_fc/announcements/reports/Introducing-Custom-Roles-for-W-B-Teams--Vmlldzo2MTMxMjQ3). |

<Info>
  Service accounts aren't users, but rather non-human identities used for automation. **Service accounts** provide API keys for automated workflows and don't consume user licenses. For more information about creating and managing service accounts, see [Use service accounts to automate workflows](/platform/hosting/iam/service-accounts/).
</Info>

<Note>
  Only enterprise licenses on Dedicated Cloud or Self-Managed deployment can assign custom roles to members in a team.
</Note>

### Remove users from a team

Remove a user from a team using the team's dashboard. W\&B preserves runs created in a team, even if the member who created the runs is no longer on that team.

1. Navigate to `https://wandb.ai/<team-name>`.
2. Select **Team settings** in the left navigation bar.
3. Select the **Users** tab.
4. Hover your mouse next to the name of the user you want to delete. Select the **action (<Icon icon="ellipsis" />)** menu when it appears.
5. From the dropdown, select **Remove user**.

## Add and manage custom roles

Custom roles let you tailor permissions beyond the built-in roles when the standard View-Only, Member, and Admin roles don't match your organization's access requirements. Use custom roles to grant specific permissions while still building on a familiar base role.

<Note>
  An Enterprise license is required to create or assign custom roles on Dedicated Cloud or Self-Managed deployments.
</Note>

Organization admins can compose a new role based on either the View-Only or Member role and add additional permissions to achieve fine-grained access control. Team admins can assign a custom role to a team member. You create custom roles at the organization level but assign them at the team level.

To create a custom role:

<Tabs>
  <Tab title="Multi-tenant Cloud">
    1. Navigate to [https://wandb.ai/home](https://wandb.ai/home).
    2. In the upper right corner of the page, select the **User menu** dropdown. Within the **Account** section of the dropdown, select **Settings**.
    3. Click **Roles**.
    4. In the **Custom roles** section, click **Create a role**.
    5. Provide a name for the role. Optionally provide a description.
    6. Choose the role to base the custom role on, either **Viewer** or **Member**.
    7. To add permissions, click the **Search permissions** field, then select one or more permissions to add.
    8. Review the **Custom role permissions** section, which summarizes the permissions the role has.
    9. Click **Create Role**.
  </Tab>

  <Tab title="Dedicated Cloud or Self-Managed">
    1. Navigate to `https://<org-name>.wandb.io/org/settings/`. Replace `<org-name>` with your organization name.
    2. In the **Custom roles** section, click **Create a role**.
    3. Provide a name for the role. Optionally provide a description.
    4. Choose the role to base the custom role on, either **Viewer** or **Member**.
    5. To add permissions, click the **Search permissions** field, then select one or more permissions to add.
    6. Review the **Custom role permissions** section, which summarizes the permissions the role has.
    7. Click **Create Role**.
  </Tab>
</Tabs>

A team admin can now assign the custom role to members of a team from the [Team settings](#invite-users-to-a-team).


# Manage access control for projects
Source: https://docs.wandb.ai/platform/hosting/iam/access-management/restricted-projects

Manage project access using visibility scopes and project-level roles

Define the scope of a W\&B project to limit who can view, edit, and submit W\&B runs to it. This page is for team and organization admins, and project owners, who need to control access to sensitive workflows or limit collaboration to a specific group of users.

You can combine two controls to configure the access level for any project within a W\&B team. **Visibility scope** is the higher-level mechanism. Use it to control which groups of users can view or submit runs in a project. For a project with *Team* or *Restricted* visibility scope, you can then use **Project level roles** to control the level of access that each user has within the project.

<Note>
  The owner of a project, a team admin, or an organization admin can set or edit a project's visibility.
</Note>

## Visibility scopes

Visibility scope determines which users in your organization can see and contribute to a project. You can choose from four project visibility scopes. From most public to most private, they are:

| Scope      | Description                                                                                                                        |
| ---------- | ---------------------------------------------------------------------------------------------------------------------------------- |
| Open       | Anyone who knows about the project can view it and submit runs or reports.                                                         |
| Public     | Anyone who knows about the project can view it. Only your team can submit runs or reports.                                         |
| Team       | Only members of the parent team can view the project and submit runs or reports. Anyone outside the team can't access the project. |
| Restricted | Only invited members from the parent team can view the project and submit runs or reports.                                         |

<Note>
  Set a project's scope to **Restricted** if you want to collaborate on workflows related to sensitive or confidential data. When you create a restricted project within a team, you can invite or add specific members from the team to collaborate on relevant experiments, artifacts, and reports.

  Unlike other project scopes, all members of a team don't get implicit access to a restricted project. At the same time, team admins can join restricted projects if needed.
</Note>

### Set visibility scope on a new or existing project

Set a project's visibility scope when you create a project or when you edit it later. The following sections describe both workflows.

<Note>
  * Only the owner of the project or a team admin can set or edit its visibility scope.
  * When a team admin enables **Make all future team projects private (public sharing not allowed)** within a team's privacy setting, that turns off **Open** and **Public** project visibility scopes for that team. In this case, your team can only use **Team** and **Restricted** scopes.
</Note>

#### Set visibility scope when you create a project

To set the visibility scope for a new project:

1. Navigate to your W\&B organization on a W\&B Multi-tenant Cloud, Dedicated Cloud, or Self-Managed instance.

2. Click the **Create a new project** button in the left-hand sidebar's **My projects** section. Alternatively, navigate to the **Projects** tab of your team and click the **Create new project** button in the upper right-hand corner.

3. After you select the parent team and enter the name of the project, select the desired scope from the **Project Visibility** dropdown.

   <Frame>
     <img alt="Creating restricted project" />
   </Frame>

   Complete the following step if you select **Restricted** visibility.

4. Provide names of one or more W\&B team members in the **Invite team members** field. Add only those members who are essential to collaborate on the project, since other team members don't get implicit access to a restricted project.

   <Frame>
     <img alt="Restricted project configuration" />
   </Frame>

   <Note>
     You can add or remove members in a restricted project later, from its **Users** tab.
   </Note>

W\&B creates the project with the selected visibility scope, and only the invited members (for a restricted project) or in-scope users can access it.

#### Edit visibility scope of an existing project

To change the visibility scope of an existing project:

1. Navigate to your W\&B project.

2. Select the **Overview** tab on the left column.

3. Click the **Edit Project Details** button on the upper right corner.

4. From the **Project Visibility** dropdown, select the desired scope.

   <Frame>
     <img alt="Editing restricted project settings" />
   </Frame>

   Complete the following step if you select **Restricted** visibility.

5. Navigate to the **Users** tab in the project, and click the **Add user** button to invite specific users to the restricted project.

W\&B updates the project's visibility scope, and access reflects the new scope along with any invited members.

<Warning>
  * All members of a team lose access to a project if you change its visibility scope from **Team** to **Restricted**, unless you invite the required team members to the project.
  * All members of a team get access to a project if you change its visibility scope from **Restricted** to **Team**.
  * If you remove a team member from the user list for a restricted project, they lose access to that project.
</Warning>

### Other key things to note for restricted scope

Keep the following behaviors in mind when working with restricted projects:

* If you want to use a team-level service account in a restricted project, you must invite or add it specifically to the project. Otherwise a team-level service account can't access a restricted project by default.
* You can't move runs from a restricted project, but you can move runs from a non-restricted project to a restricted one.
* You can convert the visibility of a restricted project to only **Team** scope, irrespective of the team privacy setting **Make all future team projects private (public sharing not allowed)**.
* If the owner of a restricted project isn't part of the parent team anymore, the team admin must change the owner to maintain access to the project.

## Project level roles

After you set a project's visibility scope, you can further refine each user's permissions inside the project by assigning a project level role. For *Team* or *Restricted* scoped projects in your team, you can assign a specific role to a user, which can differ from that user's team level role. For example, if a user has *Member* role at the team level, you can assign the *View-Only*, *Admin*, or any available custom role to that user within a *Team* or *Restricted* scope project in that team.

<Note>
  Project level roles are in preview on W\&B Multi-tenant Cloud, Dedicated Cloud, and Self-Managed instances.
</Note>

### Assign a project level role to a user

To assign a project level role:

1. Navigate to your W\&B project.
2. Select the **Overview** tab on the left column.
3. Navigate to the **Users** tab in the project.
4. Click the currently assigned role for the relevant user in the **Project Role** field, which opens a dropdown listing the other available roles.
5. Select another role from the dropdown. The change saves instantly.

<Note>
  When you change the project level role for a user to be different from their team level role, the project level role includes a **\*** to indicate the difference.
</Note>

### Other key things to note for project level roles

Keep the following behaviors in mind when assigning or changing project level roles:

* By default, project level roles for all users in a *Team* or *Restricted* scoped project **inherit** their respective team level roles.
* You **can't** change the project level role of a user who has *View-Only* role at the team level.
* If the project level role for a user within a particular project **is the same as** the team level role, and a team admin later changes the team level role, W\&B automatically changes the relevant project role to track the team level role.
* If you change the project level role for a user within a particular project such that **it differs from** the team level role, and a team admin later changes the team level role, the relevant project level role remains as is.
* If you remove a user from a *Restricted* project when their project level role differed from the team level role, and you then add the user back to the project later, they inherit the team level role due to the default behavior. If needed, change the project level role again to differ from the team level role.


# Advanced IAM configuration
Source: https://docs.wandb.ai/platform/hosting/iam/advanced_env_vars

Configure advanced IAM options for W&B with environment variables for SSO, session length, OIDC, and LDAP settings.

In addition to basic [environment variables](../env-vars), you can use environment variables to configure advanced IAM options for your [Dedicated Cloud](/platform/hosting/hosting-options/dedicated-cloud) or [Self-Managed](/platform/hosting/hosting-options/self-managed) instance. Use these variables to customize SSO behavior, session expiration, OIDC and LDAP integration, and other identity-related settings to match your organization's security and access requirements.

Choose any of the following environment variables for your instance depending on your IAM needs.

| Environment variable                | Description                                                                                                                                                                                                                                                                                                                                                                                                            |
| ----------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `DISABLE_SSO_PROVISIONING`          | Set this to `true` to turn off user auto-provisioning in your W\&B instance.                                                                                                                                                                                                                                                                                                                                           |
| `SESSION_LENGTH`                    | To change the default user session expiry time, set this variable to the desired number of hours. For example, set `SESSION_LENGTH` to `24` to configure session expiry time to 24 hours. The default value is 720 hours.                                                                                                                                                                                              |
| `GORILLA_ENABLE_SSO_GROUPS_CLAIMS`  | When you use OIDC-based SSO, set this variable to `true` to automate W\&B team membership in your instance based on your OIDC groups. You must also add a `groups` claim to the user OIDC token, formatted as a string array of all team names the user is part of.                                                                                                                                                    |
| `GORILLA_LDAP_GROUP_SYNC`           | If you use LDAP-based SSO, set it to `true` to automate W\&B team membership in your instance based on your LDAP groups.                                                                                                                                                                                                                                                                                               |
| `GORILLA_OIDC_CUSTOM_SCOPES`        | If you use OIDC-based SSO, you can specify additional [scopes](https://auth0.com/docs/get-started/apis/scopes/openid-connect-scopes) that the W\&B instance requests from your identity provider. These custom scopes don't change the SSO functionality.                                                                                                                                                              |
| `GORILLA_OIDC_SECRET`               | If you use OIDC-based SSO and your IdP requires an OIDC Client Secret, set this variable to the secret.                                                                                                                                                                                                                                                                                                                |
| `GORILLA_USE_IDENTIFIER_CLAIMS`     | If you use OIDC-based SSO, set this variable to `true` to enforce the username and full name of your users using specific OIDC claims from your identity provider. If set, ensure that you configure the enforced username and full name in the `preferred_username` and `name` OIDC claims respectively. Usernames can only contain alphanumeric characters along with underscores and hyphens as special characters. |
| `GORILLA_DISABLE_PERSONAL_ENTITY`   | When set to `true`, turns off [personal entities](/support/models/articles/what-is-the-difference-between-team-and-). Prevents creation of new personal projects in their personal entities and prevents writing to existing personal projects.                                                                                                                                                                        |
| `GORILLA_DISABLE_ADMIN_TEAM_ACCESS` | Set this to `true` to restrict Organization or Instance Admins from self-joining or adding themselves to a W\&B team, ensuring that only Data and AI personas have access to the projects within the teams.                                                                                                                                                                                                            |
| `WANDB_IDENTITY_TOKEN_FILE`         | For [identity federation](/platform/hosting/iam/identity_federation/), the absolute path to the local directory where Java Web Tokens (JWTs) are stored.                                                                                                                                                                                                                                                               |

<Warning>
  W\&B advises caution and understanding all implications before you enable some of these settings, such as `GORILLA_DISABLE_ADMIN_TEAM_ACCESS`. Contact your W\&B team with any questions.
</Warning>


# Automate user and team management
Source: https://docs.wandb.ai/platform/hosting/iam/automate_iam

Automate user and team management at scale in W&B using the SCIM API and the Python SDK API.

This page describes how to automate user and team management at scale using the [SCIM API](/platform/hosting/iam/scim) and the [Python SDK API](/models/ref/python/public-api).

## SCIM API

Use the W\&B SCIM API to manage your W\&B organization's users and teams at scale through an identity provider (IdP) like Okta or Microsoft Entra.

For more information, see [Authentication](./scim#authentication) on the SCIM reference page.

<Tip>
  W\&B's implementation includes endpoints for creating and managing custom roles and for assigning built-in and custom roles. Role endpoints aren't part of the official SCIM schema. W\&B adds role endpoints to support automated management of custom roles.
</Tip>

The following sections describe each category of the SCIM API.

### User SCIM API

The [User SCIM API](./scim#user-resource) lets you create, deactivate, fetch, and list users in a W\&B organization, and assign predefined or custom roles. For complete request and response examples, see the [SCIM reference](./scim#user-management).

<Note>
  To deactivate a user, send `PATCH /scim/Users/{id}` with `{"active": false}`. The hosting option determines the outcome: Dedicated Cloud and Self-Managed deployments retain the user record, while Multi-tenant Cloud removes the user from the organization. Reactivation isn't available in Multi-tenant Cloud. Re-add the user instead. See [Deactivate user](./scim#deactivate-user) and [Reactivate user](./scim#reactivate-user).
</Note>

### Group SCIM API

The [Group SCIM API](./scim#group-resource) lets you manage W\&B teams, including creating or removing teams in an organization. To add or remove users in an existing team, use `PATCH Group`.

<Note>
  W\&B has no notion of a "group of users having the same role." A W\&B team closely resembles a group and lets users with different roles work collaboratively on a set of related projects. Teams can consist of different groups of users. Assign each user in a team a role: team admin, member, viewer, or a custom role.

  W\&B maps Group SCIM API endpoints to W\&B teams because of the similarity between groups and W\&B teams.
</Note>

### Custom role SCIM API

The [Custom Role SCIM API](./scim#role-resource) lets you manage custom roles, including creating, listing, or updating custom roles in an organization.

<Warning>
  Delete a custom role with caution.

  To delete a custom role within a W\&B organization, use the `DELETE Role` endpoint. W\&B assigns the inherited predefined role to all users who had the custom role before deletion.

  To update the inherited role for a custom role, use the `PUT Role` endpoint. This operation doesn't affect any existing non-inherited custom permissions in the custom role.
</Warning>

## W\&B Python SDK API

Use the [W\&B Python SDK API](/models/ref/python/public-api/api) to manage organization users, teams, and team membership through the following classes:

* [`User`](/models/ref/python/public-api/user)
* [`Team`](/models/ref/python/public-api/team)
* [`Member`](/models/ref/python/public-api/member)


# Use federated identities with SDK
Source: https://docs.wandb.ai/platform/hosting/iam/identity_federation

Use identity federation with JSON Web Tokens (JWTs) to authenticate with the W&B SDK and CLI without API keys.

Use identity federation to sign in to the W\&B SDK and CLI with your organizational credentials, instead of using a long-lived API key. If your W\&B organization admin has configured SSO for your organization, you already use those credentials to sign in to the W\&B app UI. Identity federation is like SSO for the W\&B SDK, but uses JSON Web Tokens (JWTs) directly. Use identity federation as an alternative to API keys.

This page is for organization admins who configure the JWT issuer for a W\&B organization. It's also for users or service accounts that authenticate to W\&B using JWTs.

[RFC 7523](https://datatracker.ietf.org/doc/html/rfc7523) forms the underlying basis for identity federation with SDK.

<Note>
  Identity federation is available in preview for Multi-tenant Cloud, Dedicated Cloud, and Self-Managed. An [Enterprise license](/platform/hosting/enterprise-licenses) is required. For details or assistance, contact your AISE or [support](mailto:support@wandb.com).
</Note>

<Note>
  This document uses the terms "identity provider" and "JWT issuer" interchangeably. Both refer to the same thing in the context of this capability.
</Note>

## Set up the JWT issuer

Before users can authenticate with JWTs, an organization admin must set up a federation between your W\&B organization and a publicly accessible JWT issuer.

1. Navigate to the **Settings** tab in your organization dashboard.
2. In the **Authentication** option, click **Set up JWT Issuer**.
3. Add the JWT issuer URL in the text box and click **Create**.

W\&B automatically looks for an OIDC discovery document at the path `${ISSUER_URL}/.well-known/openid-configuration`. From the discovery document, W\&B locates the JSON Web Key Set (JWKS) at the relevant URL. W\&B uses the JWKS for real-time validation of the JWTs to ensure that the relevant identity provider issued them.

After this step, your W\&B organization is federated with the JWT issuer. Users in your organization can then authenticate to W\&B using JWTs issued by that provider.

## Use the JWT to access W\&B

After an organization admin sets up a JWT issuer, users can start accessing W\&B projects using JWTs issued by that identity provider. The mechanism for using JWTs is as follows:

1. You must sign in to the identity provider using one of the mechanisms available in your organization. You can access some providers in an automated manner using an API or SDK, while others are only accessible through a relevant UI. Contact your W\&B organization admin or the owner of the JWT issuer for details.
2. After you've retrieved the JWT by signing in to your identity provider, store it in a file at a secure location. Configure the absolute file path in an environment variable `WANDB_IDENTITY_TOKEN_FILE`.
3. Access your W\&B project using the W\&B SDK or CLI. The SDK or CLI automatically detects the JWT and exchanges it for a W\&B access token after validating the JWT. The W\&B access token grants access to the relevant APIs for enabling your AI workflows, such as logging runs, metrics, and artifacts. By default, the access token is stored at the path `~/.config/wandb/credentials.json`. You can change that path by specifying the environment variable `WANDB_CREDENTIALS_FILE`.

<Note>
  JWTs are short-lived credentials that address the shortcomings of long-lived credentials such as API keys and passwords. The JWT expiry time depends on your identity provider's configuration. Refresh the JWT before it expires, and ensure that it's stored in the file referenced by the environment variable `WANDB_IDENTITY_TOKEN_FILE`.

  The W\&B access token also has a default expiry duration, after which the SDK or CLI tries to refresh it using your JWT. If the user JWT has also expired by that time and isn't refreshed, authentication fails. If possible, implement the JWT retrieval and post-expiry refresh mechanism as part of the AI workload that uses the W\&B SDK or CLI.
</Note>

### JWT validation

To ensure that only valid tokens grant access, the JWT undergoes the following validations. These validations run when the SDK or CLI exchanges the JWT for a W\&B access token and then accesses a project:

* W\&B verifies the JWT signature using the JWKS at the W\&B organization level. This is the first line of defense, and if this fails, that means there's a problem with your JWKS or how your JWT is signed.

* The `iss` claim in the JWT must equal the issuer URL configured at the organization level.

* The `sub` claim in the JWT must equal the user's email address as configured in the W\&B organization.

* The `aud` claim in the JWT must equal the name of the W\&B organization that houses the project you're accessing as part of your AI workflow.

  On [Dedicated Cloud](/platform/hosting/hosting-options/dedicated-cloud) or [Self-Managed](/platform/hosting/hosting-options/self-managed) instances:

  * To skip audience validation, you can set the environment variable `FEDERATED_AUTH_AUDIENCES` to `wandb`.
  * Some organizations have specific requirements for the audience. To customize the `aud` value, set the environment variable `FEDERATED_AUTH_AUDIENCES` to a string with a comma-separated list of audience values.

* W\&B checks the `exp` claim in the JWT to determine whether the token is valid or has expired and needs to be refreshed.

## External service accounts

W\&B has long supported built-in service accounts with long-lived API keys. With identity federation for the SDK and CLI, you can also bring external service accounts that use JWTs for authentication. The organization's configured issuer must issue those JWTs. A team admin can configure external service accounts within the scope of a team, like the built-in service accounts.

To configure an external service account, a team admin must:

1. Navigate to the **Service Accounts** tab for your team.
2. Click **New service account**.
3. Provide a name for the service account.
4. Select **Federated Identity** as the **Authentication Method**, then provide a **Subject**. See [Determine the Subject value for your identity provider](#determine-the-subject-value-for-your-identity-provider).
5. Click **Create**.

After this step, the external service account is registered with the team and can use JWTs issued by the configured identity provider to access W\&B. The `sub` claim in the external service account's JWT must equal the subject configured by the team admin in the team-level **Service Accounts** tab. W\&B verifies that claim as part of [JWT validation](#jwt-validation). The `aud` claim requirement is similar to that for human user JWTs.

When [using an external service account's JWT to access W\&B](#use-the-jwt-to-access-wb), it's often easier to automate the workflow. Automation generates the initial JWT and refreshes it as needed. To attribute runs logged using an external service account to a human user, configure the environment variables `WANDB_USERNAME` or `WANDB_USER_EMAIL` for your AI workflow, similar to built-in service accounts.

<Note>
  W\&B recommends using a mix of built-in and external service accounts across your AI workloads with different levels of data sensitivity. This mix strikes a balance between flexibility and simplicity.
</Note>

### Determine the Subject value for your identity provider

The value you enter for **Subject** must exactly match the `sub` (subject) claim in JWTs issued by your IdP for the service account. W\&B applies the same comparison for every identity provider. The match is exact, case-sensitive, and whitespace-sensitive, so even a trailing space or a difference in capitalization causes authentication to fail.

The W\&B App accepts any non-empty **Subject** without validating its value, since the value is entirely dependent on the IdP. W\&B can't detect an incorrect value when you create the service account. Instead, authentication fails later, when the service account presents its JWT.

The most reliable way to determine the correct value is to read it from a real token. Obtain a sample JWT issued for the service account, base64url-decode its payload (the middle segment, between the two dots) locally, and copy the `sub` value verbatim into the **Subject** field. Don't paste JWTs into third-party online decoders, because a JWT is a credential.

The value differs by provider. The following are typical, but always confirm them against an actual token:

| Identity provider  | Where to find the `sub` value                                                                                                                                                                                                                                                                                                                                |
| ------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| Microsoft Entra ID | The service principal's **Object ID**, found under **Enterprise Applications** in the [Microsoft Entra admin center](https://entra.microsoft.com). Use the **Enterprise Application** (service principal) Object ID, not the **App registration** Object ID. For app-only (client credentials) tokens, this is typically the value Entra ID places in `sub`. |
| Google Cloud (GCP) | The `sub` value from the ID token that Google issues for the service account.                                                                                                                                                                                                                                                                                |


# Configure SSO with LDAP
Source: https://docs.wandb.ai/platform/hosting/iam/ldap

Configure LDAP-based SSO authentication for W&B Server including connection parameters and environment variables.

This guide is for W\&B Admins who enable LDAP-based single sign-on (SSO) for W\&B Server, so that users can authenticate against an existing LDAP directory instead of managing separate W\&B credentials. It explains how to configure the LDAP connection from the W\&B App system settings UI or with environment variables. It also describes the required and optional configuration parameters, including the address, base distinguished name, and attributes. You can set up either an anonymous bind, or bind with an administrator DN and password.

<Note>
  Only W\&B Admin roles can enable and configure LDAP authentication.
</Note>

## Configure LDAP connection

Choose one of the following methods to configure the LDAP connection. Use the W\&B App tab to configure LDAP through the system settings UI, or use the Environment variable tab to configure LDAP at deployment time.

<Tabs>
  <Tab title="W&B App">
    To configure LDAP through the W\&B App:

    1. Go to the W\&B App.
    2. Select your profile icon, then select **System Settings**.
    3. Toggle **Configure LDAP Client**.
    4. Add the details in the form. For more information about each input, see [Configuration parameters](#configuration-parameters).
    5. Click **Update Settings** to test your settings. This step establishes a test client or connection with W\&B Server.
    6. If your connection is verified, toggle **Enable LDAP Authentication** and select **Update Settings**.

    After you complete these steps, W\&B Server authenticates users against your LDAP directory.
  </Tab>

  <Tab title="Environment variable">
    Set an LDAP connection with the following environment variables:

    | Environment variable          | Required | Example                         |
    | ----------------------------- | -------- | ------------------------------- |
    | `LOCAL_LDAP_ADDRESS`          | Yes      | `ldaps://ldap.example.com:636`  |
    | `LOCAL_LDAP_BASE_DN`          | Yes      | `email=mail,group=gidNumber`    |
    | `LOCAL_LDAP_BIND_DN`          | No       | `cn=admin`, `dc=example,dc=org` |
    | `LOCAL_LDAP_BIND_PW`          | No       |                                 |
    | `LOCAL_LDAP_ATTRIBUTES`       | Yes      | `email=mail`, `group=gidNumber` |
    | `LOCAL_LDAP_TLS_ENABLE`       | No       |                                 |
    | `LOCAL_LDAP_GROUP_ALLOW_LIST` | No       |                                 |
    | `LOCAL_LDAP_LOGIN`            | No       |                                 |

    The following Configuration parameters section defines each environment variable. For clarity, the definition names omit the environment variable prefix `LOCAL_LDAP`.
  </Tab>
</Tabs>

## Configuration parameters

The following table lists and describes required and optional LDAP configurations.

| Environment variable | Definition                                                                                                                                                                                                                                                      | Required |
| -------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------- |
| `ADDRESS`            | The address of your LDAP server within the VPC that hosts W\&B Server.                                                                                                                                                                                          | Yes      |
| `BASE_DN`            | The root path that searches start from, required for any queries into this directory.                                                                                                                                                                           | Yes      |
| `BIND_DN`            | Path of the administrative user registered in the LDAP server. Required if the LDAP server doesn't support unauthenticated binding. If specified, W\&B Server connects to the LDAP server as this user. Otherwise, W\&B Server connects with anonymous binding. | No       |
| `BIND_PW`            | The password for the administrative user, used to authenticate the binding. If left blank, W\&B Server connects with anonymous binding.                                                                                                                         | No       |
| `ATTRIBUTES`         | Provide email and group ID attribute names as comma-separated string values.                                                                                                                                                                                    | Yes      |
| `TLS_ENABLE`         | Enable TLS.                                                                                                                                                                                                                                                     | No       |
| `GROUP_ALLOW_LIST`   | Group allowlist.                                                                                                                                                                                                                                                | No       |
| `LOGIN`              | Instructs W\&B Server to use LDAP to authenticate. Set to either `True` or `False`. Optionally set this to `False` to test the LDAP configuration. Set this to `True` to start LDAP authentication.                                                             | No       |


# Identity and access management (IAM)
Source: https://docs.wandb.ai/platform/hosting/iam/org_team_struct

Understand the three IAM scopes in W&B (organizations, teams, and projects) and how they map to your company.

W\&B Platform has three IAM scopes within W\&B: [Organizations](#organization), [Teams](#team), and [Projects](#project). Understanding how these scopes nest helps you plan how to structure access, group users, and organize AI work across your company.

The following sections describe each scope and how it maps to your company's structure, and link to detailed management guides.

## Organization

An *Organization* is the root scope in your W\&B account or instance. All actions in your account or instance take place within the context of that root scope, including managing users, managing teams, managing projects within teams, and tracking usage.

If you use [Multi-tenant Cloud](/platform/hosting/hosting-options/multi_tenant_cloud), you may have more than one organization where each may correspond to a business unit, a personal user, or a joint partnership with another business.

If you use [Dedicated Cloud](/platform/hosting/hosting-options/dedicated-cloud) or a [Self-Managed instance](/platform/hosting/hosting-options/self-managed), it corresponds to one organization. Your company may have more than one Dedicated Cloud or Self-Managed instance to map to different business units or departments, though that's strictly an optional way to manage AI practitioners across your businesses or departments.

For more information, see [Manage organizations](./access-management/manage-organization).

## Team

A *Team* is a subscope within an organization that may map to a business unit, function, department, or project team in your company. You may have more than one team in your organization depending on your deployment type and pricing plan.

AI projects are organized within the context of a team. Team admins govern access control within a team, and they may or may not be admins at the parent organization level.

For more information, see [Add and manage teams](./access-management/manage-organization#add-and-manage-teams).

## Project

A *Project* is a subscope within a team that maps to an AI project with specific intended outcomes. You may have more than one project within a team. Each project has a visibility mode that determines who can access it.

Within each project, your team's work is organized into the following building blocks. Every project comprises [Workspaces](/models/track/workspaces/) and [Reports](/models/reports/), and links to relevant [Artifacts](/models/artifacts/), [Sweeps](/models/sweeps/), and [Automations](/models/automations/).


# Manage users, groups, and roles with SCIM
Source: https://docs.wandb.ai/platform/hosting/iam/scim

Use the SCIM API to manage users, groups, and custom roles in a W&B organization with automated provisioning.

<Note>
  Watch a [video demonstrating SCIM in action](https://www.youtube.com/watch?v=Nw3QBqV0I-o) (12 min)
</Note>

## Overview

This page describes how instance and organization admins use the System for Cross-domain Identity Management (SCIM) API to automate identity management in W\&B. With the SCIM API, you can provision and deprovision users, manage team membership, and define custom roles programmatically through an identity provider or CI/CD pipeline instead of clicking through the W\&B App. SCIM groups map to W\&B Teams.

W\&B's SCIM API is compatible with identity providers such as Okta. For SSO configuration with Okta and other identity providers, see the [SSO documentation](/platform/hosting/iam/sso/).

For practical Python examples that demonstrate how to interact with the SCIM API, visit the [`wandb-scim`](https://github.com/wandb/examples/tree/master/wandb-scim) repository.

### Supported features

The SCIM API supports the following features:

* **Filtering**: The API supports filtering for `/Users` and `/Groups` endpoints.
* **PATCH operations**: Supports `PATCH` for partial resource updates.
* **ETag support**: Conditional updates using ETags for conflict detection.
* **Service account authentication**: Organization service accounts can access the API.
* **Service account lifecycle**: Provision and deprovision [team-scoped and organization-scoped service accounts](/platform/hosting/iam/service-accounts). Supported on **Multi-tenant Cloud** and on **Dedicated Cloud** and **Self-Managed** v0.81.0+.

<Note>
  If you're an admin of multiple Enterprise [Multi-tenant Cloud](/platform/hosting/hosting-options/multi_tenant_cloud) organizations, configure the organization that receives SCIM API requests so that requests made with your API key affect the correct organization. Click your profile image, click **User Settings**, then check the **Default API organization** setting.

  The chosen hosting option determines the value for the `[HOST-URL]` placeholder used in the examples on this page.

  Examples use user IDs such as `abc` and `def`. Real requests and responses use hashed values for user IDs.
</Note>

## Authentication

Every SCIM request must be authenticated as an admin principal. Organization admins can authenticate with a **Bearer token** or **HTTP Basic** credentials. Either style uses the *same API key string* where a key applies. Choose a user identity or an organization-scoped service account after reviewing the key differences in the following section.

### Key differences

The following list compares user credentials and service account credentials for SCIM authentication:

* Who should use it: Users are best for interactive, one-off admin actions. Service accounts are best for automation and integrations (CI/CD, provisioning tools).
* Credentials: Users send username and API key for Basic auth. Service accounts send only an API key (no username) for Basic auth. For Bearer auth, send only the API key in the header (no username).
* Bearer versus Basic: Bearer uses `Authorization: Bearer [API-KEY]` with the key verbatim. Basic uses `Authorization: Basic <base64(...)>` (users encode `username:API-KEY`, and service accounts encode `:API-KEY` with a leading colon and empty username).
* Scope and permissions: Use an API key from an instance or organization admin user, or from an [organization-scoped service account](/platform/hosting/iam/service-accounts/#organization-scoped-service-accounts). Keys from [team-scoped service accounts](/platform/hosting/iam/service-accounts/#team-scoped-service-accounts) can't authenticate to the SCIM API. Service accounts that use SCIM are organization-scoped and headless, which supports clearer audit trails for automation.
* Where to get credentials: Users copy their API key from User Settings. Organization-scoped service account keys are in the organization dashboard under the **Service account** tab.
* Multi-tenant Cloud: If you have access to more than one Multi-tenant Cloud organization, you must set the Default API organization to ensure that SCIM API calls are routed to the intended organization.

### Bearer token

Send the API key as a Bearer token:

```bash theme={null}
Authorization: Bearer [API-KEY]
```

The `[API-KEY]` value is the same string you'd use as the password in HTTP Basic authentication for that principal. Don't Base64-encode the key for Bearer requests.

<Note>
  Bearer authentication for the SCIM API is available in W\&B Multi-tenant Cloud, and in Dedicated Cloud and Self-Managed v0.79.0 and later.
</Note>

The following examples use `[API-KEY]` as a placeholder. Replace it with a real key from an admin user or an organization-scoped service account.

**List users**

```bash theme={null}
curl -s -S \
  -H "Authorization: Bearer [API-KEY]" \
  -H "Content-Type: application/scim+json" \
  "[HOST-URL]/scim/Users"
```

**Create a user**

```bash theme={null}
curl -s -S -X POST \
  -H "Authorization: Bearer [API-KEY]" \
  -H "Content-Type: application/scim+json" \
  "[HOST-URL]/scim/Users" \
  -d '{
    "schemas": ["urn:ietf:params:scim:schemas:core:2.0:User"],
    "userName": "dev-user2",
    "emails": [{"primary": true, "value": "dev-user2@example.com"}]
  }'
```

For more information, see [Create user](#create-user).

### Users

Use your personal admin credentials when you perform interactive admin tasks. Construct the HTTP `Authorization` header as `Basic <base64(username:API-KEY)>`.

For example, authorize as `demo:p@55w0rd`:

```bash theme={null}
Authorization: Basic ZGVtbzpwQDU1dzByZA==
```

### Service accounts

Use an organization-scoped service account for automation or integrations. Construct the HTTP `Authorization` header as `Basic <base64(:API-KEY)>` (note the leading colon and empty username). Find service account API keys in the organization dashboard under the **Service account** tab. Refer to [Organization-scoped service accounts](/platform/hosting/iam/service-accounts/#organization-scoped-service-accounts).

For example, authorize with API key `sa-p@55w0rd`:

```bash theme={null}
Authorization: Basic OnNhLXBANTV3MHJk
```

## User management

The SCIM user resource maps to W\&B users and service accounts. Use the endpoints in this section to provision, update, and remove users and service accounts in your organization, for example, when you onboard new employees, rotate service credentials, or remove access for departing users.

For service account concepts and UI workflows, see [Use service accounts to automate workflows](/platform/hosting/iam/service-accounts).

<Note>
  **Breaking change for integrations that parse SCIM User JSON**

  * In Dedicated Cloud and Self-Managed v0.80.1+, and in Multi-tenant Cloud deployments after April 30, 2026, responses from `/scim/Users` (including `GET` user, `GET` users, and `PATCH` responses that return a User) serialize `emails` as a JSON array of objects using lowercase field names (`value`, `primary`, and optional `type` or `display`), matching SCIM 2.0.
  * Deployments on older releases return `emails` as a single JSON object with PascalCase keys (`Value`, `Primary`, and similar).

  If your code reads `emails` from SCIM *responses*, treat `emails` as an array and read the primary entry (or the first element).

  Request bodies for creating or updating users already used the array form and are unchanged. The `list-users` filter `emails.value eq "..."` is also unchanged.
</Note>

### Get user

Retrieves information for a specific user or service account in your organization by user ID, or for a user by email address.

Service account responses include `accountType` (`SERVICE` for team-scoped service accounts, `ORG_SERVICE` for organization-scoped service accounts). Service accounts don't include `emails`.

#### Endpoint

* **URL**: `[HOST-URL]/scim/Users/{id}`
* **Method**: `GET`

#### Parameters

| Parameter | Type   | Required | Description               |
| --------- | ------ | -------- | ------------------------- |
| `id`      | string | Yes      | The unique ID of the user |

#### Example

<Tabs>
  <Tab title="Get User Request">
    ```bash theme={null}
    GET /scim/Users/abc
    ```
  </Tab>

  <Tab title="Get User Response">
    ```text theme={null}
    (Status 200)
    ```

    ```json theme={null}
    {
        "active": true,
        "daysActive": 42,
        "displayName": "Dev User 1",
        "emails": [
            {
                "primary": true,
                "value": "dev-user1@example.com"
            }
        ],
        "id": "abc",
        "lastActiveAt": "2023-10-15T14:32:10Z",
        "meta": {
            "resourceType": "User",
            "created": "2023-10-01T00:00:00Z",
            "lastModified": "2023-10-01T00:00:00Z",
            "location": "Users/abc"
        },
        "schemas": [
            "urn:ietf:params:scim:schemas:core:2.0:User"
        ],
        "userName": "dev-user1"
    }
    ```

    The response includes details about the user's activity in the organization:

    * **`daysActive`**: Total number of days the user has been active in the organization.
    * **`lastActiveAt`**: ISO 8601 timestamp of the user's most recent activity. Returns `null` if the user hasn't been active.

    The definition of "active" differs by deployment type:

    * **Dedicated Cloud / Self-Managed**: A user is active if they sign in, open any page in the W\&B App, log runs, use the SDK, or interact with the W\&B server in any way.
    * **Multi-tenant Cloud**: A user is active if they perform any auditable action scoped to the organization after May 8, 2025. See [Audit logging actions](/platform/hosting/monitoring-usage/audit-logging#actions) for the full list.
  </Tab>
</Tabs>

### List users

Retrieves a list of all users and service accounts in your organization. Each resource includes `accountType` (`USER`, `SERVICE`, or `ORG_SERVICE`).

#### Filter users

The `/Users` endpoint supports filtering users by username or email:

* `userName eq "value"`: Filter by username.
* `emails.value eq "value"`: Filter by email address.

##### Example

```bash theme={null}
GET /scim/Users?filter=userName eq "john.doe"
GET /scim/Users?filter=emails.value eq "john@example.com"
```

#### Endpoint

* **URL**: `[HOST-URL]/scim/Users`
* **Method**: `GET`

#### Example

<Tabs>
  <Tab title="List Users Request">
    ```bash theme={null}
    GET /scim/Users
    ```
  </Tab>

  <Tab title="List Users Response">
    ```text theme={null}
    (Status 200)
    ```

    ```json theme={null}
    {
        "Resources": [
            {
                "active": true,
                "daysActive": 42,
                "displayName": "Dev User 1",
                "emails": [
                    {
                        "primary": true,
                        "value": "dev-user1@example.com"
                    }
                ],
                "id": "abc",
                "lastActiveAt": "2023-10-15T14:32:10Z",
                "meta": {
                    "resourceType": "User",
                    "created": "2023-10-01T00:00:00Z",
                    "lastModified": "2023-10-01T00:00:00Z",
                    "location": "Users/abc"
                },
                "schemas": [
                    "urn:ietf:params:scim:schemas:core:2.0:User"
                ],
                "userName": "dev-user1"
            }
        ],
        "itemsPerPage": 9999,
        "schemas": [
            "urn:ietf:params:scim:api:messages:2.0:ListResponse"
        ],
        "startIndex": 1,
        "totalResults": 1
    }
    ```

    The response includes details about each user's activity in the organization:

    * **`daysActive`**: Total number of days the user has been active in the organization.
    * **`lastActiveAt`**: ISO 8601 timestamp of the user's most recent activity. Returns `null` if the user hasn't been active.

    The definition of "active" differs by deployment type:

    * **Dedicated Cloud / Self-Managed**: A user is active if they sign in, open any page in the W\&B App, log runs, use the SDK, or interact with the W\&B server in any way.
    * **Multi-tenant Cloud**: A user is active if they perform any auditable action scoped to the organization after May 8, 2025. See [Audit logging actions](/platform/hosting/monitoring-usage/audit-logging#actions) for the full list.
  </Tab>
</Tabs>

### Create user

Creates a new user in your organization.

#### Endpoint

* **URL**: `[HOST-URL]/scim/Users`
* **Method**: `POST`

#### Parameters

| Parameter    | Type   | Required | Description                                                                |
| ------------ | ------ | -------- | -------------------------------------------------------------------------- |
| `emails`     | array  | Yes      | Array of email objects. Must include a primary email                       |
| `userName`   | string | Yes      | The username for the new user                                              |
| `modelsSeat` | string | No       | Models seat level. One of `full`, `viewer`, or `none`. Defaults to `full`. |
| `weaveRole`  | string | No       | Weave role level. One of `full`, `viewer`, or `none`. Defaults to `full`.  |

#### Example

<Tabs>
  <Tab title="Create User Request (Dedicated/Self-Managed)">
    ```bash theme={null}
    POST /scim/Users
    ```

    ```json theme={null}
    {
        "schemas": [
            "urn:ietf:params:scim:schemas:core:2.0:User"
        ],
        "emails": [
            {
                "primary": true,
                "value": "dev-user2@example.com"
            }
        ],
        "userName": "dev-user2",
        "modelsSeat": "full",
        "weaveRole": "full"
    }
    ```
  </Tab>

  <Tab title="Create User Request (Multi-tenant)">
    ```bash theme={null}
    POST /scim/Users
    ```

    ```json theme={null}
    {
        "schemas": [
            "urn:ietf:params:scim:schemas:core:2.0:User",
            "urn:ietf:params:scim:schemas:extension:teams:2.0:User"
        ],
        "emails": [
            {
                "primary": true,
                "value": "dev-user2@example.com"
            }
        ],
        "userName": "dev-user2",
        "modelsSeat": "full",
        "weaveRole": "full",
        "urn:ietf:params:scim:schemas:extension:teams:2.0:User": {
            "teams": ["my-team"]
        }
    }
    ```
  </Tab>
</Tabs>

#### Response

<Tabs>
  <Tab title="Create User Response (Dedicated/Self-Managed)">
    ```text theme={null}
    (Status 201)
    ```

    ```json theme={null}
    {
        "active": true,
        "displayName": "Dev User 2",
        "emails": [
            {
                "primary": true,
                "value": "dev-user2@example.com"
            }
        ],
        "id": "def",
        "meta": {
            "resourceType": "User",
            "created": "2023-10-01T00:00:00Z",
            "location": "Users/def"
        },
        "schemas": [
            "urn:ietf:params:scim:schemas:core:2.0:User"
        ],
        "modelsSeat": "full",
        "weaveRole": "full",
        "userName": "dev-user2"
    }
    ```
  </Tab>

  <Tab title="Create User Response (Multi-tenant)">
    ```text theme={null}
    (Status 201)
    ```

    ```json theme={null}
    {
        "active": true,
        "displayName": "Dev User 2",
        "emails": [
            {
                "primary": true,
                "value": "dev-user2@example.com"
            }
        ],
        "id": "def",
        "meta": {
            "resourceType": "User",
            "created": "2023-10-01T00:00:00Z",
            "location": "Users/def"
        },
        "schemas": [
            "urn:ietf:params:scim:schemas:core:2.0:User",
            "urn:ietf:params:scim:schemas:extension:teams:2.0:User"
        ],
        "userName": "dev-user2",
        "organizationRole": "member",
        "modelsSeat": "full",
        "weaveRole": "full",
        "teamRoles": [
            {
                "teamName": "my-team",
                "roleName": "member"
            }
        ],
        "groups": [
            {
                "value": "my-team-id"
            }
        ]
    }
    ```
  </Tab>
</Tabs>

### Provision service account

Creates a team-scoped or organization-scoped service account in your organization. Use this endpoint to create headless identities for automation, CI/CD, or integrations that shouldn't be tied to a human user. Omit `accountType` to create a regular user instead. See [Create user](#create-user).

<Note>
  Available in **Dedicated Cloud** and **Self-Managed** v0.81.0+ and in **Multi-tenant Cloud**.

  * Set `userName` to the service account name. The API uses `userName` for the account's display name. The `displayName` field in the request body is ignored.
  * `emails` aren't required for service accounts.
  * `modelsSeat` and `weaveRole` aren't supported on create and return `400 Bad Request` if present.
  * Service accounts can't be updated with `PATCH` or `PUT`, can't be deactivated, and can't be assigned organization, team, or registry roles through SCIM. Create API keys in the W\&B App after provisioning.
</Note>

#### Endpoint

* **URL**: `[HOST-URL]/scim/Users`
* **Method**: `POST`

#### Parameters

| Parameter                                               | Type   | Required | Description                                                                                                                                                                                                                                                                                         |
| ------------------------------------------------------- | ------ | -------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `userName`                                              | string | Yes      | Unique name for the service account.                                                                                                                                                                                                                                                                |
| `accountType`                                           | string | Yes      | `SERVICE` for a [team-scoped service account](/platform/hosting/iam/service-accounts/#team-scoped-service-accounts), or `ORG_SERVICE` for an [organization-scoped service account](/platform/hosting/iam/service-accounts/#organization-scoped-service-accounts).                                   |
| `urn:ietf:params:scim:schemas:extension:teams:2.0:User` | object | Yes      | Teams extension object.                                                                                                                                                                                                                                                                             |
| `defaultTeam`                                           | string | Yes      | Sub-field of the teams extension. Name of an existing W\&B Team. The service account is created as a member of this team. For team-scoped service accounts, this is the only team they join. Organization-scoped service accounts are also added automatically to teams created later through SCIM. |
| `teams`                                                 | array  | No       | **Multi-tenant Cloud only.** Team names to add the account to. Include the same team as `defaultTeam` when you use this field.                                                                                                                                                                      |

#### Example

<Tabs>
  <Tab title="Provision Team Service Account Request">
    ```bash theme={null}
    POST /scim/Users
    ```

    ```json theme={null}
    {
        "schemas": [
            "urn:ietf:params:scim:schemas:core:2.0:User",
            "urn:ietf:params:scim:schemas:extension:teams:2.0:User"
        ],
        "userName": "sa-deploy-bot",
        "accountType": "SERVICE",
        "urn:ietf:params:scim:schemas:extension:teams:2.0:User": {
            "defaultTeam": "ml-platform"
        }
    }
    ```
  </Tab>

  <Tab title="Provision Org Service Account Request">
    ```bash theme={null}
    POST /scim/Users
    ```

    ```json theme={null}
    {
        "schemas": [
            "urn:ietf:params:scim:schemas:core:2.0:User",
            "urn:ietf:params:scim:schemas:extension:teams:2.0:User"
        ],
        "userName": "sa-ci-runner",
        "accountType": "ORG_SERVICE",
        "urn:ietf:params:scim:schemas:extension:teams:2.0:User": {
            "defaultTeam": "ml-platform"
        }
    }
    ```
  </Tab>
</Tabs>

#### Response

<Tabs>
  <Tab title="Provision Team Service Account Response">
    ```text theme={null}
    (Status 201)
    ```

    ```json theme={null}
    {
        "accountType": "SERVICE",
        "active": true,
        "displayName": "sa-deploy-bot",
        "id": "xyz",
        "meta": {
            "resourceType": "User",
            "created": "2023-10-01T00:00:00Z",
            "location": "Users/xyz"
        },
        "organizationRole": "member",
        "schemas": [
            "urn:ietf:params:scim:schemas:core:2.0:User",
            "urn:ietf:params:scim:schemas:extension:wandb:2.0:User"
        ],
        "teamRoles": [
            {
                "teamName": "ml-platform",
                "roleName": "member"
            }
        ],
        "urn:ietf:params:scim:schemas:extension:wandb:2.0:User": {
            "organizationRole": "member"
        },
        "userName": "sa-deploy-bot"
    }
    ```
  </Tab>

  <Tab title="Provision Org Service Account Response">
    ```text theme={null}
    (Status 201)
    ```

    ```json theme={null}
    {
        "accountType": "ORG_SERVICE",
        "active": true,
        "displayName": "sa-ci-runner",
        "id": "xyz",
        "meta": {
            "resourceType": "User",
            "created": "2023-10-01T00:00:00Z",
            "location": "Users/xyz"
        },
        "organizationRole": "member",
        "schemas": [
            "urn:ietf:params:scim:schemas:core:2.0:User",
            "urn:ietf:params:scim:schemas:extension:wandb:2.0:User"
        ],
        "teamRoles": [
            {
                "teamName": "ml-platform",
                "roleName": "member"
            }
        ],
        "urn:ietf:params:scim:schemas:extension:wandb:2.0:User": {
            "organizationRole": "member"
        },
        "userName": "sa-ci-runner"
    }
    ```
  </Tab>
</Tabs>

For an organization-scoped service account, `accountType` is `ORG_SERVICE`.

In **Self-Managed** deployments, `organizationRole` is `service` or `org_service` instead of `member`, matching the account type.

If the response returns one of the following errors, check the request for these common problems:

* `409 Conflict`: The request includes duplicate `userName` keys for the same service account.
* `400 Bad Request`: The request is missing `defaultTeam` or sets it to an invalid value.

### Deprovision service account

Permanently deletes a service account and its organization membership. Use this endpoint when a service account is no longer needed (for example, after you retire an automation pipeline). This is a hard delete, and the account can't be reactivated through SCIM.

<Note>
  Available in **Dedicated Cloud** and **Self-Managed** v0.81.0+ and in **Multi-tenant Cloud**. Use the service account's SCIM user `id` from the provision response or from [Get user](#get-user). Deprovisioning doesn't delete API keys that were already issued. Revoke keys separately in the W\&B App if needed.
</Note>

#### Endpoint

* **URL**: `[HOST-URL]/scim/Users/{id}`
* **Method**: `DELETE`

#### Parameters

| Parameter | Type   | Required | Description                           |
| --------- | ------ | -------- | ------------------------------------- |
| `id`      | string | Yes      | The unique ID of the service account. |

#### Example

<Tabs>
  <Tab title="Deprovision Service Account Request">
    ```bash theme={null}
    DELETE /scim/Users/xyz
    ```
  </Tab>

  <Tab title="Deprovision Service Account Response">
    ```text theme={null}
    (Status 204)
    ```
  </Tab>
</Tabs>

### Delete user

<Warning>
  **Maintain admin access**

  Ensure that at least one admin user always exists in your instance or organization. Otherwise, no user can configure or maintain your organization's W\&B account. If an organization uses SCIM or another automated process to deprovision users from W\&B, a deprovisioning operation could inadvertently remove the last remaining admin from the instance or organization.

  For assistance with developing operational procedures, or to restore admin access, contact [support](mailto:support@wandb.com).
</Warning>

Fully deletes a user from your organization. To delete a service account, see [Deprovision service account](#deprovision-service-account).

#### Endpoint

* **URL**: `[HOST-URL]/scim/Users/{id}`
* **Method**: `DELETE`

#### Parameters

| Parameter | Type   | Required | Description                         |
| --------- | ------ | -------- | ----------------------------------- |
| `id`      | string | Yes      | The unique ID of the user to delete |

#### Example

<Tabs>
  <Tab title="Delete User Request">
    ```bash theme={null}
    DELETE /scim/Users/abc
    ```
  </Tab>

  <Tab title="Delete User Response">
    ```text theme={null}
    (Status 204)
    ```
  </Tab>
</Tabs>

<Note>
  To temporarily deactivate the user, refer to the [Deactivate user](#deactivate-user) API, which uses the `PATCH` endpoint.
</Note>

### Update user email

Updates a user's primary email address.

**Not supported for Multi-tenant Cloud**, where a user's account isn't managed by the organization.

#### Endpoint

* **URL**: `[HOST-URL]/scim/Users/{id}`
* **Method**: `PATCH`

#### Parameters

| Parameter | Type   | Required | Description                 |
| --------- | ------ | -------- | --------------------------- |
| `id`      | string | Yes      | The unique ID of the user   |
| `op`      | string | Yes      | `replace`                   |
| `path`    | string | Yes      | `emails`                    |
| `value`   | array  | Yes      | Array with new email object |

#### Example

<Tabs>
  <Tab title="Update Email Request">
    ```bash theme={null}
    PATCH /scim/Users/abc
    ```

    ```json theme={null}
    {
        "schemas": ["urn:ietf:params:scim:api:messages:2.0:PatchOp"],
        "Operations": [
            {
                "op": "replace",
                "path": "emails",
                "value": [
                    {
                        "value": "newemail@example.com",
                        "primary": true
                    }
                ]
            }
        ]
    }
    ```
  </Tab>

  <Tab title="Update Email Response">
    ```text theme={null}
    (Status 200)
    ```

    ```json theme={null}
    {
        "active": true,
        "displayName": "Dev User 1",
        "emails": [
            {
                "primary": true,
                "value": "newemail@example.com"
            }
        ],
        "id": "abc",
        "meta": {
            "resourceType": "User",
            "created": "2023-10-01T00:00:00Z",
            "lastModified": "2023-10-01T00:00:00Z",
            "location": "Users/abc"
        },
        "schemas": [
            "urn:ietf:params:scim:schemas:core:2.0:User"
        ],
        "userName": "dev-user1"
    }
    ```
  </Tab>
</Tabs>

### Update user display name

Updates a user's display name.

**Not supported for Multi-tenant Cloud**, where a user's account isn't managed by the organization.

#### Endpoint

* **URL**: `[HOST-URL]/scim/Users/{id}`
* **Method**: `PATCH`

#### Parameters

| Parameter | Type   | Required | Description               |
| --------- | ------ | -------- | ------------------------- |
| `id`      | string | Yes      | The unique ID of the user |
| `op`      | string | Yes      | `replace`                 |
| `path`    | string | Yes      | `displayName`             |
| `value`   | string | Yes      | New display name          |

#### Example

<Tabs>
  <Tab title="Update Display Name Request">
    ```bash theme={null}
    PATCH /scim/Users/abc
    ```

    ```json theme={null}
    {
        "schemas": ["urn:ietf:params:scim:api:messages:2.0:PatchOp"],
        "Operations": [
            {
                "op": "replace",
                "path": "displayName",
                "value": "John Doe"
            }
        ]
    }
    ```
  </Tab>

  <Tab title="Update Display Name Response">
    ```text theme={null}
    (Status 200)
    ```

    ```json theme={null}
    {
        "active": true,
        "displayName": "John Doe",
        "emails": [
            {
                "primary": true,
                "value": "dev-user1@example.com"
            }
        ],
        "id": "abc",
        "meta": {
            "resourceType": "User",
            "created": "2025-7-01T00:00:00Z",
            "lastModified": "2025-7-01T00:00:00Z",
            "location": "users/dev-user1"
        },
        "schemas": [
            "urn:ietf:params:scim:schemas:core:2.0:User"
        ],
        "userName": "dev-user1"
    }
    ```
  </Tab>
</Tabs>

### Deactivate user

Deactivates a user in your organization. The result differs by deployment type:

* **Dedicated Cloud** / **Self-Managed**: Sets the user's `active` field to `false`. To restore a deactivated user's access to your organization, see [Reactivate user](#reactivate-user).
* **Multi-tenant Cloud**: Removes the user from the organization. To restore the user's access, re-add them to your organization. See [Create user](#create-user-request-multi-tenant). In Multi-tenant Cloud, a user's account isn't managed by the organization.

<Note>This operation works for users only, not service accounts. Deactivating a service account isn't supported. Manage team service accounts in the settings for the W\&B Team.</Note>

#### Endpoint

* **URL**: `[HOST-URL]/scim/Users/{id}`
* **Method**: `PATCH`

#### Parameters

| Parameter | Type   | Required | Description                             |
| --------- | ------ | -------- | --------------------------------------- |
| `id`      | string | Yes      | The unique ID of the user to deactivate |
| `op`      | string | Yes      | `replace`                               |
| `value`   | object | Yes      | Object with `{"active": false}`         |

#### Example

<Tabs>
  <Tab title="Deactivate User Request (Dedicated/Self-Managed)">
    ```bash theme={null}
    PATCH /scim/Users/abc
    ```

    ```json theme={null}
    {
        "schemas": ["urn:ietf:params:scim:api:messages:2.0:PatchOp"],
        "Operations": [
            {
                "op": "replace",
                "value": {"active": false}
            }
        ]
    }
    ```
  </Tab>

  <Tab title="Deactivate User Request (Multi-tenant)">
    ```bash theme={null}
    PATCH /scim/Users
    ```

    ```json theme={null}
    {
        "schemas": ["urn:ietf:params:scim:api:messages:2.0:PatchOp"],
        "Operations": [
            {
                "op": "replace",
                "value": {"active": false}
            }
        ]
    }
    ```
  </Tab>
</Tabs>

#### Response

<Tabs>
  <Tab title="Deactivate User Response (Dedicated/Self-Managed)">
    ```text theme={null}
    (Status 200)
    ```

    ```json theme={null}
    {
        "active": false,
        "displayName": "Dev User 1",
        "emails": [
            {
                "primary": true,
                "value": "dev-user1@example.com"
            }
        ],
        "id": "abc",
        "meta": {
            "resourceType": "User",
            "created": "2023-10-01T00:00:00Z",
            "lastModified": "2023-10-01T00:00:00Z",
            "location": "Users/abc"
        },
        "schemas": [
            "urn:ietf:params:scim:schemas:core:2.0:User"
        ],
        "userName": "dev-user1"
    }
    ```
  </Tab>

  <Tab title="Deactivate User Response (Multi-tenant)">
    ```text theme={null}
    (Status 200)
    ```

    ```json theme={null}
    {
        "schemas": ["urn:ietf:params:scim:api:messages:2.0:PatchOp"],
        "Operations": [
            {
                "op": "replace",
                "value": {"active": true}
            }
        ]
    }
    ```
  </Tab>
</Tabs>

### Reactivate user

Reactivates a previously deactivated user in your organization.

<Note>
  * User reactivation works for users only, not service accounts. Reactivation isn't supported for service accounts. Manage service accounts in the settings for the W\&B Team.

  * User reactivation isn't supported in [Multi-tenant Cloud](/platform/hosting/hosting-options/multi_tenant_cloud). To restore the user's access, re-add them to your organization. See [Create user](#create-user-request-multi-tenant). In Multi-tenant Cloud, a user's account isn't managed by the organization. An attempt to reactivate a user results in an HTTP `400` error. The `detail` field in the response body is returned verbatim from the API and may still use legacy product wording:
    ```json theme={null}
    {
        "schemas": [
            "urn:ietf:params:scim:api:messages:2.0:Error"
        ],
        "detail": "User reactivation operations are not supported in SaaS Cloud",
        "status": "400"
    }
    ```
</Note>

#### Endpoint

* **URL**: `[HOST-URL]/scim/Users/{id}`
* **Method**: `PATCH`

#### Parameters

| Parameter | Type   | Required | Description                             |
| --------- | ------ | -------- | --------------------------------------- |
| `id`      | string | Yes      | The unique ID of the user to reactivate |
| `op`      | string | Yes      | `replace`                               |
| `value`   | object | Yes      | Object with `{"active": true}`          |

#### Example

<Tabs>
  <Tab title="Reactivate User Request">
    ```bash theme={null}
    PATCH /scim/Users/abc
    ```

    ```json theme={null}
    {
        "schemas": ["urn:ietf:params:scim:api:messages:2.0:PatchOp"],
        "Operations": [
            {
                "op": "replace",
                "value": {"active": true}
            }
        ]
    }
    ```
  </Tab>

  <Tab title="Reactivate User Response">
    ```text theme={null}
    (Status 200)
    ```

    ```json theme={null}
    {
        "active": true,
        "displayName": "Dev User 1",
        "emails": [
            {
                "primary": true,
                "value": "dev-user1@example.com"
            }
        ],
        "id": "abc",
        "meta": {
            "resourceType": "User",
            "created": "2023-10-01T00:00:00Z",
            "lastModified": "2023-10-01T00:00:00Z",
            "location": "Users/abc"
        },
        "schemas": [
            "urn:ietf:params:scim:schemas:core:2.0:User"
        ],
        "userName": "dev-user1"
    }
    ```
  </Tab>
</Tabs>

### Assign organization role

Assigns an organization-level role to a user.

<Note>This operation works for users only, not service accounts. Custom roles aren't supported for service accounts.</Note>

#### Endpoint

* **URL**: `[HOST-URL]/scim/Users/{id}`
* **Method**: `PATCH`

#### Parameters

| Parameter | Type   | Required | Description                     |
| --------- | ------ | -------- | ------------------------------- |
| `id`      | string | Yes      | The unique ID of the user       |
| `op`      | string | Yes      | `replace`                       |
| `path`    | string | Yes      | `organizationRole`              |
| `value`   | string | Yes      | Role name (`admin` or `member`) |

<Note>
  The organization-scoped `viewer` role is deprecated and can no longer be assigned in the UI. If you use SCIM to assign the `viewer` role to a user:

  * They're assigned the `member` role in the organization.
  * Their `modelsSeat` is set to `viewer` instead of `full`. This allows view-only access to Models and full access to Registry. If no Models seats are available, a `Seat limit reached` error is returned. This can be updated later if a seat is available.
  * Their `weaveRole` is set to `viewer` instead of `full`. This allows view-only access to Weave.
  * All of their existing team and project roles are set to `viewer`.
  * They're assigned the Registry `viewer` role in registries that are visible at the organization level.

  Assigning the `member` or `admin` organization role doesn't change the user's `modelsSeat` or `weaveRole`.
</Note>

#### Example

<Tabs>
  <Tab title="Assign Org Role Request">
    ```bash theme={null}
    PATCH /scim/Users/abc
    ```

    ```json theme={null}
    {
        "schemas": ["urn:ietf:params:scim:api:messages:2.0:PatchOp"],
        "Operations": [
            {
                "op": "replace",
                "path": "organizationRole",
                "value": "admin"
            }
        ]
    }
    ```
  </Tab>

  <Tab title="Assign Org Role Response">
    ```text theme={null}
    (Status 200)
    ```

    ```json theme={null}
    {
        "active": true,
        "displayName": "Dev User 1",
        "emails": [
            {
                "primary": true,
                "value": "dev-user1@example.com"
            }
        ],
        "id": "abc",
        "meta": {
            "resourceType": "User",
            "created": "2023-10-01T00:00:00Z",
            "lastModified": "2023-10-01T00:00:00Z",
            "location": "Users/abc"
        },
        "schemas": [
            "urn:ietf:params:scim:schemas:core:2.0:User"
        ],
        "userName": "dev-user1",
        "teamRoles": [
            {
                "teamName": "team1",
                "roleName": "admin"
            }
        ],
        "organizationRole": "admin"
    }
    ```
  </Tab>
</Tabs>

### Update Models seat

Updates a user's Models seat.

#### Endpoint

* **URL**: `[HOST-URL]/scim/Users/{id}`
* **Method**: `PATCH`

#### Parameters

| Parameter | Type   | Required | Description                              |
| --------- | ------ | -------- | ---------------------------------------- |
| `id`      | string | Yes      | The unique ID of the user                |
| `op`      | string | Yes      | `replace`                                |
| `path`    | string | Yes      | `modelsSeat`                             |
| `value`   | string | Yes      | Seat level (`full`, `viewer`, or `none`) |

#### Example

<Tabs>
  <Tab title="Update Models Seat Request">
    ```bash theme={null}
    PATCH /scim/Users/abc
    ```

    ```json theme={null}
    {
        "schemas": ["urn:ietf:params:scim:api:messages:2.0:PatchOp"],
        "Operations": [
            {
                "op": "replace",
                "path": "modelsSeat",
                "value": "full"
            }
        ]
    }
    ```
  </Tab>

  <Tab title="Update Models Seat Response">
    ```text theme={null}
    (Status 200)
    ```

    ```json theme={null}
    {
        "active": true,
        "displayName": "Dev User 1",
        "emails": [
            {
                "primary": true,
                "value": "dev-user1@example.com"
            }
        ],
        "id": "abc",
        "meta": {
            "resourceType": "User",
            "created": "2023-10-01T00:00:00Z",
            "lastModified": "2023-10-01T00:00:00Z",
            "location": "Users/abc"
        },
        "schemas": [
            "urn:ietf:params:scim:schemas:core:2.0:User"
        ],
        "userName": "dev-user1",
        "organizationRole": "member",
        "modelsSeat": "full",
        "weaveRole": "full"
    }
    ```
  </Tab>
</Tabs>

### Update Weave role

Updates a user's Weave role.

#### Endpoint

* **URL**: `[HOST-URL]/scim/Users/{id}`
* **Method**: `PATCH`

#### Parameters

| Parameter | Type   | Required | Description                              |
| --------- | ------ | -------- | ---------------------------------------- |
| `id`      | string | Yes      | The unique ID of the user                |
| `op`      | string | Yes      | `replace`                                |
| `path`    | string | Yes      | `weaveRole`                              |
| `value`   | string | Yes      | Role level (`full`, `viewer`, or `none`) |

#### Example

<Tabs>
  <Tab title="Update Weave Role Request">
    ```bash theme={null}
    PATCH /scim/Users/abc
    ```

    ```json theme={null}
    {
        "schemas": ["urn:ietf:params:scim:api:messages:2.0:PatchOp"],
        "Operations": [
            {
                "op": "replace",
                "path": "weaveRole",
                "value": "full"
            }
        ]
    }
    ```
  </Tab>

  <Tab title="Update Weave Role Response">
    ```text theme={null}
    (Status 200)
    ```

    ```json theme={null}
    {
        "active": true,
        "displayName": "Dev User 1",
        "emails": [
            {
                "primary": true,
                "value": "dev-user1@example.com"
            }
        ],
        "id": "abc",
        "meta": {
            "resourceType": "User",
            "created": "2023-10-01T00:00:00Z",
            "lastModified": "2023-10-01T00:00:00Z",
            "location": "Users/abc"
        },
        "schemas": [
            "urn:ietf:params:scim:schemas:core:2.0:User"
        ],
        "userName": "dev-user1",
        "organizationRole": "member",
        "modelsSeat": "full",
        "weaveRole": "full"
    }
    ```
  </Tab>
</Tabs>

### Assign team role

Assigns a team-level role to a user.

<Note>This operation works for users only, not service accounts. Custom roles aren't supported for service accounts.</Note>

#### Endpoint

* **URL**: `[HOST-URL]/scim/Users/{id}`
* **Method**: `PATCH`

#### Parameters

| Parameter | Type   | Required | Description                                     |
| --------- | ------ | -------- | ----------------------------------------------- |
| `id`      | string | Yes      | The unique ID of the user                       |
| `op`      | string | Yes      | `replace`                                       |
| `path`    | string | Yes      | `teamRoles`                                     |
| `value`   | array  | Yes      | Array of objects with `teamName` and `roleName` |

#### Example

<Tabs>
  <Tab title="Assign Team Role Request">
    ```bash theme={null}
    PATCH /scim/Users/abc
    ```

    ```json theme={null}
    {
        "schemas": ["urn:ietf:params:scim:api:messages:2.0:PatchOp"],
        "Operations": [
            {
                "op": "replace",
                "path": "teamRoles",
                "value": [
                    {
                        "roleName": "admin",
                        "teamName": "team1"
                    }
                ]
            }
        ]
    }
    ```
  </Tab>

  <Tab title="Assign Team Role Response">
    ```text theme={null}
    (Status 200)
    ```

    ```json theme={null}
    {
        "active": true,
        "displayName": "Dev User 1",
        "emails": [
            {
                "primary": true,
                "value": "dev-user1@example.com"
            }
        ],
        "id": "abc",
        "meta": {
            "resourceType": "User",
            "created": "2023-10-01T00:00:00Z",
            "lastModified": "2023-10-01T00:00:00Z",
            "location": "Users/abc"
        },
        "schemas": [
            "urn:ietf:params:scim:schemas:core:2.0:User"
        ],
        "userName": "dev-user1",
        "teamRoles": [
            {
                "teamName": "team1",
                "roleName": "admin"
            }
        ],
        "organizationRole": "admin"
    }
    ```
  </Tab>
</Tabs>

### Add to Registry

Adds a user to a registry with an assigned registry-level role.

<Note>This operation works for users only, not service accounts. Custom roles aren't supported for service accounts.</Note>

#### Endpoint

* **URL**: `[HOST-URL]/scim/Users/{id}`
* **Method**: `PATCH`

#### Parameters

| Parameter | Type   | Required | Description                                         |
| --------- | ------ | -------- | --------------------------------------------------- |
| `id`      | string | Yes      | The unique ID of the user                           |
| `op`      | string | Yes      | `add`                                               |
| `path`    | string | Yes      | `registryRoles`                                     |
| `value`   | array  | Yes      | Array of objects with `registryName` and `roleName` |

#### Example

<Tabs>
  <Tab title="Add to Registry Request">
    ```bash theme={null}
    PATCH /scim/Users/abc
    ```

    ```json theme={null}
    {
        "schemas": ["urn:ietf:params:scim:api:messages:2.0:PatchOp"],
        "Operations": [
            {
                "op": "replace",
                "path": "registryRoles",
                "value": [
                    {
                        "roleName": "admin",
                        "registryName": "hello-registry"
                    }
                ]
            }
        ]
    }
    ```
  </Tab>

  <Tab title="Add to Registry Response">
    ```text theme={null}
    (Status 200)
    ```

    ```json theme={null}
    {
        "active": true,
        "displayName": "Dev User 1",
        "emails": [
            {
                "primary": true,
                "value": "dev-user1@example.com"
            }
        ],
        "id": "abc",
        "meta": {
            "resourceType": "User",
            "created": "2023-10-01T00:00:00Z",
            "lastModified": "2023-10-01T00:00:00Z",
            "location": "Users/abc"
        },
        "schemas": [
            "urn:ietf:params:scim:schemas:core:2.0:User"
        ],
        "userName": "dev-user1",
        "registryRoles": [
            {
                "registryName": "hello-registry",
                "roleName": "admin"
            }
        ],
        "organizationRole": "admin"
    }
    ```
  </Tab>
</Tabs>

### Remove from Registry

Removes a user from a registry.

<Note>
  * The remove operations follow RFC 7644 SCIM protocol specifications. Use the filter syntax `"registryRoles[registryName eq \"{registry_name}\"]"` to remove a user from a specific registry, or `"registryRoles"` to remove the user from all registries.
  * This operation works for users only, not service accounts. Remove service accounts from a registry in the settings for the W\&B Team.
</Note>

#### Endpoint

* **URL**: `[HOST-URL]/scim/Users/{id}`
* **Method**: `PATCH`

#### Parameters

| Parameter | Type   | Required | Description                                                                 |
| --------- | ------ | -------- | --------------------------------------------------------------------------- |
| `id`      | string | Yes      | The unique ID of the user                                                   |
| `op`      | string | Yes      | `remove`                                                                    |
| `path`    | string | Yes      | `"registryRoles[registryName eq \"{registry_name}\"]"` or `"registryRoles"` |

#### Example

<Tabs>
  <Tab title="Remove from Registry Request">
    ```bash theme={null}
    PATCH /scim/Users/abc
    ```

    ```json theme={null}
    {
        "schemas": ["urn:ietf:params:scim:api:messages:2.0:PatchOp"],
        "Operations": [
            {
                "op": "replace",
                "path": "registryRoles[registryName eq \"goodbye-registry\"]"
            }
        ]
    }
    ```
  </Tab>

  <Tab title="Remove from Registry Response">
    ```text theme={null}
    (Status 200)
    ```

    ```json theme={null}
    {
        "active": true,
        "displayName": "Dev User 1",
        "emails": [
            {
                "primary": true,
                "value": "dev-user1@example.com"
            }
        ],
        "id": "abc",
        "meta": {
            "resourceType": "User",
            "created": "2023-10-01T00:00:00Z",
            "lastModified": "2023-10-01T00:00:00Z",
            "location": "Users/abc"
        },
        "schemas": [
            "urn:ietf:params:scim:schemas:core:2.0:User"
        ],
        "userName": "dev-user1",
        "registryRoles": [
            {
                "registryName": "hello-registry",
                "roleName": "admin"
            }
        ],
        "organizationRole": "admin"
    }
    ```
  </Tab>
</Tabs>

<Tabs>
  <Tab title="Remove from ALL Registries Request">
    ```bash theme={null}
    PATCH /scim/Users/abc
    ```

    ```json theme={null}
    {
        "schemas": ["urn:ietf:params:scim:api:messages:2.0:PatchOp"],
        "Operations": [
            {
                "op": "replace",
                "path": "registryRoles"
            }
        ]
    }
    ```
  </Tab>

  <Tab title="Remove from ALL Registries Response">
    ```text theme={null}
    (Status 200)
    ```

    ```json theme={null}
    {
        "active": true,
        "displayName": "Dev User 1",
        "emails": [
            {
                "primary": true,
                "value": "dev-user1@example.com"
            }
        ],
        "id": "abc",
        "meta": {
            "resourceType": "User",
            "created": "2023-10-01T00:00:00Z",
            "lastModified": "2023-10-01T00:00:00Z",
            "location": "Users/abc"
        },
        "schemas": [
            "urn:ietf:params:scim:schemas:core:2.0:User"
        ],
        "userName": "dev-user1",
        "organizationRole": "admin"
    }
    ```
  </Tab>
</Tabs>

## Group resource

The SCIM group resource maps to a W\&B Team. Use the endpoints in this section to create teams, manage team membership, and (optionally) configure team-level storage from your identity provider or automation. When you create a SCIM group in your IAM, it creates and maps to a W\&B Team, and other SCIM group operations act on the team. To configure custom storage during team creation, include `storageBucket` in the request.

### Service accounts

When you create a W\&B Team using SCIM, all organization-level service accounts are automatically added to the team, to maintain the service account's access to team resources.

### Filter groups

The `/Groups` endpoint supports filtering to search for specific teams.

#### Supported filters

The `/Groups` endpoint supports the following filter:

* `displayName eq "value"`: Filter by team display name.

#### Example

```bash theme={null}
GET /scim/Groups?filter=displayName eq "engineering-team"
```

### Get team

Retrieve team information by providing the team's unique ID.

#### Endpoint

* **URL**: `[HOST-URL]/scim/Groups/{id}`
* **Method**: `GET`

#### Example

<Tabs>
  <Tab title="Request">
    ```bash theme={null}
    GET /scim/Groups/ghi
    ```
  </Tab>

  <Tab title="Response">
    ```text theme={null}
    (Status 200)
    ```

    ```json theme={null}
    {
        "displayName": "acme-devs",
        "id": "ghi",
        "members": [
            {
                "Value": "abc",
                "Ref": "",
                "Type": "",
                "Display": "dev-user1"
            }
        ],
        "meta": {
            "resourceType": "Group",
            "created": "2023-10-01T00:00:00Z",
            "lastModified": "2023-10-01T00:00:00Z",
            "location": "Groups/ghi"
        },
        "schemas": [
            "urn:ietf:params:scim:schemas:core:2.0:Group"
        ]
    }
    ```
  </Tab>
</Tabs>

### List teams

Retrieve a list of teams.

#### Endpoint

* **URL**: `[HOST-URL]/scim/Groups`
* **Method**: `GET`

#### Example

<Tabs>
  <Tab title="Request">
    ```bash theme={null}
    GET /scim/Groups
    ```
  </Tab>

  <Tab title="Response">
    ```text theme={null}
    (Status 200)
    ```

    ```json theme={null}
    {
        "Resources": [
            {
                "displayName": "acme-devs",
                "id": "ghi",
                "members": [
                    {
                        "Value": "abc",
                        "Ref": "",
                        "Type": "",
                        "Display": "dev-user1"
                    }
                ],
                "meta": {
                    "resourceType": "Group",
                    "created": "2023-10-01T00:00:00Z",
                    "lastModified": "2023-10-01T00:00:00Z",
                    "location": "Groups/ghi"
                },
                "schemas": [
                    "urn:ietf:params:scim:schemas:core:2.0:Group"
                ]
            }
        ],
        "itemsPerPage": 9999,
        "schemas": [
            "urn:ietf:params:scim:api:messages:2.0:ListResponse"
        ],
        "startIndex": 1,
        "totalResults": 1
    }
    ```
  </Tab>
</Tabs>

### Create team

Creates a new team resource.

#### Endpoint

* **URL**: `[HOST-URL]/scim/Groups`
* **Method**: `POST`

#### Supported fields

| Field           | Type               | Required                                                  |
| --------------- | ------------------ | --------------------------------------------------------- |
| `displayName`   | String             | Yes                                                       |
| `members`       | Multi-Valued Array | Yes (`value` sub-field is required and maps to a user ID) |
| `storageBucket` | Object             | No                                                        |

You can configure team-level [Bring your own bucket (BYOB)](/platform/hosting/data-security/secure-storage-connector) during team creation by including a `storageBucket` object. If omitted, the team uses default or instance-level storage. Provision the bucket (policy, CORS, credentials) and determine the storage address format per provider using the BYOB guide. The `storageBucket` object has the following sub-fields:

* **Required**: `name` (bucket name), `provider` (one of `COREWEAVE`, `AWS`, `AZURE`, `GCP`, or `MINIO`). The value is case-sensitive. Use uppercase as shown.
* **Optional**: `path` (path prefix within the bucket), `kmsKeyId` (KMS key for encryption, for example for AWS), `awsExternalId` (AWS cross-account access), `azureTenantId` (Azure tenant ID), `azureClientId` (Azure managed identity client ID).

W\&B validates that the bucket exists and is reachable before creating the team. If validation fails, the SCIM request fails and the team isn't created.

An invalid `provider` value returns `400 Bad Request` with a SCIM error that lists the allowed values.

#### Examples

These examples show how to create a team without custom storage and with BYOB storage on a specific provider. Select a tab for the desired storage configuration to see an example request, and select the **Response** tab for an example response.

<Tabs>
  <Tab title="Request (no BYOB)">
    ```bash theme={null}
    POST /scim/Groups
    ```

    ```json theme={null}
    {
        "schemas": ["urn:ietf:params:scim:schemas:core:2.0:Group"],
        "displayName": "wandb-support",
        "members": [
            {
                "value": "def"
            }
        ]
    }
    ```
  </Tab>

  <Tab title="CoreWeave">
    ```bash theme={null}
    POST /scim/Groups
    Content-Type: application/scim+json
    ```

    ```json theme={null}
    {
      "schemas": ["urn:ietf:params:scim:schemas:core:2.0:Group"],
      "displayName": "ml-training-team",
      "members": [
        {
          "value": "user@example.com",
          "display": "user@example.com"
        }
      ],
      "storageBucket": {
        "name": "wandb-coreweave-bucket",
        "provider": "COREWEAVE",
        "path": "ml-training/experiments"
      }
    }
    ```
  </Tab>

  <Tab title="AWS S3">
    ```bash theme={null}
    POST /scim/Groups
    Content-Type: application/scim+json
    ```

    ```json theme={null}
    {
      "schemas": ["urn:ietf:params:scim:schemas:core:2.0:Group"],
      "displayName": "ml-team",
      "members": [
        {
          "value": "user@example.com",
          "display": "user@example.com"
        }
      ],
      "storageBucket": {
        "name": "my-company-wandb-data",
        "provider": "AWS",
        "path": "ml-team/experiments",
        "kmsKeyId": "arn:aws:kms:us-east-1:123456789012:key/12345678-1234-1234-1234-123456789012",
        "awsExternalId": "wandb-external-id-abc123"
      }
    }
    ```
  </Tab>

  <Tab title="Azure">
    ```bash theme={null}
    POST /scim/Groups
    Content-Type: application/scim+json
    ```

    ```json theme={null}
    {
      "schemas": ["urn:ietf:params:scim:schemas:core:2.0:Group"],
      "displayName": "research-team",
      "members": [],
      "storageBucket": {
        "name": "wandbstorage",
        "provider": "AZURE",
        "path": "research/artifacts",
        "azureTenantId": "12345678-1234-1234-1234-123456789012",
        "azureClientId": "87654321-4321-4321-4321-210987654321"
      }
    }
    ```
  </Tab>

  <Tab title="GCP">
    ```bash theme={null}
    POST /scim/Groups
    Content-Type: application/scim+json
    ```

    ```json theme={null}
    {
      "schemas": ["urn:ietf:params:scim:schemas:core:2.0:Group"],
      "displayName": "data-science-team",
      "members": [
        {
          "value": "VXNlcjox",
          "display": "jane.doe@example.com"
        },
        {
          "value": "VXNlcjoy",
          "display": "john.smith@example.com"
        }
      ],
      "storageBucket": {
        "name": "my-gcs-bucket",
        "provider": "GCP",
        "path": "data-science/runs"
      }
    }
    ```
  </Tab>

  <Tab title="Response">
    ```text theme={null}
    (Status 201)
    ```

    ```json theme={null}
    {
        "displayName": "wandb-support",
        "id": "jkl",
        "members": [
            {
                "Value": "def",
                "Ref": "",
                "Type": "",
                "Display": "dev-user2"
            }
        ],
        "meta": {
            "resourceType": "Group",
            "created": "2023-10-01T00:00:00Z",
            "lastModified": "2023-10-01T00:00:00Z",
            "location": "Groups/jkl"
        },
        "schemas": [
            "urn:ietf:params:scim:schemas:core:2.0:Group"
        ]
    }
    ```
  </Tab>
</Tabs>

### Update team

Updates an existing team's membership list.

#### Endpoint

* **URL**: `[HOST-URL]/scim/Groups/{id}`
* **Method**: `PATCH`
* **Supported operations**: `add` member, `remove` member, `replace` members.

<Note>
  - The remove operations follow RFC 7644 SCIM protocol specifications. Use the filter syntax `members[value eq "{user_id}"]` to remove a specific user, or `members` to remove all users from the team.

    **User identification**: The `{user_id}` in member operations can be either of the following:

    * A W\&B user ID.
    * An email address (for example, "[user@example.com](mailto:user@example.com)").
  - These operations work for users only, not service accounts. Update a team's service accounts in the settings for the W\&B Team.
</Note>

<Info>
  Replace `{team_id}` with the actual team ID and `{user_id}` with the actual user ID or email address in your requests.
</Info>

### Replace team members

Replaces all members of a team with a new list.

<Note>This operation works for users only, not service accounts. Manage service accounts in the settings for the W\&B Team.</Note>

#### Endpoint

* **URL**: `[HOST-URL]/scim/Groups/{id}`
* **Method**: `PUT`

<Tabs>
  <Tab title="Request">
    ```bash theme={null}
    PUT /scim/Groups/{team_id}
    ```

    ```json theme={null}
    {
        "schemas": ["urn:ietf:params:scim:schemas:core:2.0:Group"],
        "displayName": "acme-devs",
        "members": [
            {
                "value": "{user_id_1}"
            },
            {
                "value": "{user_id_2}"
            }
        ]
    }
    ```
  </Tab>

  <Tab title="Response">
    ```text theme={null}
    (Status 200)
    ```

    ```json theme={null}
    {
        "displayName": "acme-devs",
        "id": "ghi",
        "members": [
            {
                "Value": "user_id_1",
                "Ref": "",
                "Type": "",
                "Display": "user1"
            },
            {
                "Value": "user_id_2",
                "Ref": "",
                "Type": "",
                "Display": "user2"
            }
        ],
        "meta": {
            "resourceType": "Group",
            "created": "2023-10-01T00:00:00Z",
            "lastModified": "2023-10-01T00:01:00Z",
            "location": "Groups/ghi"
        },
        "schemas": [
            "urn:ietf:params:scim:schemas:core:2.0:Group"
        ]
    }
    ```
  </Tab>
</Tabs>

### Add a user to a team

Adds `dev-user2` to `acme-devs`:

<Note>This operation works for users only, not service accounts. Manage service accounts in the settings for the W\&B Team.</Note>

<Tabs>
  <Tab title="Request">
    ```bash theme={null}
    PATCH /scim/Groups/{team_id}
    ```

    ```json theme={null}
    {
        "schemas": ["urn:ietf:params:scim:api:messages:2.0:PatchOp"],
        "Operations": [
            {
                "op": "add",
                "path": "members",
                "value": [
                    {
                        "value": "{user_id}"
                    }
                ]
            }
        ]
    }
    ```
  </Tab>

  <Tab title="Response">
    ```text theme={null}
    (Status 200)
    ```

    ```json theme={null}
    {
        "displayName": "acme-devs",
        "id": "ghi",
        "members": [
            {
                "Value": "abc",
                "Ref": "",
                "Type": "",
                "Display": "dev-user1"
            },
            {
                "Value": "def",
                "Ref": "",
                "Type": "",
                "Display": "dev-user2"
            }
        ],
        "meta": {
            "resourceType": "Group",
            "created": "2023-10-01T00:00:00Z",
            "lastModified": "2023-10-01T00:01:00Z",
            "location": "Groups/ghi"
        },
        "schemas": [
            "urn:ietf:params:scim:schemas:core:2.0:Group"
        ]
    }
    ```
  </Tab>
</Tabs>

### Remove a specific user from a team

Removes `dev-user2` from `acme-devs`:

<Note>This operation works for users only, not service accounts. Manage service accounts in the settings for the W\&B Team.</Note>

<Tabs>
  <Tab title="Request">
    ```bash theme={null}
    PATCH /scim/Groups/{team_id}
    ```

    ```json theme={null}
    {
        "schemas": ["urn:ietf:params:scim:api:messages:2.0:PatchOp"],
        "Operations": [
            {
                "op": "remove",
                "path": "members[value eq \"{user_id}\"]"
            }
        ]
    }
    ```
  </Tab>

  <Tab title="Response">
    ```text theme={null}
    (Status 200)
    ```

    ```json theme={null}
    {
        "displayName": "acme-devs",
        "id": "ghi",
        "members": [
            {
                "Value": "abc",
                "Display": "dev-user1"
            }
        ],
        "meta": {
            "resourceType": "Group",
            "created": "2023-10-01T00:00:00Z",
            "lastModified": "2023-10-01T00:01:00Z",
            "location": "Groups/ghi"
        },
        "schemas": [
            "urn:ietf:params:scim:schemas:core:2.0:Group"
        ]
    }
    ```
  </Tab>
</Tabs>

### Remove all users from a team

Removes all users from `acme-devs`:

<Note>This operation works for users only, not service accounts. Manage service accounts in the settings for the W\&B Team.</Note>

<Tabs>
  <Tab title="Request">
    ```bash theme={null}
    PATCH /scim/Groups/{team_id}
    ```

    ```json theme={null}
    {
        "schemas": ["urn:ietf:params:scim:api:messages:2.0:PatchOp"],
        "Operations": [
            {
                "op": "remove",
                "path": "members"
            }
        ]
    }
    ```
  </Tab>

  <Tab title="Response">
    ```text theme={null}
    (Status 200)
    ```

    ```json theme={null}
    {
        "displayName": "acme-devs",
        "id": "ghi",
        "members": null,
        "meta": {
            "resourceType": "Group",
            "created": "2023-10-01T00:00:00Z",
            "lastModified": "2023-10-01T00:01:00Z",
            "location": "Groups/ghi"
        },
        "schemas": [
            "urn:ietf:params:scim:schemas:core:2.0:Group"
        ]
    }
    ```
  </Tab>
</Tabs>

### Delete team

The SCIM API doesn't support deleting teams because additional data is linked to teams. Delete teams from the W\&B App to confirm you want everything deleted.

## Role resource

The SCIM role resource maps to W\&B custom roles. Use the endpoints in this section to create and maintain custom roles programmatically (for example, to keep role definitions in sync with your access policies). The `/Roles` endpoints aren't part of the official SCIM schema. W\&B adds `/Roles` endpoints to support automated management of custom roles in W\&B organizations.

### Get custom role

Retrieve information for a custom role by providing the role's unique ID.

#### Endpoint

* **URL**: `[HOST-URL]/scim/Roles/{id}`
* **Method**: `GET`

#### Example

<Tabs>
  <Tab title="Request">
    ```bash theme={null}
    GET /scim/Roles/abc
    ```
  </Tab>

  <Tab title="Response">
    ```text theme={null}
    (Status 200)
    ```

    ```json theme={null}
    {
        "description": "A sample custom role for example",
        "id": "Um9sZTo3",
        "inheritedFrom": "member", // indicates the predefined role
        "meta": {
            "resourceType": "Role",
            "created": "2023-11-20T23:10:14Z",
            "lastModified": "2023-11-20T23:31:23Z",
            "location": "Roles/Um9sZTo3"
        },
        "name": "Sample custom role",
        "organizationID": "T3JnYW5pemF0aW9uOjE0ODQ1OA==",
        "permissions": [
            {
                "name": "artifact:read",
                "isInherited": true // inherited from member predefined role
            },
            ...
            ...
            {
                "name": "project:update",
                "isInherited": false // custom permission added by admin
            }
        ],
        "schemas": [
            ""
        ]
    }
    ```
  </Tab>
</Tabs>

### List custom roles

Retrieve information for all custom roles in the W\&B organization.

#### Endpoint

* **URL**: `[HOST-URL]/scim/Roles`
* **Method**: `GET`

#### Example

<Tabs>
  <Tab title="Request">
    ```bash theme={null}
    GET /scim/Roles
    ```
  </Tab>

  <Tab title="Response">
    ```text theme={null}
    (Status 200)
    ```

    ```json theme={null}
    {
       "Resources": [
            {
                "description": "A sample custom role for example",
                "id": "Um9sZTo3",
                "inheritedFrom": "member", // indicates the predefined role that the custom role inherits from
                "meta": {
                    "resourceType": "Role",
                    "created": "2023-11-20T23:10:14Z",
                    "lastModified": "2023-11-20T23:31:23Z",
                    "location": "Roles/Um9sZTo3"
                },
                "name": "Sample custom role",
                "organizationID": "T3JnYW5pemF0aW9uOjE0ODQ1OA==",
                "permissions": [
                    {
                        "name": "artifact:read",
                        "isInherited": true // inherited from member predefined role
                    },
                    ...
                    ...
                    {
                        "name": "project:update",
                        "isInherited": false // custom permission added by admin
                    }
                ],
                "schemas": [
                    ""
                ]
            },
            {
                "description": "Another sample custom role for example",
                "id": "Um9sZToxMg==",
                "inheritedFrom": "viewer", // indicates the predefined role that the custom role inherits from
                "meta": {
                    "resourceType": "Role",
                    "created": "2023-11-21T01:07:50Z",
                    "location": "Roles/Um9sZToxMg=="
                },
                "name": "Sample custom role 2",
                "organizationID": "T3JnYW5pemF0aW9uOjE0ODQ1OA==",
                "permissions": [
                    {
                        "name": "launchagent:read",
                        "isInherited": true // inherited from viewer predefined role
                    },
                    ...
                    ...
                    {
                        "name": "run:stop",
                        "isInherited": false // custom permission added by admin
                    }
                ],
                "schemas": [
                    ""
                ]
            }
        ],
        "itemsPerPage": 9999,
        "schemas": [
            "urn:ietf:params:scim:api:messages:2.0:ListResponse"
        ],
        "startIndex": 1,
        "totalResults": 2
    }
    ```
  </Tab>
</Tabs>

### Create custom role

Creates a new custom role in the W\&B organization.

#### Endpoint

* **URL**: `[HOST-URL]/scim/Roles`
* **Method**: `POST`

#### Supported fields

| Field           | Type         | Required                                                                                                                                                                                                                             |
| --------------- | ------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `name`          | String       | Name of the custom role                                                                                                                                                                                                              |
| `description`   | String       | Description of the custom role                                                                                                                                                                                                       |
| `permissions`   | Object array | Array of permission objects where each object includes a `name` string field that has value of the form `w&bobject:operation`. For example, a permission object for delete operation on W\&B runs would have `name` as `run:delete`. |
| `inheritedFrom` | String       | The predefined role that the custom role inherits from. It can be either `member` or `viewer`.                                                                                                                                       |

#### Example

<Tabs>
  <Tab title="Request">
    ```bash theme={null}
    POST /scim/Roles
    ```

    ```json theme={null}
    {
        "schemas": ["urn:ietf:params:scim:schemas:core:2.0:Role"],
        "name": "Sample custom role",
        "description": "A sample custom role for example",
        "permissions": [
            {
                "name": "project:update"
            }
        ],
        "inheritedFrom": "member"
    }
    ```
  </Tab>

  <Tab title="Response">
    ```text theme={null}
    (Status 201)
    ```

    ```json theme={null}
    {
        "description": "A sample custom role for example",
        "id": "Um9sZTo3",
        "inheritedFrom": "member", // indicates the predefined role
        "meta": {
            "resourceType": "Role",
            "created": "2023-11-20T23:10:14Z",
            "lastModified": "2023-11-20T23:31:23Z",
            "location": "Roles/Um9sZTo3"
        },
        "name": "Sample custom role",
        "organizationID": "T3JnYW5pemF0aW9uOjE0ODQ1OA==",
        "permissions": [
            {
                "name": "artifact:read",
                "isInherited": true // inherited from member predefined role
            },
            ...
            ...
            {
                "name": "project:update",
                "isInherited": false // custom permission added by admin
            }
        ],
        "schemas": [
            ""
        ]
    }
    ```
  </Tab>
</Tabs>

### Update custom role

The following sections describe how to add or remove permissions on an existing custom role.

#### Add permissions to role

Adds permissions to an existing custom role.

##### Endpoint

* **URL**: `[HOST-URL]/scim/Roles/{id}`
* **Method**: `PATCH`

<Tabs>
  <Tab title="Request">
    ```bash theme={null}
    PATCH /scim/Roles/{role_id}
    ```

    ```json theme={null}
    {
        "schemas": ["urn:ietf:params:scim:api:messages:2.0:PatchOp"],
        "Operations": [
            {
                "op": "add",
                "path": "permissions",
                "value": [
                    {
                        "name": "project:delete"
                    },
                    {
                        "name": "run:stop"
                    }
                ]
            }
        ]
    }
    ```
  </Tab>

  <Tab title="Response">
    ```text theme={null}
    (Status 200)
    ```

    Returns the updated role with new permissions added.
  </Tab>
</Tabs>

#### Remove a permission from a role

Removes permissions from an existing custom role.

##### Endpoint

* **URL**: `[HOST-URL]/scim/Roles/{id}`
* **Method**: `PATCH`

<Tabs>
  <Tab title="Request">
    ```bash theme={null}
    PATCH /scim/Roles/{role_id}
    ```

    ```json theme={null}
    {
        "schemas": ["urn:ietf:params:scim:api:messages:2.0:PatchOp"],
        "Operations": [
            {
                "op": "remove",
                "path": "permissions",
                "value": [
                    {
                        "name": "project:update"
                    }
                ]
            }
        ]
    }
    ```
  </Tab>

  <Tab title="Response">
    ```text theme={null}
    (Status 200)
    ```

    Returns the updated role with specified permissions removed.
  </Tab>
</Tabs>

### Replace custom role

Replaces an entire custom role definition.

#### Endpoint

* **URL**: `[HOST-URL]/scim/Roles/{id}`
* **Method**: `PUT`

<Tabs>
  <Tab title="Request">
    ```bash theme={null}
    PUT /scim/Roles/{role_id}
    ```

    ```json theme={null}
    {
        "schemas": ["urn:ietf:params:scim:schemas:core:2.0:Role"],
        "name": "Updated custom role",
        "description": "Updated description for the custom role",
        "permissions": [
            {
                "name": "project:read"
            },
            {
                "name": "run:read"
            },
            {
                "name": "artifact:read"
            }
        ],
        "inheritedFrom": "viewer"
    }
    ```
  </Tab>

  <Tab title="Response">
    ```text theme={null}
    (Status 200)
    ```

    Returns the replaced role definition.
  </Tab>
</Tabs>

### Delete custom role

Delete a custom role in the W\&B organization. **Use this operation with caution**. The predefined role that the custom role inherited from is reassigned to all users who held the custom role before the deletion.

#### Endpoint

* **URL**: `[HOST-URL]/scim/Roles/{id}`
* **Method**: `DELETE`

#### Example

<Tabs>
  <Tab title="Request">
    ```bash theme={null}
    DELETE /scim/Roles/abc
    ```
  </Tab>

  <Tab title="Response">
    ```text theme={null}
    (Status 204 No Content)
    ```
  </Tab>
</Tabs>

## Advanced features

The following sections describe optional capabilities (ETag-based concurrency control and standard error responses) that help SCIM integrations behave safely in production.

### ETag support

The SCIM API supports ETags for conditional updates to prevent concurrent modification conflicts. This matters when multiple admins or automated systems update the same resource, because it ensures that one update doesn't silently overwrite another. ETags are returned in the `ETag` response header and the `meta.version` field.

#### ETags

To use ETags, follow these steps:

1. **Get current ETag**: When you GET a resource, note the ETag header in the response.
2. **Conditional update**: Include the ETag in the `If-Match` header when updating.

#### Example

```text theme={null}
# Get user and note ETag
GET /scim/Users/abc
# Response includes: ETag: W/"xyz123"

# Update with ETag
PATCH /scim/Users/abc
If-Match: W/"xyz123"

{
    "schemas": ["urn:ietf:params:scim:api:messages:2.0:PatchOp"],
    "Operations": [
        {
            "op": "replace",
            "path": "organizationRole",
            "value": "admin"
        }
    ]
}
```

A `412 Precondition Failed` error response indicates that the resource has been modified since you retrieved it.

### Error handling

The SCIM API returns standard SCIM error responses:

| Status Code | Description                                     |
| ----------- | ----------------------------------------------- |
| `200`       | Success                                         |
| `201`       | Created                                         |
| `204`       | No Content (successful deletion)                |
| `400`       | Bad Request: invalid parameters or request body |
| `401`       | Unauthorized: authentication failed             |
| `403`       | Forbidden: insufficient permissions             |
| `404`       | Not Found: resource doesn't exist               |
| `409`       | Conflict: resource already exists               |
| `412`       | Precondition Failed: ETag mismatch              |
| `500`       | Internal Server Error                           |

### Implementation differences per deployment type

W\&B maintains two separate SCIM API implementations, and the features differ between them. Review the following table before you integrate with SCIM to confirm that the operations you rely on are available on your deployment type.

| Feature                           | Multi-tenant Cloud | Dedicated Cloud and Self-Managed |
| --------------------------------- | ------------------ | -------------------------------- |
| Update user email                 | -                  | ✓                                |
| Update user display name          | -                  | ✓                                |
| User deactivation                 | ✓                  | ✓                                |
| User reactivation                 | -                  | ✓                                |
| Multiple emails per user          | ✓                  | -                                |
| Set `modelsSeat` on create/update | ✓                  | ✓                                |
| Set `weaveRole` on create/update  | ✓                  | ✓                                |

## Limitations

Keep the following constraints in mind when you design SCIM integrations:

* **Maximum results**: 9,999 items per request.
* **Dedicated Cloud and Self-Managed**: Only support one email per user.
* **Team deletion**: Not supported through SCIM (use the W\&B web interface).
* **User reactivation**: Not supported in Multi-tenant Cloud environments.
* **Seat limits**: Operations may fail if organization seat limits are reached.


# Use service accounts to automate workflows
Source: https://docs.wandb.ai/platform/hosting/iam/service-accounts

Manage automated or non-interactive workflows using org and team scoped service accounts

A service account represents a non-human or machine user that can automatically perform common tasks across projects within a team or across teams. Service accounts are ideal for CI/CD pipelines, automated training jobs, and other machine-to-machine workflows.

This page explains the scopes available for service accounts, walks through how to create and manage them, and outlines best practices for using them safely in production automation. It's intended for organization and team admins who provision credentials for automated systems.

## Key benefits

Key benefits of service accounts:

* **No license consumption**: Service accounts do not consume user seats or licenses
* **Dedicated API keys**: Secure credentials for automated workflows
* **User attribution**: Optionally attribute automated runs to human users
* **Enterprise-ready**: Built for production automation at scale
* **Delegated operations**: Service accounts operate on behalf of the user or organization that creates them

## Overview

Service accounts provide a secure way to automate W\&B workflows without using personal user credentials or hardcoded credentials. You can create them at two scopes:

* **Organization-scoped**: Created by org admins, with access across all teams.
* **Team-scoped**: Created by team admins, with access limited to a specific team.

A service account's API key lets the caller read from or write to projects within the service account's scope. This enables centralized management of automated workflows for experiment tracking in W\&B Models or logging traces in W\&B Weave.

Service accounts are useful for:

* **CI/CD pipelines**: Automatically log model training runs from GitHub Actions, GitLab CI, or Jenkins.
* **Scheduled jobs**: Nightly model retraining, periodic evaluation runs, or data validation workflows.
* **Production monitoring**: Log inference metrics and model performance from production systems.
* **Jupyter notebooks**: Shared notebooks in JupyterHub or Google Colab environments.
* **Kubernetes jobs**: Automated workflows running in Kubernetes clusters.
* **Airflow/Prefect/Dagster**: ML pipeline orchestration tools.

<Note>
  Service accounts are available on [Dedicated Cloud](/platform/hosting/hosting-options/dedicated-cloud), [Self-Managed instances](/platform/hosting/hosting-options/self-managed) with an enterprise license, and enterprise accounts in [Multi-tenant Cloud](/platform/hosting/hosting-options/multi_tenant_cloud).
</Note>

## Organization-scoped service accounts

Use an organization-scoped service account when your automation needs to read or write across projects in multiple teams. Service accounts scoped to an organization have permissions to read and write in all projects in the organization, regardless of the team, with the exception of [restricted projects](/platform/hosting/iam/access-management/restricted-projects/#visibility-scopes). Before an organization-scoped service account can access a restricted project, an admin of that project must explicitly add the service account to the project.

### Create an organization-scoped service account

To create a new organization-scoped service account and API key:

1. Log in to W\&B, click your user profile icon, then:
   * **Dedicated Cloud** or **Self-Managed**: Click **Organization Dashboard**, then click **Service Accounts**.
   * **Multi-tenant Cloud**: Click **Service Accounts**.
2. Click **Create service account**.
3. Provide a name and select a default team.
4. Click **Create**.
5. Find the service account you just created.
6. Click the **action (<Icon icon="ellipsis" />)** menu, then click **Create API key**.
7. Provide a name for the API key, then click **Create**.
8. Copy the API key and store it securely.
9. Click **Done**.

<Warning>
  The full API key is only shown once at creation time. After you close the dialog, you cannot view the full API key again. Only the key ID (first part of the key) is visible in your settings. If you lose the full API key, you must create a new API key.
</Warning>

<Note>
  An organization-scoped service account requires a default team, even though it has access to non-restricted projects owned by all teams within the organization. This helps to prevent a workload from failing if the `WANDB_ENTITY` variable isn't set in the environment for your model training or generative AI app. To use an organization-scoped service account for a project in a different team, you must set the `WANDB_ENTITY` environment variable to that team.
</Note>

## Team-scoped service accounts

Use a team-scoped service account when you want to limit automation to a single team's projects, following the principle of least privilege. A team-scoped service account can read and write in all projects within its team, except to [restricted projects](/platform/hosting/iam/access-management/restricted-projects/#visibility-scopes) in that team. Before a team-scoped service account can access a restricted project, an admin of that project must explicitly add the service account to the project.

### Create a team-scoped service account

To create a new team-scoped service account and API key:

1. In your team's settings, click **Service Accounts**.
2. Click **New Team Service Account**.
3. Provide a name for the service account.
4. Set Authentication Method to **Generate API key** (default). If you select **Federated Identity**, the service account cannot own API keys.
5. Click **Create**.
6. Find the service account you just created.
7. Click the **action (<Icon icon="ellipsis" />)** menu, then click **Create API key**.
8. Provide a name for the API key, then click **Create**.
9. Copy the API key and store it securely.
10. Click **Done**.

<Warning>
  The full API key is only shown once at creation time. After you close the dialog, you cannot view the full API key again. Only the key ID (first part of the key) is visible in your settings. If you lose the full API key, you must create a new API key.
</Warning>

### Create additional API keys for a service account

To create an API key owned by a service account:

1. Navigate to the **Service Accounts** tab in your team or organization settings.
2. Find the service account in the list.
3. Click the **action (<Icon icon="ellipsis" />)** menu, then click **Create API key**.
4. Provide a name for the API key, then click **Create**.
5. Copy the displayed API key immediately and store it securely.
6. Click **Done**.

You can create multiple API keys for a single service account to support different environments or workflows.

<Warning>
  The full API key is only shown once at creation time. After you close the dialog, you cannot view the full API key again. Only the key ID (first part of the key) is visible in your settings. If you lose the full API key, you must create a new API key.
</Warning>

### Delete a service account API key

To delete an API key owned by an organization or team service account:

1. Go to [Organization settings](https://wandb.ai/account-settings/), then click **API Keys**.
2. Find the API key. The list includes all API keys owned by organization and team service accounts. You can search or filter by key name or ID, and you can sort by any column.
3. Click the delete button.

If you don't configure a team in your model training or generative AI app environment that uses a team-scoped service account, the model runs or weave traces log to the named project within the service account's parent team. In such a scenario, user attribution using the `WANDB_USERNAME` or `WANDB_USER_EMAIL` variables *doesn't work* unless the referenced user is part of the service account's parent team.

<Warning>
  A team-scoped service account can't log runs to a [team or restricted-scoped project](/platform/hosting/iam/access-management/restricted-projects/#visibility-scopes) in a team different from its parent team, but it can log runs to an open visibility project within another team.
</Warning>

### External service accounts

If you'd prefer to issue credentials through your own identity provider instead of managing W\&B-native API keys, use external service accounts. In addition to built-in service accounts, W\&B also supports team-scoped external service accounts with the W\&B SDK and CLI using [Identity federation](./identity_federation#external-service-accounts) with identity providers (IdPs) that issue JSON Web Tokens (JWTs).

## Best practices

After you create the service accounts you need, follow these recommendations to ensure secure and efficient use of service accounts in your organization:

* **Use a secrets manager**: Store service account API keys in a secure secrets management system (for example, AWS Secrets Manager, HashiCorp Vault, Azure Key Vault) rather than in plain text configuration files.

* **Principle of least privilege**: Create team-scoped service accounts when possible, rather than organization-scoped accounts, to limit access to only necessary projects.

* **Unique service accounts per use case**: Create separate service accounts for different automation workflows (for example, one for CI/CD, another for scheduled retraining) to improve auditability and enable granular access control.

* **Regular audits**: Periodically review active service accounts and remove those no longer in use. Check the audit logs to monitor service account activity.

* **Secure API key handling**:
  * Never commit API keys to version control.
  * Use environment variables to pass keys to applications.
  * Rotate keys if they're accidentally exposed.

* **Naming conventions**: Use descriptive names that indicate the service account's purpose:
  * Good: `ci-model-training`, `nightly-eval-pipeline`, `prod-inference-monitor`
  * Avoid: `service-account-1`, `test-sa`, `temp`

* **User attribution**: When multiple team members use the same automation workflow, set `WANDB_USERNAME` or `WANDB_USER_EMAIL` to track who triggered each run:
  ```bash theme={null}
  export WANDB_API_KEY="[SERVICE_ACCOUNT_KEY]"
  export WANDB_USERNAME="john.doe@company.com"
  ```

* **Environment configuration**: For team-scoped service accounts, always set the `WANDB_ENTITY` to ensure runs log to the correct team:
  ```bash theme={null}
  export WANDB_ENTITY="ml-team"
  export WANDB_PROJECT="production-models"
  ```

* **Error handling**: Implement proper error handling and alerts for failed authentication to identify issues with service account credentials quickly.

* **Documentation**: Maintain documentation of:
  * Which service accounts exist and their purposes.
  * Which systems or workflows use each service account.
  * Contact information for the team responsible for each account.

## Troubleshooting

If a service account isn't behaving as expected, check the following common issues and solutions:

* **"Unauthorized" errors**: Verify the API key is correctly set and the service account has access to the target project.
* **Runs not appearing**: Check that `WANDB_ENTITY` is set to the correct team name.
* **User attribution not working**: Ensure the user specified in `WANDB_USERNAME` is a member of the team.
* **Access denied to restricted projects**: Explicitly add the service account to the restricted project's access list.


# Configure SSO with OIDC
Source: https://docs.wandb.ai/platform/hosting/iam/sso

Configure SSO using OpenID Connect with identity providers like Okta, Azure AD, and AWS Cognito for W&B instances.

This guide is for administrators of W\&B Dedicated Cloud or Self-Managed instances who want to enable single sign-on (SSO) using an OpenID Connect (OIDC) compatible identity provider. By the end, you've configured your identity provider, connected it to W\&B so that users can sign in through your organization's existing identity system, and you can manage user identities and group memberships through providers like Okta, Keycloak, Auth0, Google, and Entra.

## OpenID Connect

W\&B supports the following OIDC authentication flows for integrating with external Identity Providers (IdPs):

* Implicit flow with form post.
* Authorization code flow with Proof Key for Code Exchange (PKCE).

These flows authenticate users and provide W\&B with the identity information (in the form of ID tokens) needed to manage access control.

The ID token is a JWT that contains the user's identity information, such as their name, username, email, and group memberships. W\&B uses this token to authenticate the user and map them to appropriate roles or groups in the system.

In the context of W\&B, access tokens authorize requests to APIs on behalf of the user, but because W\&B's primary concern is user authentication and identity, it only requires the ID token.

You can use environment variables to [configure IAM options](/platform/hosting/iam/advanced_env_vars) for your [Dedicated Cloud](/platform/hosting/hosting-options/dedicated-cloud) or [Self-Managed](/platform/hosting/hosting-options/self-managed) instance.

To assist with configuring Identity Providers for [Dedicated Cloud](/platform/hosting/hosting-options/dedicated-cloud) or [Self-Managed](/platform/hosting/hosting-options/self-managed) deployments, follow these guidelines. If you're using W\&B Multi-tenant Cloud, reach out to [support@wandb.com](mailto:support@wandb.com) for assistance.

## Configure your IdP

The following sections describe how to configure your identity provider (IdP) for OIDC. Complete the configuration steps for your IdP first. You use the resulting **Client ID**, **Issuer URL**, and (optionally) **Client Secret** when you set up SSO in W\&B in the next section. Select the tab for your IdP for details.

<Tabs>
  <Tab title="Cognito">
    Follow this procedure to set up AWS Cognito as your IdP. At the end, you have a **Client ID** and **OIDC issuer URL** to use when you configure W\&B.

    1. Sign in to your AWS account and navigate to the [AWS Cognito](https://aws.amazon.com/cognito/) App.

       <Frame>
         <img alt="AWS Cognito setup" />
       </Frame>

    2. Provide an allowed callback URL to configure the application in your IdP. Add `http(s)://[YOUR-W-AND-B-HOST]/oidc/callback` as the callback URL. Replace `[YOUR-W-AND-B-HOST]` with your W\&B host path.

    3. If your IdP supports universal logout, set the Logout URL to `http(s)://[YOUR-W-AND-B-HOST]`. Replace `[YOUR-W-AND-B-HOST]` with your W\&B host path.

       For example, if your application runs at `https://wandb.mycompany.com`, replace `[YOUR-W-AND-B-HOST]` with `wandb.mycompany.com`.

       The following image demonstrates how to provide allowed callback and sign-out URLs in AWS Cognito.

       <Frame>
         <img alt="Host configuration" />
       </Frame>

       `wandb/local` uses the [`implicit` grant with the `form_post` response type](https://auth0.com/docs/get-started/authentication-and-authorization-flow/implicit-flow-with-form-post) by default.

       You can also configure `wandb/local` to perform an `authorization_code` grant that uses the [PKCE Code Exchange](https://www.oauth.com/oauth2-servers/pkce/) flow.

    4. Select one or more OAuth grant types to configure how AWS Cognito delivers tokens to your app.

    5. W\&B requires specific OpenID Connect (OIDC) scopes. Select the following from AWS Cognito App:

       * `openid`
       * `profile`
       * `email`

       For example, your AWS Cognito App UI should look similar to the following image:

       <Frame>
         <img alt="Required fields" />
       </Frame>

       Select the **Auth Method** in the settings page or set the `OIDC_AUTH_METHOD` environment variable to specify which grant `wandb/local` uses.

       You must set the **Auth Method** to `pkce`.

    6. You need a **Client ID** and the URL of your OIDC issuer. The OpenID discovery document must be available at `$OIDC_ISSUER/.well-known/openid-configuration`.

       For example, you can generate your issuer URL by appending your User Pool ID to the Cognito IdP URL from the **App Integration** tab within the **User Pools** section:

       <Frame>
         <img alt="AWS Cognito issuer URL" />
       </Frame>

       Don't use the Cognito domain for the IdP URL. Cognito provides its discovery document at `https://cognito-idp.$REGION.amazonaws.com/$USER_POOL_ID`.

    Next, [Set up SSO in W\&B](#set-up-sso-in-w%26b).
  </Tab>

  <Tab title="Okta">
    Follow this procedure to set up Okta as your IdP. At the end, you have a **Client ID** and **OIDC issuer URL** to use when you configure W\&B.

    1. Sign in to the [Okta Portal](https://login.okta.com/).

    2. On the left side, select **Applications** and then **Applications** again.
       <Frame>
         <img alt="Okta Applications menu" />
       </Frame>

    3. Click **Create App integration**.
       <Frame>
         <img alt="Create App integration button" />
       </Frame>

    4. On the screen named **Create a new app integration**, select **OIDC - OpenID Connect** and **Single-Page Application**. Then click **Next**.
       <Frame>
         <img alt="OIDC Single-Page Application selection" />
       </Frame>

    5. On the screen named **New Single-Page App Integration**, complete the values as follows and click **Save**:
       * **App integration name**, for example `W&B`.
       * **Grant type**: Select both **Authorization Code** and **Implicit (hybrid)**.
       * **Sign-in redirect URIs**: `https://[YOUR-W-AND-B-URL]/oidc/callback`.
       * **Sign-out redirect URIs**: `https://[YOUR-W-AND-B-URL]/logout`.
       * **Assignments**: Select **Skip group assignment for now**.
       <Frame>
         <img alt="Single-Page App configuration" />
       </Frame>

    6. On the overview screen of the Okta application you created, make note of the **Client ID** under **Client Credentials** under the **General** tab:
       <Frame>
         <img alt="Okta Client ID location" />
       </Frame>

    7. To identify the Okta **OIDC Issuer URL**, select **Settings** and then **Account** on the left side.
       The Okta UI shows the company name under **Organization Contact**.
       <Frame>
         <img alt="Okta organization settings" />
       </Frame>

    The OIDC issuer URL has the following format: `https://[COMPANY].okta.com`. Replace `[COMPANY]` with the corresponding value. Make note of it.

    Next, [Set up SSO in W\&B](#set-up-sso-in-w%26b).
  </Tab>

  <Tab title="Entra">
    Azure AD (Entra ID) supports two OIDC configuration modes for W\&B. Choose the configuration that matches your security requirements:

    * [Public client](#public-client): Uses PKCE without a client secret. Simpler to configure, suitable for most deployments.
    * [Confidential client](#confidential-client): Uses PKCE with a client secret. Required if you need to set the `GORILLA_OIDC_SECRET` environment variable.

    <Warning>
      Don't mix configurations. If you select **Single-page application** in Azure AD, don't provide a client secret. If you need a client secret, you must select **Web** as the platform type.
    </Warning>

    <AccordionGroup>
      <Accordion title="Public client">
        Use this configuration if you don't need to specify a client secret. It's suitable for deployments without advanced security requirements.

        1. Sign in to the [Azure Portal](https://portal.azure.com/).
        2. Navigate to **Microsoft Entra ID** service and select **App registrations** from the left sidebar.
        3. Click **New registration** at the top of the page.
        4. On the **Register an application** screen, configure the following:
           * **Name**: Enter a descriptive name.
           * **Supported account types**: Keep the default **Single tenant** or modify as needed.
           * **Redirect URI**: Select platform type **Single-page application** and enter `https://[YOUR-W-AND-B-URL]/oidc/callback`.
           * Click **Register**.
        5. After registration, note the following values from the Overview page:
           * **Application (client) ID**: Your OIDC Client ID.
           * **Directory (tenant) ID**: Your OIDC Issuer URL.
           <Frame>
             <img alt="Application and Directory IDs" />
           </Frame>
        6. Configure authentication settings:
           * Select **Authentication** from the left sidebar.
           * Under **Front-channel logout URL**, enter `https://[YOUR-W-AND-B-URL]/logout`.
           * Click **Save**.

        Make a note of the following details:

        * **OIDC Client ID**: The Application (client) ID from step 5.
        * **OIDC Issuer URL**: `https://login.microsoftonline.com/[TENANT-ID]/v2.0` (replace `[TENANT-ID]` with your Directory ID from step 5).

        When configuring W\&B, use:

        * **Auth Method**: `pkce`.
        * **OIDC Client Secret**: Leave empty (don't set `GORILLA_OIDC_SECRET`).

        Next, [Set up SSO in W\&B](#set-up-sso-in-w%26b).
      </Accordion>

      <Accordion title="Confidential client">
        Use this configuration if you need to authenticate using a client secret.

        1. Sign in to the [Azure Portal](https://portal.azure.com/).
        2. Navigate to **Microsoft Entra ID** service and select **App registrations** from the left sidebar.
        3. Click **New registration** at the top of the page.
        4. On the **Register an application** screen, configure the following:
           * **Name**: Enter a descriptive name.
           * **Supported account types**: Keep the default **Single tenant** or modify as needed.
           * **Redirect URI**: Select platform type **Web** and enter `https://[YOUR-W-AND-B-URL]/oidc/callback`.
           * Click **Register**.
        5. After registration, note the following values from the Overview page:
           * **Application (client) ID**: Your OIDC Client ID.
           * **Directory (tenant) ID**: Your OIDC Issuer URL.
           <Frame>
             <img alt="Application and Directory IDs" />
           </Frame>
        6. Configure authentication settings:
           * Select **Authentication** from the left sidebar.
           * Under **Front-channel logout URL**, enter `https://[YOUR-W-AND-B-URL]/logout`.
           * Click **Save**.
        7. Create a client secret:
           * Select **Certificates & secrets** from the left sidebar.
           * Click **New client secret**.
           * Add a description for the secret.
           * Choose an expiration period.
           * Click **Add**. <Warning>Copy and save the secret **Value** immediately (not the Secret ID).</Warning>
           <Frame>
             <img alt="Client secret value" />
           </Frame>

        Make a note of the following details:

        * **OIDC Client ID**: The Application (client) ID from step 5.
        * **OIDC Client Secret**: The secret value from step 7.
        * **OIDC Issuer URL**: `https://login.microsoftonline.com/[TENANT-ID]/v2.0` (replace `[TENANT-ID]` with your Directory ID from step 5).

        When configuring W\&B, use:

        * **Auth Method**: `pkce`.
        * **OIDC Client Secret**: Set the `GORILLA_OIDC_SECRET` environment variable to the secret value from step 7.

        <Note>
          The v2.0 endpoint supports both personal Microsoft accounts and work/school accounts. If your organization requires the v1.0 endpoint, use `https://login.microsoftonline.com/[TENANT-ID]` instead.
        </Note>

        Next, [Set up SSO in W\&B](#set-up-sso-in-w%26b).
      </Accordion>
    </AccordionGroup>
  </Tab>
</Tabs>

## Set up SSO in W\&B

After you finish configuring your IdP, complete the following steps in W\&B to connect the IdP and enable SSO for your instance.

To set up SSO, you must have administrator privileges and the following information:

* **OIDC Client ID**.
* **OIDC Auth method** (`implicit` or `pkce`).
* **OIDC Issuer URL**.
* **OIDC Client Secret** (optional, depends on how you've set up your IdP).

If your IdP requires an OIDC Client Secret, specify it by passing the [environment variables](/platform/hosting/env-vars) `GORILLA_OIDC_SECRET`:

* In the W\&B App, go to **System Console** > **Settings** > **Advanced** > **User Spec** and add `GORILLA_OIDC_SECRET` to the `extraENV` section as shown in the following example.
* In Helm, configure `values.global.extraEnv` as shown in the following example.
  ```yaml theme={null}
  values:
  global:
      extraEnv:
      GORILLA_OIDC_SECRET="[YOUR-SECRET]"
  ```

<Note>
  If you can't sign in to your instance after configuring SSO, you can restart the instance with the `LOCAL_RESTORE=true` environment variable set. This outputs a temporary password to the containers logs and disables SSO. After you resolve any issues with SSO, you must remove that environment variable to enable SSO again.
</Note>

<Tabs>
  <Tab title="System Console">
    Use this tab if you deploy W\&B with the W\&B Kubernetes Operator. The System Console is the successor to the System Settings page. It's available with the [W\&B Kubernetes Operator](/platform/hosting/self-managed/operator) based deployment.

    1. Refer to [Access the W\&B Management Console](/platform/hosting/self-managed/operator#access-the-wb-management-console).

    2. Navigate to **Settings**, then **Authentication**. Select **OIDC** in the **Type** dropdown.
       <Frame>
         <img alt="System Console OIDC configuration" />
       </Frame>

    3. Enter the values.

    4. Click **Save**.

    5. Sign out and then sign back in, this time using the IdP sign-in screen.

    ## Find your customer namespace

    Before you can configure team-level BYOB with CoreWeave storage on W\&B Dedicated Cloud or Self-Managed, you must obtain your organization's **Customer Namespace**. You can view and copy it from the bottom of the **Authentication** tab.

    For detailed instructions on configuring CoreWeave storage with your Customer Namespace, see [CoreWeave requirements for Dedicated Cloud or Self-Managed](/platform/hosting/data-security/secure-storage-connector#coreweave-customer-namespace).
  </Tab>

  <Tab title="System settings">
    1. Sign in to your W\&B instance.

    2. Navigate to the W\&B App.

       <Frame>
         <img alt="W&B App navigation" />
       </Frame>

    3. From the dropdown, select **System Settings**:

       <Frame>
         <img alt="System Settings dropdown" />
       </Frame>

    4. Enter your **Issuer**, **Client ID**, and **Authentication Method**.

    5. Select **Update settings**.

    <Frame>
      <img alt="Update settings button" />
    </Frame>
  </Tab>
</Tabs>

<Note>
  If you can't sign in to your instance after configuring SSO, you can restart the instance with the `LOCAL_RESTORE=true` environment variable set. This outputs a temporary password to the containers logs and disables SSO. After you resolve any issues with SSO, you must remove that environment variable to enable SSO again.
</Note>

## Security Assertion Markup Language (SAML)

W\&B doesn't support SAML.


# Manage bucket storage and costs
Source: https://docs.wandb.ai/platform/hosting/managing-bucket-storage

Understand how W&B uses object storage, how deletion maps to bucket bytes, and how to reduce storage usage and costs.

When you use [Bring your own bucket (BYOB)](/platform/hosting/data-security/secure-storage-connector), [W\&B Dedicated Cloud](/platform/hosting/hosting-options/dedicated-cloud), or [W\&B Self-Managed](/platform/hosting/hosting-options/self-managed), your team often pays cloud storage providers directly. This page explains what occupies your bucket, how W\&B removes objects after deletion in the app or API, and what you should expect. Use it to understand bucket usage, plan cleanup, and set realistic expectations for when storage is reclaimed.

This page is intended for W\&B administrators and operators who manage object storage for Self-Managed, Dedicated Cloud, or BYOB deployments.

## What uses bucket space

W\&B stores several categories of data in your configured object storage. The [BYOB overview](/platform/hosting/data-security/secure-storage-connector#data-stored-in-the-central-database-vs-buckets) lists examples, including experiment files and metrics, artifact files, media files, run files, and exported history in Parquet form. Together these drive bucket size and cost.

## How W\&B removes data from storage

Deletion in the W\&B App or [Public API](/models/ref/python/public-api/api) updates W\&B metadata first. Removing a run, artifact, or file from the product doesn't guarantee an immediate drop in reported bucket usage. Object storage cleanup runs as background work that can lag, especially on busy instances.

### Artifacts

Deleted artifacts are soft-deleted, then processed by artifact garbage collection. Self-managed deployments must set `GORILLA_ARTIFACT_GC_ENABLED` and meet provider requirements such as versioning or soft delete. See [Delete an artifact](/models/artifacts/delete-artifacts) and [Configure environment variables](/platform/hosting/env-vars).

### Run data and run files

After runs or run-associated files are deleted, permanent removal of the underlying stored objects is controlled separately from artifacts. On Dedicated Cloud and Self-Managed deployments, `GORILLA_DATA_RETENTION_PERIOD` sets how long deleted run data is retained before it can be removed from storage. This setting does not delete artifacts. See [Configure environment variables](/platform/hosting/env-vars), [Data retention policy](/platform/hosting/hosting-options/dedicated-cloud#data-retention-policy) for Dedicated Cloud, and [Delete runs](/models/runs/delete-runs#when-deleted-run-data-is-removed-from-storage) for how run and file deletion relates to storage.

## What to expect from background cleanup

Garbage collection and related jobs that free object storage run without timing guarantees. W\&B does not guarantee that a given object disappears from your bucket within a specific time after you delete content in the UI or API. For projects with a large number of files per run, such as when logging many media files per run, expect longer delays before storage usage is released.

Monitor your bucket in your cloud provider and contact [W\&B Support](mailto:support@wandb.ai) or your account team if cleanup appears stuck.

## Reduce bucket usage

This section describes the recommended order of operations for freeing space in your bucket, starting with safe product flows and ending with direct bucket operations that require more care.

Use supported product flows first:

* [Delete runs in the W\&B App](/models/runs/delete-runs#ui) or [with Python](/models/runs/delete-runs#python) when you no longer need them.
* [Delete artifacts](/models/artifacts/delete-artifacts) you no longer need, and use [Artifact TTL](/models/artifacts/ttl) where it fits your workflow.

If you must reclaim space immediately, operators with access to the bucket may delete specific object keys directly in cloud storage.

:::caution
Deleting object keys directly in your bucket bypasses W\&B and can cause data loss or break access for the app. Review the following before you proceed:

* Objects you remove are **no longer available to download** through W\&B.
* You should delete **only** keys you intend to remove. Incorrect deletes can break access to data the app still references.
* If your bucket uses **object versioning** or **provider soft delete** (for example on Google Cloud Storage), storage charges can persist until non-current versions or soft-deleted objects expire under your cloud lifecycle rules.
  :::

For high-level usage in W\&B Multi-tenant Cloud, organization admins can review storage-related usage from organization settings. See [Billing settings](/platform/app/settings-page/billing-settings).

## Troubleshooting

If deletions don't appear correctly in the W\&B App after you use the Public API, upgrade the W\&B Python SDK to a current release and retry. Large per-run file counts can increase how long background cleanup takes across the instance.

For scripted cleanup patterns that match your deployment, contact [W\&B Support](mailto:support@wandb.ai) or your account team.

## Related documentation

For more information, see the following resources:

* [Delete runs](/models/runs/delete-runs#delete-runs)
* [Delete an artifact](/models/artifacts/delete-artifacts)
* [Configure environment variables](/platform/hosting/env-vars)
* [Bring your own bucket (BYOB)](/platform/hosting/data-security/secure-storage-connector)


# Track user activity with audit logs
Source: https://docs.wandb.ai/platform/hosting/monitoring-usage/audit-logging

Access, fetch, and analyze W&B audit logs across deployment types, including the log schema and tracked actions.

Use W\&B audit logs to track user activity within your organization and to conform to your enterprise governance requirements. This page is for organization-level admins who need to access, fetch, and analyze audit log data across W\&B deployment types. Audit logs are available in JSON format. Refer to [Audit log schema](#audit-log-schema).

How you access audit logs depends on your W\&B platform deployment type:

| W\&B Platform deployment type                                              | Audit logs access mechanism                                                                                                                                                                                                                                                                                                 |
| -------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| [Dedicated Cloud](/platform/hosting/hosting-options/dedicated-cloud)       | <ul><li>[Instance-level BYOB](/platform/hosting/data-security/secure-storage-connector): Synced to instance-level bucket (BYOB) every 10 minutes. Also available with [the API](#fetch-audit-logs-using-api).</li><li>Default instance-level storage: Available only with [the API](#fetch-audit-logs-using-api).</li></ul> |
| [Multi-tenant Cloud](/platform/hosting/hosting-options/multi_tenant_cloud) | Available for Enterprise plans only. Available only with [the API](#fetch-audit-logs-using-api).                                                                                                                                                                                                                            |
| [Self-Managed](/platform/hosting/hosting-options/self-managed)             | Synced to instance-level bucket every 10 minutes. Also available with [the API](#fetch-audit-logs-using-api).                                                                                                                                                                                                               |

After you fetch audit logs, you can analyze them with tools like [Pandas](https://pandas.pydata.org/docs/index.html), [Amazon Redshift](https://aws.amazon.com/redshift/), [Google BigQuery](https://cloud.google.com/bigquery), or [Microsoft Fabric](https://www.microsoft.com/microsoft-fabric). Some audit log analysis tools don't support JSON. Refer to the documentation for your analysis tool for guidelines and requirements to transform the JSON-formatted audit logs before analysis.

For more details about the format of the logs, see [Audit log schema](#audit-log-schema) and [Actions](#actions).

## Audit log retention

The following recommendations help you retain audit logs to meet your organization's governance and compliance obligations:

* If you must retain audit logs for a specific period of time, W\&B recommends periodically transferring logs to long-term storage, either with storage buckets or the Audit Logging API.
* If you are subject to the [Health Insurance Portability and Accountability Act of 1996 (HIPAA)](https://hhs.gov/hipaa/for-professionals/index.html), you must retain audit logs for a minimum of 6 years in an environment where no internal or external actor can delete or modify them before the end of the mandatory retention period. For HIPAA-compliant [Dedicated Cloud](/platform/hosting/hosting-options/dedicated-cloud) instances with [BYOB](/platform/hosting/data-security/secure-storage-connector), you must configure guardrails for your managed storage, including any long-term retention storage.

## Audit log schema

Use this schema to interpret the fields returned in each audit log entry. The following table shows all keys that can appear in an audit log entry, ordered alphabetically. Depending on the action and the circumstances, a specific log entry may include only a subset of the possible fields.

| Key                       | Definition                                                                                                                                                          |
| ------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `action`                  | The [action](#actions) of the event.                                                                                                                                |
| `actor_email`             | The email address of the user that initiated the action, if applicable.                                                                                             |
| `actor_ip`                | The IP address of the user that initiated the action.                                                                                                               |
| `actor_user_id`           | The ID of the logged-in user who performed the action, if applicable.                                                                                               |
| `artifact_asset`          | The artifact ID associated with the action, if applicable.                                                                                                          |
| `artifact_digest`         | The artifact digest associated with the action, if applicable.                                                                                                      |
| `artifact_qualified_name` | The full name of the artifact associated with the action, if applicable.                                                                                            |
| `artifact_sequence_asset` | The artifact sequence ID associated with the action, if applicable.                                                                                                 |
| `cli_version`             | The version of the Python SDK that initiated the action, if applicable.                                                                                             |
| `entity_asset`            | The entity or team ID associated with the action, if applicable.                                                                                                    |
| `entity_name`             | The entity or team name associated with the action, if applicable.                                                                                                  |
| `project_asset`           | The project associated with the action, if applicable.                                                                                                              |
| `project_name`            | The name of the project associated with the action, if applicable.                                                                                                  |
| `report_asset`            | The report ID associated with the action, if applicable.                                                                                                            |
| `report_name`             | The name of the report associated with the action, if applicable.                                                                                                   |
| `response_code`           | The HTTP response code for the action, if applicable.                                                                                                               |
| `timestamp`               | The time of the event in [RFC3339 format](https://www.rfc-editor.org/rfc/rfc3339). For example, `2023-01-23T12:34:56Z` represents January 23, 2023 at 12:34:56 UTC. |
| `user_asset`              | The user asset the action impacts (rather than the user performing the action), if applicable.                                                                      |
| `user_email`              | The email address of the user the action impacts (rather than the email address of the user performing the action), if applicable.                                  |

### Personally identifiable information (PII)

Personally identifiable information (PII), such as email addresses and the names of projects, teams, and reports, is available only with the API endpoint option:

* For [Self-Managed](/platform/hosting/hosting-options/self-managed) and [Dedicated Cloud](/platform/hosting/hosting-options/dedicated-cloud), an organization admin can [exclude PII](#exclude-pii) when fetching audit logs.
* For [Multi-tenant Cloud](/platform/hosting/hosting-options/multi_tenant_cloud), the API endpoint always returns relevant fields for audit logs, including PII. This isn't configurable.

## Before you begin

Before you fetch audit logs, confirm that you meet the following prerequisites for your deployment type:

* Organization-level admins can fetch audit logs. If you receive a `403` error, ensure that you or your service account has adequate permission.
* **Multi-tenant Cloud**: If you're a member of multiple Multi-tenant Cloud organizations, you must configure the **Default API organization**, which determines where audit logging API calls are routed. Otherwise, you receive the following error:

  ```text theme={null}
  user is associated with multiple organizations but no valid org ID found in user info
  ```

  To specify your default API organization:

  1. Click your profile image, then click **User Settings**.
  2. For **Default API organization**, select an organization.

  This doesn't apply to a service account, which can be a member of only one Multi-tenant Cloud organization.

## Fetch audit logs

To retrieve audit logs from the W\&B Audit Logging API, follow these steps. After you complete this procedure, you have a set of newline-separated JSON objects you can analyze using your preferred tooling.

1. Determine the correct API endpoint for your instance:

   * [Self-Managed](/platform/hosting/hosting-options/self-managed): `[WANDB-PLATFORM-URL]/admin/audit_logs`
   * [Dedicated Cloud](/platform/hosting/hosting-options/dedicated-cloud): `[INSTANCE-NAME].wandb.io/admin/audit_logs`
   * [Multi-tenant Cloud (Enterprise required)](/platform/hosting/hosting-options/multi_tenant_cloud): `https://api.wandb.ai/audit_logs`

   In the following steps, replace `[API-ENDPOINT]` with your API endpoint.
2. Optional: Construct query parameters to append to the endpoint. In the following steps, replace `[PARAMETERS]` with the resulting string.
   * `anonymize`: If the URL includes the parameter `anonymize=true`, W\&B removes any PII. Otherwise, PII is included. Refer to [Exclude PII when fetching audit logs](#exclude-pii-when-fetching-audit-logs). Not supported for Multi-tenant Cloud, where all fields are included, including PII.
   * Configure the date window of logs to fetch with a combination of `numDays` and `startDate`. Each parameter is optional, and they interact.
     * If neither parameter is included, only today's logs are fetched.
     * `numDays`: An integer that indicates the number of days backward from `startDate` to fetch logs. If you omit it or set it to `0`, logs are fetched for `startDate` only. On **Multi-tenant Cloud**, you can fetch a maximum of 7 days of audit logs, even if you set `numDays` to a larger value.
     * `startDate`: Optional. Use `startDate=YYYY-MM-DD` to set the newest calendar day in the range. If you omit it or set it to today, the range ends today and `numDays` counts backward from that day.
       * Supported in **Multi-tenant Cloud** only with an Enterprise license.
       * Supported in **Dedicated Cloud** and **Self-Managed** v0.80.0 and above.
3. Construct the fully qualified endpoint URL in the format `[API-ENDPOINT]?[PARAMETERS]`.
4. Execute an HTTP `GET` request on the fully qualified API endpoint with a web browser or a tool like [Postman](https://www.postman.com/downloads/), [HTTPie](https://httpie.io/), or cURL.

The API response contains newline-separated JSON objects. Objects include the fields described in the [schema](#audit-log-schema), just like when audit logs are synced to an instance-level bucket. In those cases, the audit logs are located in the `/wandb-audit-logs` directory in your bucket.

### Use basic authentication

You must authenticate each request to the audit logs API. To use basic authentication with your API key to access the audit logs API, set the HTTP request's `Authorization` header to the string `Basic` followed by a space, then the base64-encoded string in the format `[USERNAME]:[API-KEY]`. In other words, replace the username and API key with your values separated with a `:` character, then base64-encode the result. For example, to authorize as `demo:p@55w0rd`, set the header to `Authorization: Basic ZGVtbzpwQDU1dzByZA==`.

### Exclude PII when fetching audit logs <a />

For [Self-Managed](/platform/hosting/hosting-options/self-managed) and [Dedicated Cloud](/platform/hosting/hosting-options/dedicated-cloud), a W\&B organization or instance admin can exclude PII when fetching audit logs. For [Multi-tenant Cloud](/platform/hosting/hosting-options/multi_tenant_cloud), the API endpoint always returns relevant fields for audit logs, including PII. This isn't configurable.

To exclude PII, pass the `anonymize=true` URL parameter. For example, to get audit logs for user activity within the last week and exclude PII, if your W\&B instance URL is `https://mycompany.wandb.io`, use an API endpoint like:

```text theme={null}
https://mycompany.wandb.io/admin/audit_logs?anonymize=true&[ADDITIONAL-PARAMETERS].
```

## Actions

Each audit log entry records one of the following actions. Use this reference to interpret the `action` field in a log entry. The following table describes possible actions that W\&B can record, sorted alphabetically.

| Action                        | Definition                                                                        |
| ----------------------------- | --------------------------------------------------------------------------------- |
| `artifact:create`             | Artifact is created.                                                              |
| `artifact:delete`             | Artifact is deleted.                                                              |
| `artifact:read`               | Artifact is read.                                                                 |
| `project:delete`              | Project is deleted.                                                               |
| `project:read`                | Project is read.                                                                  |
| `report:read`                 | Report is read. <sup><a href="#1">1</a></sup>                                     |
| `run:delete_many`             | Batch of runs is deleted.                                                         |
| `run:delete`                  | Run is deleted.                                                                   |
| `run:stop`                    | Run is stopped.                                                                   |
| `run:undelete_many`           | Batch of runs is restored from trash.                                             |
| `run:update_many`             | Batch of runs is updated.                                                         |
| `run:update`                  | Run is updated.                                                                   |
| `sweep:create_agent`          | Sweep agent is created.                                                           |
| `team:create_service_account` | Service account is created for the team.                                          |
| `team:create`                 | Team is created.                                                                  |
| `team:delete`                 | Team is deleted.                                                                  |
| `team:invite_user`            | User is invited to team.                                                          |
| `team:uninvite`               | User or service account is uninvited from team.                                   |
| `user:create_api_key`         | API key for the user or service account is created. <sup><a href="#1">1</a></sup> |
| `user:create`                 | User is created. <sup><a href="#1">1</a></sup>                                    |
| `user:deactivate`             | User is deactivated. <sup><a href="#1">1</a></sup>                                |
| `user:delete_api_key`         | API key for the user or service account is deleted. <sup><a href="#1">1</a></sup> |
| `user:initiate_login`         | User initiates log in. <sup><a href="#1">1</a></sup>                              |
| `user:login`                  | User logs in. <sup><a href="#1">1</a></sup>                                       |
| `user:logout`                 | User logs out. <sup><a href="#1">1</a></sup>                                      |
| `user:permanently_delete`     | User is permanently deleted. <sup><a href="#1">1</a></sup>                        |
| `user:reactivate`             | User is reactivated. <sup><a href="#1">1</a></sup>                                |
| `user:read`                   | User profile is read. <sup><a href="#1">1</a></sup>                               |
| `user:update`                 | User is updated. <sup><a href="#1">1</a></sup>                                    |

<a>1</a>: On [Multi-tenant Cloud](/platform/hosting/hosting-options/multi_tenant_cloud), audit logs are not collected for:

* Open or Public projects.
* The `report:read` action.
* `User` actions that aren't tied to a specific organization.


# W&B Mobile App (iOS)
Source: https://docs.wandb.ai/platform/hosting/monitoring-usage/mobile-app

Track training runs, review console logs, view line plots and system metrics, and explore W&B Models projects from your iPhone or iPad.

<Columns>
  <Frame>
    <img alt="W&B mobile app project panel with metric charts" />
  </Frame>

  <Frame>
    <img alt="W&B mobile app Manage notifications screen" />
  </Frame>

  <Frame>
    <img alt="W&B mobile app Projects list and team picker" />
  </Frame>

  <Frame>
    <img alt="W&B mobile app runs list for a project" />
  </Frame>
</Columns>

The W\&B Mobile App for iOS keeps you connected to your W\&B Models projects wherever you go. Use it on your iPhone or iPad to track training runs and read run logs in real time, view line plots and system metrics, explore project histories, and follow your team's progress so you can monitor experiments without opening your laptop. This page describes the app's features and shows you how to set up notifications.

<Info>
  The mobile app is only available for [Multi-tenant Cloud](/platform/hosting/hosting-options/multi_tenant_cloud) accounts. It isn't available for [Dedicated Cloud](/platform/hosting/hosting-options/dedicated-cloud) or [Self-managed](/platform/hosting/hosting-options/self-managed) deployments.
</Info>

## Download the app

Install the app from the App Store to get started.

<Card title="Download on the App Store" href="https://apps.apple.com/us/app/6755162576" icon="apple" />

## Features

The following features are available in the mobile app.

* **Explore projects**: Browse and search across your W\&B projects.
  * **Full project history**: Metric charts on project panels include data from *all runs* in the project, not only the latest run.
  * **Star projects**: See your most important projects at a glance. Click the star icon next to a project to star it. To filter the list to only starred projects, click the **Starred** tab at the top of the list.
* **Track experiments**: View run status, metrics, and line plots for your experiments in real time.
  * **Live updates**: Charts refresh as new data arrives. The **Runs** tab and project run lists update automatically when new runs start.
  * **Run overview tab**: On each run, open the **Overview** tab for a dedicated summary of run details alongside metrics and logs. Long run config values wrap to multiple lines instead of truncating.
  * **View system metrics**: On each run, open the **Metrics** tab to view [system metrics](/models/ref/python/experiments/system-metrics) that W\&B logs automatically during training, such as GPU utilization, CPU usage, memory, disk I/O, and network traffic. System metrics appear in a collapsible **System** section alongside your logged metrics. Each metric displays as a line chart with the same tooltip and search behavior as other metrics on the tab.
  * **Mobile-optimized panel grouping**: Panels are automatically grouped by name into collapsible sections. You can collapse the ungrouped metrics section to focus on named sections. Sections are a single level (not nested), and grouping follows the same rules as [workspace panels](/models/app/features/panels) in the W\&B web app, so the layout stays consistent when you move between desktop and your phone.
  * **Star panels**: See your most important panels at a glance. When viewing a run or a project, click the star icon at the top of a panel to star it. To filter the list to only starred panels, click the **Starred** tab at the top of the list.
  * **Search panels**: When viewing a run or a project, use the **panel search** field at the bottom of the screen to filter which runs appear in each chart. You can search with [JavaScript regular expressions](https://www.w3schools.com/js/js_regexp.asp) to match patterns in run names.
  * **Chart tooltips**: On line charts, tooltips show metric values with up to four decimal places so you can read small changes accurately.
  * **Stop runs**: When viewing an in-progress run, click the **action (<Icon icon="ellipsis" />)** menu, then click **Stop run**.
  * **View console logs**: View [console logs](/models/runs/view-logged-runs#logs) for active or completed runs, or download logs for completed runs. The app polls active runs for new log lines every 5 seconds. The most recent 10,000 lines display by default, and you can scroll backward to view older logs. Log search stays responsive even for large outputs.
* **Switch teams**: Use the **team picker** to change teams.
* **Stay informed**: Get updates on your experiments without opening your laptop using notifications. **Remove alerts** you no longer need from the **Notifications** tab by swiping them away.

## Set up notifications

Configure notifications to receive updates about your experiments on your mobile device. The following sections describe the two kinds of notifications you can configure.

### Metric threshold alerts

Use metric threshold alerts to get notified when a metric crosses a value you specify.

1. Navigate to a run.
2. Tap the graph to enter fullscreen view.
3. Tap the bell icon in the top right.
4. Set notifications that trigger when future runs cross the specified metric threshold.

### Run failure alerts

Use run failure alerts to get notified when runs in a project fail.

1. Navigate to a project.
2. Tap the **action (<Icon icon="ellipsis" />)** menu in the top right.
3. Select **Run failed alert** to receive notifications for future run failures in that project.

The **Manage notifications** screen lists both kinds of settings in one place, with short descriptions that explain what each section controls.


# View organization activity
Source: https://docs.wandb.ai/platform/hosting/monitoring-usage/org_dashboard

View user status, activity, and usage trends in the W&B organization dashboard across deployment types.

This page describes how to view activity in your W\&B organization, including user status, activity trends over time, and how to export user details as a CSV. Organization admins use this information to audit access, monitor adoption, and report on usage. Select your deployment type in each section to continue.

## View user status and activity

The following procedure shows how to open the dashboard that lists every user in your organization and interpret each user's status.

<Tabs>
  <Tab title="Dedicated Cloud / Self-Managed">
    1. Navigate to the **Organization Dashboard**:

       * **Dedicated Cloud**: `https://[ORG-NAME].io/org/dashboard/`. Replace `[ORG-NAME]` with your organization name.
       * **Self-Managed**: `https://[YOUR-W&B-SERVER-IP]/org/dashboard`. Replace `[YOUR-W&B-SERVER-IP]` with your deployment's IP address.

       The **Users** tab opens by default and lists every user in the organization.
    2. To sort the list by user status, click the **Last Active** column header. Each user's status is one of the following:
       * **Invite pending**: An invitation was sent but not yet accepted.
       * **A timestamp**: The user accepted the invitation and has signed in at least once. The timestamp indicates the most recent activity.
       * **Deactivated**: An admin revoked the user's access.
       * **No status (hyphen)**: The user was previously active but hasn't been active in the last six months.
    3. Hover over a user's **Last Active** field to see the date you added the user and their total active days.

    A user is *active* if they do any of the following:

    * Log in to W\&B.
    * Open any page in the W\&B App.
    * Log runs.
    * Use the SDK to track an experiment.
    * Interact with the W\&B server in any way.
  </Tab>

  <Tab title="Multi-tenant Cloud">
    1. Open the [**Members** page](https://wandb.ai/account-settings/wandb/members/). The table lists every user in your organization.
    2. Click the **Last Active** column header to sort by user status. Each user's status is one of the following:
       * **A timestamp**: The user has signed in at least once. The timestamp indicates the most recent activity.
       * **No status (hyphen)**: The user hasn't yet been active within the organization.

    A user is *active* if they perform any auditable action scoped to the organization after May 8, 2025. For a full list, see [Actions](/platform/hosting/monitoring-usage/audit-logging#actions) in **Audit logging**.
  </Tab>
</Tabs>

## View activity over time

After you understand individual user status, use the activity views to track how usage in your organization changes over time.

<Tabs>
  <Tab title="Dedicated Cloud / Self-Managed">
    Use the **Activity** tab to see how many users have been active during a given period.

    1. Open the **Organization Dashboard**. See [View user status and activity](#view-user-status-and-activity).
    2. Click **Activity**.
    3. Review the following plots:
       * **Total active users**: Unique active users during the selected period (defaults to 3 months).
       * **Users active over time**: Fluctuation of active users over the period (defaults to 6 months). Hover over a point to see the exact count on that date.

    To change the reporting period, use the drop-down above a plot. Options are **Last 30 days**, **Last 3 months**, **Last 6 months**, **Last 12 months**, and **All time**.
  </Tab>

  <Tab title="Multi-tenant Cloud">
    Use the **Activity Dashboard** to view aggregate activity.

    1. Click your user icon in the upper-right corner of the W\&B App.
    2. Under **Account**, click **Users**.
    3. Above the table of users, review the Activity Panel:
       * **Active user count**: Unique active users during the selected period (defaults to 3 months).
       * **Weekly active users**: Users active per week.
       * **Most active user**: Top-10 users ranked by active days and last-active date.
    4. To change the date range, click the date picker in the upper-right corner and choose a new value: **7 days**, **30 days** (the default), **90 days**, **6 months**, or **12 months**. All plots update automatically.
  </Tab>
</Tabs>

## Export user details

When you need to share user details outside the dashboard or analyze them in another tool, export the user list as a CSV. From the **Users** tab you can download a CSV that lists each user's details (username, email address, last-active time, roles, and more).

<Tabs>
  <Tab title="Dedicated Cloud / Self-Managed">
    1. In the **Users** tab, click the **action (<Icon icon="ellipsis" />)** menu next to **Invite new user**.
    2. Click **Export as CSV**.

    ### Export CSV schema

    The CSV export uses the comma (`,`) as the separator, encloses strings in double quotes, and includes a header row that defines these columns:

    * `"Name"`
    * `"Username"`
    * `"Org Role"`
    * `"Models Seat"`
    * `"Weave Access"`
    * `"Email"`
    * `"Teams"`
    * `"Last Active"`
  </Tab>

  <Tab title="Multi-tenant Cloud">
    1. In the **Users** tab, click the **action (<Icon icon="ellipsis" />)** menu in the upper-right corner.
    2. Select **Export as CSV** to download the file.

    ### Export CSV schema

    The CSV export uses the comma (`,`) as the separator, encloses strings in double quotes, and includes a header row that defines these columns:

    * `"Name"`
    * `"Username"`
    * `"Last Active"`
    * `"Role"`
    * `"Email"`
    * `"Teams"`
    * `"Status"`
    * `"Number of Reports"`
    * `"Number of Runs"`
    * `"Number of active days"`
    * `"Models Seat"`
    * `"Weave Access"`
  </Tab>
</Tabs>


# Configure Slack alerts
Source: https://docs.wandb.ai/platform/hosting/monitoring-usage/slack-alerts

Create and configure a Slack application to receive W&B Server alerts, notifications, and monitoring updates.

Integrate W\&B Server with [Slack](https://slack.com/) so that your W\&B instance can dispatch alerts and notifications to a Slack workspace your team already uses. This page walks W\&B Server administrators through creating a Slack application, configuring its OAuth scopes and redirect URL, and registering the application with W\&B.

<Note>
  Watch a [video demonstrating setting up Slack alerts on W\&B Dedicated Cloud deployment](https://www.youtube.com/watch?v=JmvKb-7u-oU) (6 min).
</Note>

## Create the Slack application

W\&B Server uses a custom Slack application as the bridge for delivering alerts. Follow this procedure to create that application in the Slack workspace where you want to receive notifications.

1. Visit [https://api.slack.com/apps](https://api.slack.com/apps) and select **Create an App**.

   <Frame>
     <img alt="Create an App button" />
   </Frame>

2. Provide a name for your app in the **App Name** field.

3. Select a Slack workspace where you want to develop your app. Ensure that the Slack workspace you use is the same workspace you intend to use for alerts.

   <Frame>
     <img alt="App name and workspace selection" />
   </Frame>

## Configure the Slack application

After you create the Slack application, grant it the permissions to post messages and accept the redirect from W\&B during the OAuth handshake.

1. On the left sidebar, select **OAuth & Permissions**.

   <Frame>
     <img alt="OAuth & Permissions menu" />
   </Frame>

2. In the **Scopes** section, grant the bot the `incoming_webhook` scope. Scopes give your app permission to perform actions in your development workspace.

   For more information about OAuth scopes for bots, see [Understanding OAuth scopes for bots](https://api.slack.com/legacy/oauth-scopes) in the Slack API documentation.

   <Frame>
     <img alt="Bot token scopes" />
   </Frame>

3. Configure the **Redirect URL** to point to your W\&B installation. Use the same URL that you set as your host URL in your local system settings. You can specify multiple URLs if you have different DNS mappings to your instance.

   <Frame>
     <img alt="Redirect URLs configuration" />
   </Frame>

4. Select **Save URLs**.

5. Optional: Under **Restrict API Token Usage**, specify an IP or IP range to allowlist for your W\&B instances. Limiting the allowed IP address helps secure your Slack application.

## Register your Slack application with W\&B

Register the Slack application you configured with your W\&B instance so that W\&B can use it to dispatch alerts.

1. Navigate to the **System Settings** or **System Console** page of your W\&B instance, depending on your deployment.

2. Depending on the System page you're on, follow one of the following options:

   * If you're in the **System Console**: go to **Settings** then to **Notifications**.

     <Frame>
       <img alt="System Console notifications" />
     </Frame>

   * If you're in the **System Settings**: toggle the **Enable a custom Slack application to dispatch alerts** to enable a custom Slack application.

     <Frame>
       <img alt="Enable Slack application toggle" />
     </Frame>

3. Supply your **Slack client ID** and **Slack secret**, then select **Save**. To find your application's client ID and secret, navigate to **Basic Information** in **Settings**.

4. To verify that everything works, set up a Slack integration in the W\&B app.

Your W\&B Server is now registered with the Slack application and can dispatch alerts to the configured Slack workspace.


# Disable automatic updates for W&B Server
Source: https://docs.wandb.ai/platform/hosting/self-managed/disable-automatic-app-version-updates

Learn how to disable automatic updates for W&B Server.

This page shows administrators of Self-Managed W\&B deployments how to disable automatic version upgrades for W\&B Server and pin its version to a specific release. Pinning a version gives you control over when upgrades happen. This control is useful when you need to coordinate upgrades with internal change management processes or validate a release in a staging environment before you roll it out. These instructions work only for deployments managed by the [W\&B Kubernetes Operator](/platform/hosting/self-managed/operator).

<Note>
  W\&B supports a major W\&B Server release for 12 months from its initial release date. Customers with **Self-Managed** instances are responsible for upgrading in time to maintain support. Avoid staying on an unsupported version. W\&B recommends that customers with **Self-Managed** instances update their deployments with the latest release at least once per quarter to maintain support and to receive the latest features, performance improvements, and fixes.
</Note>

## Requirements

* W\&B Kubernetes Operator `v1.13.0` or newer
* System Console `v2.12.2` or newer

To verify that you meet these requirements, refer to the W\&B Custom Resource or Helm chart for your instance. Check the `version` values for the `operator-wandb` and `system-console` components.

## Disable automatic updates

To disable automatic updates, use the System Console to pin W\&B Server to a specific version, then check the Operator reconciliation logs to confirm that version pinning is active.

1. Log in to the W\&B App as a user with the `admin` role.
2. Click the user icon at the top, then click **System Console**.
3. Go to **Settings** > **Advanced**, then select the **Other** tab.
4. In the **Disable Auto Upgrades** section, turn on **Pin specific version**.
5. Click the **Select a version** drop-down list, then select a W\&B Server version.
6. Click **Save**.

   <Frame>
     <img alt="System Console showing Pin specific version turned on with a selected W&B Server version" />
   </Frame>

   Automatic upgrades are turned off and W\&B Server is pinned at the version you selected.
7. Verify that automatic upgrades are turned off. Go to the **Operator** tab and search the reconciliation logs for the string `Version pinning is enabled`.

```text theme={null}
│info 2025-04-17T17:24:16Z wandb default No changes found
│info 2025-04-17T17:24:16Z wandb default Active spec found
│info 2025-04-17T17:24:16Z wandb default Desired spec
│info 2025-04-17T17:24:16Z wandb default License
│info 2025-04-17T17:24:16Z wandb default Version Pinning is enabled
│info 2025-04-17T17:24:16Z wandb default Found Weights & Biases instance, processing the spec...
│info 2025-04-17T17:24:16Z wandb default === Reconciling Weights & Biases instance...
```


# Deploy on Air-Gapped Kubernetes
Source: https://docs.wandb.ai/platform/hosting/self-managed/on-premises-deployments/kubernetes-airgapped

Deploy W&B Platform in air-gapped and disconnected Kubernetes environments

## Introduction

This guide provides step-by-step instructions to deploy the W\&B Platform in air-gapped, disconnected, or restricted network customer-managed environments. By following this guide, you set up an internal container registry and Helm repository to host W\&B images and charts, install the W\&B Kubernetes Operator, and deploy the W\&B Platform without requiring outbound internet connectivity. This guide targets platform administrators and DevOps engineers who manage Kubernetes infrastructure in regulated or isolated networks.

Air-gapped deployments are common in the following environments:

* Secure government facilities.
* Financial institutions with strict network isolation.
* Healthcare organizations with compliance requirements.
* Industrial control systems (ICS) environments.
* Research facilities with classified networks.

Run these commands in a shell console with proper access to the Kubernetes cluster. You can adapt these commands to work with any CI/CD tooling you use to deploy Kubernetes applications.

For standard on-premises Kubernetes deployments with internet connectivity, see [Deploy W\&B with Kubernetes Operator](/platform/hosting/self-managed/operator).

## Prerequisites

Before starting, ensure your air-gapped environment meets the following requirements.

### Version requirements

| Software   | Minimum version                                                                                                                 |
| ---------- | ------------------------------------------------------------------------------------------------------------------------------- |
| Kubernetes | v1.34 or newer ([Supported Kubernetes versions](https://kubernetes.io/releases/patch-releases/))                                |
| Helm       | v3.x                                                                                                                            |
| MySQL      | v8.0.x is required, v8.0.32 or newer; v8.0.44 or newer is recommended.<br />Aurora MySQL 3.x releases, must be v3.05.2 or newer |
| Redis      | v7.x                                                                                                                            |

### SSL/TLS requirements

W\&B requires a valid signed SSL/TLS certificate for secure communication between clients and the server. SSL/TLS termination must occur on the ingress/load balancer. The W\&B Server application does not terminate SSL or TLS connections.

**Important**: W\&B does not support self-signed certificates and custom CAs. Using self-signed certificates will cause challenges for users and is not supported.

If possible, using a service like [Let's Encrypt](https://letsencrypt.org) is a great way to provide trusted certificates to your load balancer. Services like Caddy and Cloudflare manage SSL for you.

If your security policies require SSL communication within your trusted networks, consider using a tool like Istio and [side car containers](https://istio.io/latest/docs/reference/config/networking/sidecar/).

### Hardware requirements

**CPU Architecture**: W\&B runs on Intel (x86) CPU architecture only. ARM is not supported.

**Sizing**: For CPU, memory, and disk sizing recommendations for Kubernetes nodes and MySQL, see the [Sizing section](/platform/hosting/self-managed/ref-arch/#sizing) in the reference architecture. Requirements vary based on whether you're running Models, Weave, or both.

### MySQL database

W\&B requires an external MySQL database.

For production, W\&B strongly recommends using managed database services:

* [AWS RDS Aurora MySQL](https://aws.amazon.com/rds/aurora/)
* [Google Cloud SQL for MySQL](https://cloud.google.com/sql/mysql)
* [Azure Database for MySQL](https://azure.microsoft.com/en-us/products/mysql/)

Managed database services provide automated backups, monitoring, high availability, patching, and reduce operational overhead.

See the [reference architecture](/platform/hosting/self-managed/ref-arch/#mysql) for complete MySQL requirements, including sizing recommendations and configuration parameters. For database creation SQL, see the [bare-metal guide](/platform/hosting/self-managed/operator/#mysql-database). For questions about your deployment's database configuration, contact [support](mailto:support@wandb.com) or your AISE.

For MySQL configuration parameters for self-managed instances, see the [reference architecture MySQL configuration section](/platform/hosting/self-managed/ref-arch#mysql-configuration-parameters).

### Redis

W\&B depends on a single-node Redis 7.x deployment used by W\&B's components for job queuing and data caching. For convenience during testing and development of proofs of concept, W\&B Self-Managed includes a local Redis deployment that is not appropriate for production deployments.

For production deployments, W\&B can connect to a Redis instance in the following environments:

* [AWS Elasticache](https://aws.amazon.com/elasticache/)
* [Google Cloud Memory Store](https://cloud.google.com/memorystore?hl=en)
* [Azure Cache for Redis](https://azure.microsoft.com/en-us/products/cache)
* Redis deployment hosted in your cloud or on-premise infrastructure

### Object storage

W\&B requires object storage with pre-signed URL and CORS support.

**Recommended storage providers:**

* [Amazon S3](https://aws.amazon.com/s3/): Object storage service offering industry-leading scalability, data availability, security, and performance.
* [Google Cloud Storage](https://cloud.google.com/storage): Managed service for storing unstructured data at scale.
* [Azure Blob Storage](https://azure.microsoft.com/en-us/products/storage/blobs): Cloud-based object storage solution for storing massive amounts of unstructured data.
* [CoreWeave AI Object Storage](https://docs.coreweave.com/products/storage/object-storage): High-performance, S3-compatible object storage service optimized for AI workloads.
* Enterprise S3-compatible storage: [MinIO Enterprise (AIStor)](https://www.min.io/product/aistor), [NetApp StorageGRID](https://www.netapp.com/data-storage/storagegrid/), or other enterprise-grade solutions

<Note>
  MinIO Open Source is in [maintenance mode](https://github.com/minio/minio) with no active development or pre-compiled binaries. For production deployments, W\&B recommends using managed object storage services or enterprise S3-compatible solutions such as MinIO Enterprise (AIStor).
</Note>

For detailed bucket provisioning instructions including IAM policies, CORS configuration, and access setup, see the [Bring Your Own Bucket (BYOB) guide](/platform/hosting/data-security/secure-storage-connector).

See the [reference architecture object storage section](/platform/hosting/self-managed/ref-arch/#object-storage) for complete requirements.

For detailed object storage provisioning guidance, see the [Bring Your Own Bucket (BYOB)](/platform/hosting/data-security/secure-storage-connector) guide. In air-gapped environments, you'll typically use on-premises S3-compatible storage such as MinIO Enterprise, NetApp StorageGRID, or Dell ECS.

### Air-gapped specific requirements

In addition to the preceding standard requirements, air-gapped deployments require the following:

* **Internal container registry**: Access to a private container registry such as Harbor, JFrog Artifactory, or Nexus, with all required W\&B images.
* **Internal Helm repository**: Access to a private Helm chart repository with W\&B Helm charts.
* **Image transfer capability**: A method to transfer container images from an internet-connected system to your air-gapped registry.
* **License file**: A valid W\&B Enterprise license. To obtain a license (for example, from an internet-connected machine), see the [License](/platform/hosting/self-managed/requirements#license) section on the Requirements page, or contact your W\&B account team.

For complete infrastructure requirements, including networking and load balancer configuration, see the [reference architecture](/platform/hosting/self-managed/ref-arch#infrastructure-requirements).

## Prepare your air-gapped environment

The following steps prepare your air-gapped environment to host the W\&B container images and Helm charts. Complete these steps before installing the operator or deploying the platform.

### Step 1: Set up internal container registry

Because the Kubernetes cluster cannot pull images from public registries, all required container images must be available in your internal air-gapped container registry before deployment.

<Note>
  You are responsible for tracking the W\&B Operator's requirements and maintaining your container registry with updated images regularly. For the most current list of required container images and versions, refer to the Helm chart, or contact [W\&B Support](mailto:support@wandb.com) or your assigned W\&B support engineer.
</Note>

#### Core W\&B component containers

The following core images are required:

* [`docker.io/wandb/controller`](https://hub.docker.com/r/wandb/controller): W\&B Kubernetes Operator.
* [`docker.io/wandb/local`](https://hub.docker.com/r/wandb/local): W\&B application server.
* [`docker.io/wandb/console`](https://hub.docker.com/r/wandb/console): W\&B management console.
* [`docker.io/wandb/megabinary`](https://hub.docker.com/r/wandb/megabinary): W\&B microservices (API, executor, glue, parquet).

#### Dependency containers

The following third-party dependency images are required:

* [`docker.io/bitnamilegacy/redis`](https://hub.docker.com/r/bitnamilegacy/redis): Required for local Redis deployment during testing and development. For production Redis requirements, see the [Redis section](#redis) in Prerequisites.
* [`docker.io/otel/opentelemetry-collector-contrib`](https://hub.docker.com/r/otel/opentelemetry-collector-contrib): OpenTelemetry agent for collecting metrics and logs.
* [`quay.io/prometheus/prometheus`](https://quay.io/repository/prometheus/prometheus): Prometheus for metrics collection.
* [`quay.io/prometheus-operator/prometheus-config-reloader`](https://quay.io/repository/prometheus-operator/prometheus-config-reloader): Prometheus dependency.

#### Get the complete image list

To extract the complete list of required images and versions from the Helm chart:

1. On an internet-connected system, download the W\&B Helm charts from the [W\&B Helm charts repository](https://github.com/wandb/helm-charts):

   ```bash theme={null}
   # Clone the helm-charts repository
   git clone https://github.com/wandb/helm-charts.git
   cd helm-charts
   ```

2. Inspect the `values.yaml` files to identify all container images and their versions:

   ```bash theme={null}
   # Extract image references from the operator chart
   helm show values charts/operator | grep -E "repository:|tag:" | grep -v "^#"

   # Extract image references from the platform chart
   helm show values charts/operator-wandb | grep -E "repository:|tag:" | grep -v "^#"
   ```

   Alternatively, use this command to extract only the repository names (without version tags):

   ```bash theme={null}
   helm show values charts/operator-wandb \
     | awk -F': *' '/^[[:space:]]*repository:/{print $2}' \
     | grep -v "^#" \
     | sort -u
   ```

   The list of repositories looks similar to the following:

   ```text theme={null}
   wandb/controller
   wandb/local
   wandb/console
   wandb/megabinary
   wandb/weave-python
   wandb/weave-trace
   otel/opentelemetry-collector-contrib
   prometheus/prometheus
   prometheus-operator/prometheus-config-reloader
   bitnamilegacy/redis
   ```

   To get the specific version tags for each image, use the preceding first command (`grep -E "repository:|tag:"`), which shows both repository names and their corresponding version tags.

#### Transfer images to air-gapped registry

1. On an internet-connected system, pull and save all required images.

   <Note>
     Replace version numbers in the following examples with the actual versions from your Helm chart inspection in the preceding step. The versions shown here are examples and become outdated over time.
   </Note>

   Use shell variables to manage versions consistently:

   ```bash theme={null}
   # Set version variables (update these based on your Helm chart versions)
   CONTROLLER_VERSION="1.13.3"
   APP_VERSION="0.59.2"
   CONSOLE_VERSION="2.12.2"

   # Pull images
   docker pull wandb/controller:${CONTROLLER_VERSION}
   docker pull wandb/local:${APP_VERSION}
   docker pull wandb/console:${CONSOLE_VERSION}
   docker pull wandb/megabinary:${APP_VERSION}
   # ... pull all other required images with their versions

   # Save images to .tar files
   docker save wandb/controller:${CONTROLLER_VERSION} -o wandb-controller-${CONTROLLER_VERSION}.tar
   docker save wandb/local:${APP_VERSION} -o wandb-local-${APP_VERSION}.tar
   docker save wandb/console:${CONSOLE_VERSION} -o wandb-console-${CONSOLE_VERSION}.tar
   docker save wandb/megabinary:${APP_VERSION} -o wandb-megabinary-${APP_VERSION}.tar
   # ... save all other images
   ```

2. Transfer the `.tar` files to your air-gapped environment using your approved method, such as a USB drive or secure file transfer.

3. In your air-gapped environment, load and push images to your internal registry:

   ```bash theme={null}
   # Set the same version variables used above
   CONTROLLER_VERSION="1.13.3"
   APP_VERSION="0.59.2"
   CONSOLE_VERSION="2.12.2"
   INTERNAL_REGISTRY="registry.yourdomain.com"

   # Load images
   docker load -i wandb-controller-${CONTROLLER_VERSION}.tar
   docker load -i wandb-local-${APP_VERSION}.tar
   docker load -i wandb-console-${CONSOLE_VERSION}.tar
   docker load -i wandb-megabinary-${APP_VERSION}.tar
   # ... load all other images

   # Tag for internal registry
   docker tag wandb/controller:${CONTROLLER_VERSION} ${INTERNAL_REGISTRY}/wandb/controller:${CONTROLLER_VERSION}
   docker tag wandb/local:${APP_VERSION} ${INTERNAL_REGISTRY}/wandb/local:${APP_VERSION}
   docker tag wandb/console:${CONSOLE_VERSION} ${INTERNAL_REGISTRY}/wandb/console:${CONSOLE_VERSION}
   docker tag wandb/megabinary:${APP_VERSION} ${INTERNAL_REGISTRY}/wandb/megabinary:${APP_VERSION}
   # ... tag all other images

   # Push to internal registry
   docker push ${INTERNAL_REGISTRY}/wandb/controller:${CONTROLLER_VERSION}
   docker push ${INTERNAL_REGISTRY}/wandb/local:${APP_VERSION}
   docker push ${INTERNAL_REGISTRY}/wandb/console:${CONSOLE_VERSION}
   docker push ${INTERNAL_REGISTRY}/wandb/megabinary:${APP_VERSION}
   # ... push all other images
   ```

### Step 2: Set up internal Helm chart repository

With the container images in place, the Kubernetes Operator also needs access to the W\&B Helm charts. Ensure the following Helm charts are available in your internal Helm repository:

* [W\&B Operator chart](https://github.com/wandb/helm-charts/tree/main/charts/operator)
* [W\&B Platform chart](https://github.com/wandb/helm-charts/tree/main/charts/operator-wandb)

1. On an internet-connected system, download the charts:

   ```bash theme={null}
   # Add W&B Helm repository
   helm repo add wandb https://wandb.github.io/helm-charts
   helm repo update

   # Download the charts
   helm pull wandb/operator --version 1.13.3
   helm pull wandb/operator-wandb --version 0.18.0
   ```

2. Transfer the `.tgz` chart files to your air-gapped environment and upload them to your internal Helm repository according to your repository's procedures.

   The `operator` chart deploys the W\&B Kubernetes Operator (Controller Manager). The `operator-wandb` chart deploys the W\&B Platform using the values configured in the Custom Resource (CR).

### Step 3: Configure Helm repository access

Configure your local Helm client in the air-gapped environment to point to your internal repository so that subsequent install commands can locate the charts.

1. In your air-gapped environment, configure Helm to use your internal repository:

   ```bash theme={null}
   helm repo add local-repo https://charts.yourdomain.com
   helm repo update
   ```

2. Verify the charts are available:

   ```bash theme={null}
   helm search repo local-repo/operator
   helm search repo local-repo/operator-wandb
   ```

## Deploy W\&B in air-gapped environment

With your internal registry and Helm repository in place, you can now install the Kubernetes Operator, configure external services, and deploy the W\&B Platform.

### Step 4: Install the Kubernetes Operator

The W\&B Kubernetes Operator (controller manager) manages the W\&B Platform components. To install it in an air-gapped environment, configure it to use your internal container registry.

1. Create a `values.yaml` file with the following content:

   ```yaml theme={null}
   image:
     repository: registry.yourdomain.com/wandb/controller
     tag: 1.13.3

   airgapped: true
   ```

   <Note>
     Replace the repository and tag with the actual versions you transferred to your internal registry in Step 1. The version shown here (`1.13.3`) is an example and becomes outdated over time.
   </Note>

2. Install the operator and Custom Resource Definition (CRD):

   ```bash theme={null}
   helm upgrade --install operator local-repo/operator \
     --namespace wandb \
     --create-namespace \
     --values values.yaml
   ```

3. Verify the operator is running:

   ```bash theme={null}
   kubectl get pods -n wandb
   ```

   You should see the operator pod in a `Running` state.

The W\&B Kubernetes Operator is now installed and ready to deploy the W\&B Platform from your internal chart repository.

For full details about supported values, refer to the [Kubernetes operator GitHub repository values file](https://github.com/wandb/helm-charts/blob/main/charts/operator/values.yaml).

### Step 5: Set up MySQL database

Before configuring the W\&B Custom Resource, set up an external MySQL database. For production deployments, W\&B strongly recommends using managed database services where available. However, if you're running your own MySQL instance, create the database and user:

Create a database and a user with the following SQL commands. Replace `SOME_PASSWORD` with a secure password of your choice:

```sql theme={null}
CREATE USER 'wandb_local'@'%' IDENTIFIED BY 'SOME_PASSWORD';
CREATE DATABASE wandb_local CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;
GRANT ALL ON wandb_local.* TO 'wandb_local'@'%' WITH GRANT OPTION;
```

For MySQL configuration parameters, see the [reference architecture MySQL configuration section](/platform/hosting/self-managed/ref-arch#mysql-configuration-parameters).

### Step 6: Configure W\&B Custom Resource

After installing the W\&B Kubernetes Operator, configure the Custom Resource (CR) to point to your internal Helm repository and container registry. This configuration ensures the Kubernetes operator uses your internal registry and repository when deploying the required components of the W\&B Platform, instead of attempting to reach public sources.

<Note>
  The following example configuration includes image version tags that become outdated over time. Replace all `tag:` values with the actual versions you transferred to your internal registry in Step 1.
</Note>

Create a file named `wandb.yaml` with the following content:

```yaml theme={null}
apiVersion: apps.wandb.com/v1
kind: WeightsAndBiases
metadata:
  labels:
    app.kubernetes.io/instance: wandb
    app.kubernetes.io/name: weightsandbiases
  name: wandb
  namespace: wandb

spec:
  chart:
    url: https://charts.yourdomain.com
    name: operator-wandb
    version: 0.18.0

  values:
    global:
      host: https://wandb.yourdomain.com
      license: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
      
      bucket:
        accessKey: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
        secretKey: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
        name: s3.yourdomain.com:9000
        path: wandb
        provider: s3
        region: us-east-1
      
      mysql:
        database: wandb
        host: mysql.yourdomain.com
        password: [YOUR-MYSQL-PASSWORD]
        port: 3306
        user: wandb
      
      redis:
        host: redis.yourdomain.com
        port: 6379
        password: [YOUR-REDIS-PASSWORD]
      
      api:
        enabled: true
      
      glue:
        enabled: true
      
      executor:
        enabled: true
      
      extraEnv:
        ENABLE_REGISTRY_UI: 'true'

    # Configure all component images to use internal registry
    app:
      image:
        repository: registry.yourdomain.com/wandb/local
        tag: 0.59.2

    console:
      image:
        repository: registry.yourdomain.com/wandb/console
        tag: 2.12.2

    api:
      image:
        repository: registry.yourdomain.com/wandb/megabinary
        tag: 0.59.2

    executor:
      image:
        repository: registry.yourdomain.com/wandb/megabinary
        tag: 0.59.2

    glue:
      image:
        repository: registry.yourdomain.com/wandb/megabinary
        tag: 0.59.2

    parquet:
      image:
        repository: registry.yourdomain.com/wandb/megabinary
        tag: 0.59.2

    weave:
      image:
        repository: registry.yourdomain.com/wandb/weave-python
        tag: 0.59.2

    otel:
      image:
        repository: registry.yourdomain.com/otel/opentelemetry-collector-contrib
        tag: 0.97.0

    prometheus:
      server:
        image:
          repository: registry.yourdomain.com/prometheus/prometheus
          tag: v2.47.0
      configmapReload:
        prometheus:
          image:
            repository: registry.yourdomain.com/prometheus-operator/prometheus-config-reloader
            tag: v0.67.0

    ingress:
      annotations:
        nginx.ingress.kubernetes.io/proxy-body-size: "0"
      class: nginx
```

<Note>
  Replace all placeholder values, such as hostnames, passwords, and tags, with your actual configuration values. The preceding example shows the most commonly used components.
</Note>

Depending on your deployment needs, you may also need to configure image repositories for additional components such as the following:

* `settingsMigrationJob`
* `weave-trace`
* `filestream`
* `flat-runs-table`

Refer to the [W\&B Helm repository values file](https://github.com/wandb/helm-charts/blob/main/charts/operator-wandb/values.yaml) for the complete list of configurable components.

### Step 7: Deploy the W\&B Platform

Applying the Custom Resource triggers the operator to install the W\&B Platform components defined in the `operator-wandb` chart, using the configuration and image references from `wandb.yaml`.

1. Apply the W\&B Custom Resource to deploy the platform:

   ```bash theme={null}
   kubectl apply -f wandb.yaml
   ```

2. Monitor the deployment progress:

   ```bash theme={null}
   # Watch pods being created
   kubectl get pods -n wandb --watch

   # Check deployment status
   kubectl get weightsandbiases -n wandb

   # View operator logs
   kubectl logs -n wandb deployment/wandb-operator-controller-manager
   ```

   The deployment may take several minutes as the operator creates all necessary components.

## OpenShift configuration

W\&B supports deployment on air-gapped OpenShift Kubernetes clusters. OpenShift deployments require additional security context configurations because of OpenShift's stricter security policies. If you're deploying on OpenShift, apply the configurations in this section in addition to the preceding steps.

### OpenShift security context constraints

OpenShift uses Security Context Constraints (SCCs) to control pod permissions. By default, OpenShift assigns the `restricted` SCC to pods, which prevents running as root and requires specific user IDs.

#### Option 1: Use restricted SCC (recommended)

Configure W\&B components to run with the restricted SCC by setting appropriate security contexts in your Custom Resource:

```yaml theme={null}
spec:
  values:
    # Configure security contexts for all pods
    app:
      podSecurityContext:
        fsGroup: 1000
        runAsUser: 1000
        runAsNonRoot: true
      securityContext:
        allowPrivilegeEscalation: false
        capabilities:
          drop:
            - ALL
        runAsNonRoot: true
        seccompProfile:
          type: RuntimeDefault

    console:
      podSecurityContext:
        fsGroup: 1000
        runAsUser: 1000
        runAsNonRoot: true
      securityContext:
        allowPrivilegeEscalation: false
        capabilities:
          drop:
            - ALL
        runAsNonRoot: true
        seccompProfile:
          type: RuntimeDefault

    # Repeat for other components: api, executor, glue, parquet, weave
```

#### Option 2: Create custom SCC (if required)

If your deployment requires capabilities not available in the `restricted` SCC, create a custom SCC:

```yaml theme={null}
apiVersion: security.openshift.io/v1
kind: SecurityContextConstraints
metadata:
  name: wandb-scc
allowHostDirVolumePlugin: false
allowHostIPC: false
allowHostNetwork: false
allowHostPID: false
allowHostPorts: false
allowPrivilegeEscalation: false
allowPrivilegedContainer: false
allowedCapabilities: []
defaultAddCapabilities: []
fsGroup:
  type: MustRunAs
  ranges:
    - min: 1000
      max: 65535
readOnlyRootFilesystem: false
requiredDropCapabilities:
  - ALL
runAsUser:
  type: MustRunAsRange
  uidRangeMin: 1000
  uidRangeMax: 65535
seLinuxContext:
  type: MustRunAs
supplementalGroups:
  type: RunAsAny
volumes:
  - configMap
  - downwardAPI
  - emptyDir
  - persistentVolumeClaim
  - projected
  - secret
```

1. Apply the SCC:

   ```bash theme={null}
   oc apply -f wandb-scc.yaml
   ```

2. Bind the SCC to the W\&B service accounts:

   ```bash theme={null}
   oc adm policy add-scc-to-user wandb-scc -z wandb-app -n wandb
   oc adm policy add-scc-to-user wandb-scc -z wandb-console -n wandb
   ```

### OpenShift routes

OpenShift uses Routes instead of standard Kubernetes Ingress. Configure W\&B to use OpenShift Routes:

```yaml theme={null}
spec:
  values:
    ingress:
      enabled: false
    
    route:
      enabled: true
      host: wandb.apps.openshift.yourdomain.com
      tls:
        enabled: true
        termination: edge
        insecureEdgeTerminationPolicy: Redirect
```

### OpenShift image pull configuration

If your OpenShift cluster uses an internal image registry with authentication:

1. Create an image pull secret:

   ```bash theme={null}
   kubectl create secret docker-registry wandb-registry-secret \
     --docker-server=registry.yourdomain.com \
     --docker-username=[USERNAME] \
     --docker-password=[PASSWORD] \
     --namespace=wandb
   ```

2. Reference the secret in your Custom Resource:

   ```yaml theme={null}
   spec:
     values:
       imagePullSecrets:
         - name: wandb-registry-secret
   ```

### OpenShift complete example

The following example shows a complete CR for OpenShift air-gapped deployment:

<Note>
  Replace all `tag:` values in this example with the actual versions you transferred to your internal registry in Step 1. The versions shown are examples and become outdated over time.
</Note>

```yaml theme={null}
apiVersion: apps.wandb.com/v1
kind: WeightsAndBiases
metadata:
  name: wandb
  namespace: wandb

spec:
  chart:
    url: https://charts.yourdomain.com
    name: operator-wandb
    version: 0.18.0

  values:
    global:
      host: https://wandb.apps.openshift.yourdomain.com
      license: [YOUR-LICENSE]
      
      bucket:
        accessKey: [YOUR-ACCESS-KEY]
        secretKey: [YOUR-SECRET-KEY]
        name: s3.yourdomain.com:9000
        path: wandb
        provider: s3
        region: us-east-1
      
      mysql:
        database: wandb
        host: mysql.yourdomain.com
        password: [YOUR-MYSQL-PASSWORD]
        port: 3306
        user: wandb
      
      redis:
        host: redis.yourdomain.com
        port: 6379
        password: [YOUR-REDIS-PASSWORD]

    # OpenShift-specific: Use Routes instead of Ingress
    ingress:
      enabled: false
    
    route:
      enabled: true
      host: wandb.apps.openshift.yourdomain.com
      tls:
        enabled: true
        termination: edge

    # Image pull secret for internal registry
    imagePullSecrets:
      - name: wandb-registry-secret

    # Security contexts for OpenShift restricted SCC
    app:
      image:
        repository: registry.yourdomain.com/wandb/local
        tag: 0.59.2
      podSecurityContext:
        fsGroup: 1000
        runAsUser: 1000
        runAsNonRoot: true
      securityContext:
        allowPrivilegeEscalation: false
        capabilities:
          drop:
            - ALL
        runAsNonRoot: true
        seccompProfile:
          type: RuntimeDefault

    console:
      image:
        repository: registry.yourdomain.com/wandb/console
        tag: 2.12.2
      podSecurityContext:
        fsGroup: 1000
        runAsUser: 1000
        runAsNonRoot: true
      securityContext:
        allowPrivilegeEscalation: false
        capabilities:
          drop:
            - ALL
        runAsNonRoot: true
        seccompProfile:
          type: RuntimeDefault

    # Repeat security contexts for: api, executor, glue, parquet, weave
    # (abbreviated for clarity)
```

<Note>
  Contact [W\&B Support](mailto:support@wandb.com) or your assigned W\&B support engineer for comprehensive OpenShift configuration examples tailored to your security requirements.
</Note>

## Verify your installation

After deploying W\&B, verify the installation is working correctly so that you can confirm the platform is reachable, pods are healthy, and the deployment is using only your internal resources.

Follow the general verification steps, then complete the additional air-gapped checks in the following section.

To verify the installation, W\&B recommends using the [W\&B CLI](/models/ref/cli/). The verify command executes several tests that verify all components and configurations.

<Note>
  This step assumes that the first admin user account is created with the browser.
</Note>

Follow these steps to verify the installation:

1. Install the W\&B CLI:

```bash theme={null}
pip install wandb
```

2. Log in to W\&B:

```bash theme={null}
wandb login --host=https://YOUR_DNS_DOMAIN
```

For example:

```bash theme={null}
wandb login --host=https://wandb.company-name.com
```

3. Verify the installation:

```bash theme={null}
wandb verify
```

A successful installation and fully working W\&B deployment shows the following output:

```console theme={null}
Default host selected:  https://wandb.company-name.com
Find detailed logs for this test at: /var/folders/pn/b3g3gnc11_sbsykqkm3tx5rh0000gp/T/tmpdtdjbxua/wandb
Checking if logged in...................................................✅
Checking signed URL upload..............................................✅
Checking ability to send large payloads through proxy...................✅
Checking requests to base url...........................................✅
Checking requests made over signed URLs.................................✅
Checking CORs configuration of the bucket...............................✅
Checking wandb package version is up to date............................✅
Checking logged metrics, saving and downloading a file..................✅
Checking artifact save and download workflows...........................✅
```

Contact W\&B Support if you encounter errors.

### Additional air-gapped verification

For air-gapped deployments, also verify:

1. **Image pull**: Confirm all pods successfully pulled images from your internal registry:

   ```bash theme={null}
   kubectl get pods -n wandb -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.phase}{"\t"}{.status.containerStatuses[*].image}{"\n"}{end}'
   ```

   All images should point to your internal registry and all pods should be in `Running` state.

2. **External connectivity**: Verify W\&B is not attempting external connections (it shouldn't in air-gapped mode):

   ```bash theme={null}
   kubectl logs -n wandb deployment/wandb-app --tail=100 | grep -i "connection"
   ```

3. **License validation**: Access the W\&B console and verify your license is active.

## Troubleshooting

### Image pull errors

If pods fail to pull images, check the following:

* Verify images exist in your internal registry.
* Check that the image pull secret is correctly configured.
* Verify network connectivity from Kubernetes nodes to the registry.
* Check registry authentication credentials.

To test an image pull manually:

```bash theme={null}
kubectl run test-pull --image=registry.yourdomain.com/wandb/local:0.59.2 --namespace=wandb
kubectl logs test-pull -n wandb
kubectl delete pod test-pull -n wandb
```

### OpenShift SCC errors

If pods fail with permission errors on OpenShift:

```bash theme={null}
# Check which SCC is being used
oc get pod [POD-NAME] -n wandb -o yaml | grep scc

# Check service account permissions
oc describe scc wandb-scc
oc get rolebinding -n wandb
```

### Helm chart not found

If the operator can't find the platform chart, check the following:

* Verify the chart repository URL in the Custom Resource.
* Check that the operator pod can reach your internal Helm repository.
* Verify the chart exists in your repository:

  ```bash theme={null}
  helm search repo local-repo/operator-wandb
  ```

## Frequently asked questions

### Can I use a different ingress class?

Yes, configure your ingress class by modifying the ingress settings in your Custom Resource:

```yaml theme={null}
spec:
  values:
    ingress:
      class: your-ingress-class
```

### How do I handle certificate bundles with multiple certificates?

Split the certificates into multiple entries in the `customCACerts` section:

```yaml theme={null}
spec:
  values:
    customCACerts:
      cert1.crt: |
        -----BEGIN CERTIFICATE-----
        ...
        -----END CERTIFICATE-----
      cert2.crt: |
        -----BEGIN CERTIFICATE-----
        ...
        -----END CERTIFICATE-----
```

### How do I prevent automatic updates?

To configure the operator to not automatically update W\&B, do the following:

* Set `airgapped: true` in the operator installation (this disables automatic update checks).
* Control version updates by manually updating the `spec.chart.version` in your Custom Resource.
* Optionally, disable automatic updates from the W\&B System Console.

See [Disable automatic app version updates](/platform/hosting/self-managed/disable-automatic-app-version-updates) for more details.

<Note>
  W\&B strongly recommends customers with Self-Managed instances update their deployments with the latest release at minimum once per quarter to maintain support and receive the latest features, performance improvements, and fixes. W\&B supports a major release for 12 months from its initial release date. Refer to [Release policies and processes](/release-notes/release-policies).
</Note>

### Does the deployment work with no connection to public repositories?

Yes. When `airgapped: true` is set in the operator configuration, the Kubernetes operator uses only your internal resources and doesn't attempt to connect to public repositories.

### How do I update W\&B in an air-gapped environment?

To update W\&B:

1. Pull new container images on an internet-connected system.
2. Transfer images to your air-gapped registry.
3. Upload new Helm charts to your internal repository.
4. Update the `spec.chart.version` and image tags in your Custom Resource.
5. Apply the updated Custom Resource.

   The operator performs a rolling update of the W\&B components.

## Next steps

After successful deployment, complete the following tasks:

* **Configure user authentication**: Set up [SSO](/platform/hosting/iam/sso) or other authentication methods.
* **Set up monitoring**: Configure monitoring for your W\&B instance and infrastructure.
* **Plan for updates**: Review the [Server upgrade process](/platform/hosting/server-upgrade-process) and establish an update cadence.
* **Configure backups**: Establish backup procedures for your MySQL database.
* **Document your process**: Create runbooks for your specific air-gapped update procedures.

## Get help

If you encounter issues during deployment:

* Review the [Reference Architecture](/platform/hosting/self-managed/ref-arch) for infrastructure guidance.
* Check the [Operator guide](/platform/hosting/self-managed/operator) for configuration details.
* Contact [W\&B Support](mailto:support@wandb.com) or your assigned W\&B support engineer.
* For OpenShift-specific issues, reference Red Hat OpenShift documentation.


# Deploy W&B with Kubernetes Operator
Source: https://docs.wandb.ai/platform/hosting/self-managed/operator

Deploy W&B Platform with Kubernetes Operator on cloud or on-premises

## Overview

This page shows platform administrators how to deploy and manage W\&B Server on Kubernetes (cloud or on-premises) using the W\&B Kubernetes Operator. By the end, you have a running W\&B Server installation that the operator manages and upgrades automatically. Use this guide if you self-manage a W\&B deployment and need an installation method that works across cloud, on-premises, and air-gapped environments.

The W\&B Kubernetes Operator is the recommended way to deploy W\&B Server on Kubernetes (cloud or on-premises). For an overview of the operator, why W\&B uses it, and how configuration hierarchy works, see [Self-Managed](/platform/hosting/hosting-options/self-managed#about-the-wb-kubernetes-operator).

## Before you begin

Before deploying W\&B with the Kubernetes Operator, ensure your infrastructure meets all requirements:

1. **Review infrastructure requirements**: See the [Self-Managed infrastructure requirements](/platform/hosting/self-managed/requirements/) page for details on:

* Software version requirements (Kubernetes, MySQL, Redis, Helm)
* Hardware requirements (CPU architecture, sizing recommendations)
* Kubernetes cluster configuration
* Networking, SSL/TLS, and DNS requirements

2. **Obtain a W\&B Server license**: See the [License](/platform/hosting/self-managed/requirements#license) section on the Requirements page.
3. **Provision external services**: Set up MySQL, Redis, and object storage before deployment.

For additional context, see the [reference architecture](/platform/hosting/self-managed/ref-arch/) page.

### MySQL database

W\&B requires an external MySQL database.

For production, W\&B strongly recommends using managed database services:

* [AWS RDS Aurora MySQL](https://aws.amazon.com/rds/aurora/)
* [Google Cloud SQL for MySQL](https://cloud.google.com/sql/mysql)
* [Azure Database for MySQL](https://azure.microsoft.com/en-us/products/mysql/)

Managed database services provide automated backups, monitoring, high availability, patching, and reduce operational overhead.

See the [reference architecture](/platform/hosting/self-managed/ref-arch/#mysql) for complete MySQL requirements, including sizing recommendations and configuration parameters. For database creation SQL, see the [bare-metal guide](/platform/hosting/self-managed/operator/#mysql-database). For questions about your deployment's database configuration, contact [support](mailto:support@wandb.com) or your AISE.

For complete MySQL setup instructions including configuration parameters and database creation, see the [MySQL section in the requirements page](/platform/hosting/self-managed/requirements/#mysql-database).

### Redis

W\&B depends on a single-node Redis 7.x deployment used by W\&B's components for job queuing and data caching. For convenience during testing and development of proofs of concept, W\&B Self-Managed includes a local Redis deployment that is not appropriate for production deployments.

For production deployments, W\&B can connect to a Redis instance in the following environments:

* [AWS Elasticache](https://aws.amazon.com/elasticache/)
* [Google Cloud Memory Store](https://cloud.google.com/memorystore?hl=en)
* [Azure Cache for Redis](https://azure.microsoft.com/en-us/products/cache)
* Redis deployment hosted in your cloud or on-premise infrastructure

See the [External Redis configuration section](#external-redis) for details on how to configure an external Redis instance in Helm values.

### Object storage

W\&B requires object storage with pre-signed URL and CORS support.

**Recommended storage providers:**

* [Amazon S3](https://aws.amazon.com/s3/): Object storage service offering industry-leading scalability, data availability, security, and performance.
* [Google Cloud Storage](https://cloud.google.com/storage): Managed service for storing unstructured data at scale.
* [Azure Blob Storage](https://azure.microsoft.com/en-us/products/storage/blobs): Cloud-based object storage solution for storing massive amounts of unstructured data.
* [CoreWeave AI Object Storage](https://docs.coreweave.com/products/storage/object-storage): High-performance, S3-compatible object storage service optimized for AI workloads.
* Enterprise S3-compatible storage: [MinIO Enterprise (AIStor)](https://www.min.io/product/aistor), [NetApp StorageGRID](https://www.netapp.com/data-storage/storagegrid/), or other enterprise-grade solutions

<Note>
  MinIO Open Source is in [maintenance mode](https://github.com/minio/minio) with no active development or pre-compiled binaries. For production deployments, W\&B recommends using managed object storage services or enterprise S3-compatible solutions such as MinIO Enterprise (AIStor).
</Note>

For detailed bucket provisioning instructions including IAM policies, CORS configuration, and access setup, see the [Bring Your Own Bucket (BYOB) guide](/platform/hosting/data-security/secure-storage-connector).

See the [reference architecture object storage section](/platform/hosting/self-managed/ref-arch/#object-storage) for complete requirements.

### Provision your storage bucket

Before configuring W\&B, provision your object storage bucket with proper IAM policies, CORS configuration, and access credentials.

**See the [Bring Your Own Bucket (BYOB) guide](/platform/hosting/data-security/secure-storage-connector) for detailed step-by-step provisioning instructions for:**

* Amazon S3 (including IAM policies and bucket policies)
* Google Cloud Storage (including PubSub notifications)
* Azure Blob Storage (including managed identities)
* CoreWeave AI Object Storage
* S3-compatible storage (MinIO Enterprise, NetApp StorageGRID, and other enterprise solutions)

See the [Object storage configuration section](#object-storage-bucket) for details on how to configure object storage in Helm values.

### OpenShift Kubernetes clusters

W\&B supports deployment on [OpenShift Kubernetes clusters](https://www.redhat.com/en/technologies/cloud-computing/openshift) in cloud, on-premises, and air-gapped environments.

<Note>
  W\&B recommends you install with the official W\&B Helm chart.
</Note>

#### Run the container as an un-privileged user

OpenShift and similar orchestrators often reject containers that run as root, so W\&B containers must be configured to run as a non-root user that still belongs to the root group. By default, containers use a `$UID` of 999. Specify `$UID` >= 100000 and a `$GID` of 0 if your orchestrator requires the container run with a non-root user.

<Note>
  W\&B must start as the root group (`$GID=0`) for file system permissions to function properly.
</Note>

Configure security contexts for each W\&B component. For example, to configure the API component:

```yaml theme={null}
api:
  install: true
  image:
    repository: wandb/megabinary
    tag: 0.74.1  # Replace with your actual version
  pod:
    securityContext:
      fsGroup: 10001
      fsGroupChangePolicy: Always
      runAsGroup: 0
      runAsNonRoot: true
      runAsUser: 10001
      seccompProfile:
        type: RuntimeDefault
  container:
    securityContext:
      allowPrivilegeEscalation: false
      capabilities:
        drop:
          - ALL
      privileged: false
      readOnlyRootFilesystem: false
```

If needed, configure a custom security context for other components like `app` or `console`. For details, see [Custom security context](#custom-security-context).

## Deploy W\&B Server application

<Note>
  **The W\&B Kubernetes Operator with Helm is the recommended installation method** for all W\&B self-managed deployments, including cloud, on-premises, and air-gapped environments.
</Note>

Choose your deployment method:

<Tabs>
  <Tab title="Helm CLI">
    W\&B provides a Helm chart to deploy the W\&B Kubernetes Operator to a Kubernetes cluster. This approach lets you deploy W\&B Server with Helm CLI or a continuous delivery tool like ArgoCD.

    For deployment-specific considerations, see [Environment-specific considerations](#environment-specific-considerations) and [Deploy with Terraform on public cloud](#deploy-with-terraform-on-public-cloud). For disconnected environments, see [Deploy on Air-Gapped Kubernetes](/platform/hosting/self-managed/on-premises-deployments/kubernetes-airgapped/).

    Follow these steps to install the W\&B Kubernetes Operator with Helm CLI:

    1. Add the W\&B Helm repository. The W\&B Helm chart is available in the W\&B Helm repository:
       ```shell theme={null}
       helm repo add wandb https://charts.wandb.ai
       helm repo update
       ```

    2. Install the Operator on a Kubernetes cluster:
       ```shell theme={null}
       helm upgrade --install operator wandb/operator -n wandb-cr --create-namespace
       ```

    3. Configure the W\&B operator custom resource to trigger the W\&B Server installation. Create a file named `operator.yaml` with your W\&B deployment configuration. Refer to [Configuration Reference](#configuration-reference-for-wb-server) for all available options.

       Here's a minimal example configuration:

       ```yaml theme={null}
       apiVersion: apps.wandb.com/v1
       kind: WeightsAndBiases
       metadata:
         labels:
           app.kubernetes.io/name: weightsandbiases
           app.kubernetes.io/instance: wandb
         name: wandb
         namespace: default
       spec:
         values:
           global:
             host: https://<HOST_URI>
             license: eyJhbGnUzaH...j9ZieKQ2x5GGfw
             bucket:
               <details depend on the provider>
             mysql:
               <redacted>
           ingress:
             annotations:
               <redacted>
       ```

    4. Start the Operator with your custom configuration so that it can install, configure, and manage the W\&B Server application:

       ```shell theme={null}
       kubectl apply -f operator.yaml
       ```

       Wait until the deployment completes. This takes a few minutes.

    5. To verify the installation using the web UI, create the first admin user account, then follow the verification steps outlined in [Verify the installation](#verify-the-installation).

    After these steps complete, you have a W\&B Kubernetes Operator running in the `wandb-cr` namespace and a W\&B Server application that the operator manages from your `operator.yaml` custom resource.
  </Tab>

  <Tab title="Terraform">
    Deploy W\&B using Terraform for infrastructure-as-code deployments. Choose between:

    * **Helm Terraform Module**: Deploys the operator to existing Kubernetes infrastructure.
    * **Cloud Terraform Modules**: Complete infrastructure plus application deployment for AWS, Google Cloud, and Azure.

    For deployment-specific considerations, see [Environment-specific considerations](#environment-specific-considerations) and [Deploy with Terraform on public cloud](#deploy-with-terraform-on-public-cloud). For disconnected environments, see [Deploy on Air-Gapped Kubernetes](/platform/hosting/self-managed/on-premises-deployments/kubernetes-airgapped/).

    #### Helm Terraform Module

    This method supports customized deployments tailored to specific requirements, using Terraform's infrastructure-as-code approach for consistency and repeatability. The official W\&B [Helm-based Terraform Module](https://registry.terraform.io/modules/wandb/wandb/helm/latest) is available on the Terraform Registry.

    Use the following code as a starting point. It includes all necessary configuration options for a production grade deployment:

    ```hcl theme={null}
    module "wandb" {
      source  = "wandb/wandb/helm"

      spec = {
        values = {
          global = {
            host    = "https://<HOST_URI>"
            license = "eyJhbGnUzaH...j9ZieKQ2x5GGfw"

            bucket = {
              <details depend on the provider>
            }

            mysql = {
              <redacted>
            }
          }

          ingress = {
            annotations = {
              "a" = "b"
              "x" = "y"
            }
          }
        }
      }
    }
    ```

    The configuration options are the same as described in [Configuration Reference](#configuration-reference-for-wb-server), but the syntax must follow the HashiCorp Configuration Language (HCL). The Terraform module creates the W\&B custom resource definition (CRD).

    To see how W\&B themselves use the Helm Terraform module to deploy Dedicated Cloud installations for customers, follow these links:

    * [AWS](https://github.com/wandb/terraform-aws-wandb/blob/45e1d746f53e78e73e68f911a1f8cad5408e74b6/main.tf#L225)
    * [Azure](https://github.com/wandb/terraform-azurerm-wandb/blob/170e03136b6b6fc758102d59dacda99768854045/main.tf#L155)
    * [Google Cloud](https://github.com/wandb/terraform-google-wandb/blob/49ddc3383df4cefc04337a2ae784f57ce2a2c699/main.tf#L189)

    #### Cloud Terraform Modules

    W\&B provides a set of Terraform Modules for AWS, Google Cloud, and Azure. These modules deploy the complete infrastructure, including Kubernetes clusters, load balancers, and MySQL databases, along with the W\&B Server application. The W\&B Kubernetes Operator is included in these official W\&B cloud-specific Terraform Modules with the following versions:

    | Terraform Registry                                                  | Source Code                                                                                          | Version |
    | ------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------- | ------- |
    | [AWS](https://registry.terraform.io/modules/wandb/wandb/aws/latest) | [https://github.com/wandb/terraform-aws-wandb](https://github.com/wandb/terraform-aws-wandb)         | v4.0.0+ |
    | [Azure](https://github.com/wandb/terraform-azurerm-wandb)           | [https://github.com/wandb/terraform-azurerm-wandb](https://github.com/wandb/terraform-azurerm-wandb) | v2.0.0+ |
    | [Google Cloud](https://github.com/wandb/terraform-google-wandb)     | [https://github.com/wandb/terraform-google-wandb](https://github.com/wandb/terraform-google-wandb)   | v2.0.0+ |

    These modules install the W\&B Kubernetes Operator as part of the deployment, so you can use it to manage W\&B Server in your cloud environment without additional setup.

    For detailed instructions on using these cloud-specific modules, see [Deploy with Terraform on public cloud](#deploy-with-terraform-on-public-cloud).
  </Tab>
</Tabs>

### Verify the installation

To verify the installation, W\&B recommends using the [W\&B CLI](/models/ref/cli/). The verify command executes several tests that verify all components and configurations.

<Note>
  This step assumes that the first admin user account is created with the browser.
</Note>

Follow these steps to verify the installation:

1. Install the W\&B CLI:

```bash theme={null}
pip install wandb
```

2. Log in to W\&B:

```bash theme={null}
wandb login --host=https://YOUR_DNS_DOMAIN
```

For example:

```bash theme={null}
wandb login --host=https://wandb.company-name.com
```

3. Verify the installation:

```bash theme={null}
wandb verify
```

A successful installation and fully working W\&B deployment shows the following output:

```console theme={null}
Default host selected:  https://wandb.company-name.com
Find detailed logs for this test at: /var/folders/pn/b3g3gnc11_sbsykqkm3tx5rh0000gp/T/tmpdtdjbxua/wandb
Checking if logged in...................................................✅
Checking signed URL upload..............................................✅
Checking ability to send large payloads through proxy...................✅
Checking requests to base url...........................................✅
Checking requests made over signed URLs.................................✅
Checking CORs configuration of the bucket...............................✅
Checking wandb package version is up to date............................✅
Checking logged metrics, saving and downloading a file..................✅
Checking artifact save and download workflows...........................✅
```

Contact W\&B Support if you encounter errors.

## Enable the MCP server

The [W\&B MCP Server](/platform/mcp-server) ships as an optional subchart in `operator-wandb`. When enabled, the operator deploys an in-cluster MCP server exposed through your existing ingress at `<global.host>/mcp`, so any MCP-compatible client can connect using a W\&B API key. This is the same server W\&B runs as the hosted offering at `https://mcp.withwandb.com/mcp`, but pointed at your deployment's data.

For end-user client configuration and the tool catalog, see [Use the W\&B MCP server](/platform/mcp-server). This section only covers the operator-side enablement.

### Prerequisites

Make sure your deployment meets the following requirements before you enable the MCP server:

* **Chart version**: `operator-wandb` `0.42.3` or later. The `mcp-server` subchart was introduced in `0.42.1`, but the Datadog and privacy fields used in the following example were added later.
* **Weave Traces enabled**: the MCP server depends on Weave Traces for trace tools and for the `WF_TRACE_SERVER_URL` default. Set `weave-trace.install: true`. If Weave Traces isn't enabled, the Helm render fails with `mcp-server requires weave-trace.install=true`.
* **Reachable ingress**: `global.host` must already resolve and route to the W\&B ingress. The MCP pod reads `WANDB_BASE_URL` from `global.host` and is available at `<global.host>/mcp`.
* **Node capacity**: the MCP pod requests `500m` CPU and `1Gi` memory by default (limits `2` CPU and `4Gi` memory). Confirm your node pool has enough headroom before you enable the subchart.

### Enable the subchart

Enable the `mcp-server` subchart so that the operator deploys an in-cluster MCP server and extends your existing W\&B ingress with a `/mcp` route. Add the following to the `spec.values` block of your existing `WeightsAndBiases` custom resource (CR), alongside your existing `global`, `ingress`, and other overrides. The Datadog block is optional, but recommended when a Datadog Agent DaemonSet already collects pod logs and traces in your cluster.

```yaml theme={null}
spec:
  values:
    weave-trace:
      install: true

    mcp-server:
      install: true
      image:
        repository: us-docker.pkg.dev/wandb-production/public/wandb/mcp-server
        tag: "0.3.3"
      datadog:
        enabled: true
        mode: "agent"
        service: "wandb-mcp-server-<environment>"
        env: "<environment>"
        deploymentType: "self-managed"
        customer: "<customer-name>"
        extraTags:
          - "region:<region>"
          - "tier:<tier>"
      privacy:
        logLevel: "standard"
```

Configure each block:

* **`weave-trace.install: true`**: required unless you set `mcp-server.env.WF_TRACE_SERVER_URL` yourself.
* **`datadog.mode: "agent"`**: use for Kubernetes deployments where the Datadog Agent DaemonSet owns log and trace collection. In agent mode, the MCP pod doesn't need a Datadog API key.
* **`datadog.service`, `env`, `deploymentType`, `customer`, `extraTags`**: set these to match your deployment's observability naming conventions. Set `customer` to an empty string if you don't want a customer tag.
* **`privacy.logLevel`**: use `"standard"` for most self-managed Kubernetes installations. This redacts free-text parameter values in logs while preserving deployment identifiers that operators commonly use for debugging. Use `"strict"` when entity, project, run, or user identifiers should not remain in plaintext logs. Use `"off"` only when you explicitly want plaintext logging for those values.

Apply the change to trigger reconciliation:

```bash theme={null}
kubectl apply -f operator.yaml
```

The operator creates a `wandb-mcp-server` deployment and service in the release namespace, and extends the W\&B ingress with a `/mcp` path.

### Verify the MCP server

Wait for the pod to become `Running`, then check the health endpoint in-cluster and through the ingress:

```bash theme={null}
kubectl get pod -l app.kubernetes.io/component=mcp-server
kubectl port-forward svc/wandb-mcp-server 8080:8080
curl -s http://localhost:8080/mcp/health

curl -s "https://<HOST_URI>/mcp/health"
```

Both requests should return `200 OK`. The in-cluster check confirms the pod is healthy. The ingress check confirms routing. If the in-cluster check returns `200 OK` but the ingress check returns `404 Not Found`, see [Troubleshooting](#troubleshooting). If you enabled Datadog, MCP server logs should also appear in Datadog with the configured `mcp-server.datadog.service` and `mcp-server.datadog.env` values.

### Connect a client

After the MCP server is healthy, configure your MCP client to use `https://<HOST_URI>/mcp` with a W\&B API key as the bearer token. For IDE and agent configurations, see [Use the W\&B MCP server](/platform/mcp-server).

### Troubleshooting

| Symptom                                                                                    | Cause and fix                                                                                                                                                                                                                                                                               |
| ------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `helm render` fails with `mcp-server requires weave-trace.install=true`                    | Add `weave-trace.install: true` to `spec.values`. The MCP server depends on Weave Traces for trace tools.                                                                                                                                                                                   |
| `wandb-mcp-server` pod stuck in `Pending` with `Insufficient cpu` or `Insufficient memory` | Add node capacity, or lower `mcp-server.resources.requests` in your CR. Defaults are `500m` CPU and `1Gi` memory.                                                                                                                                                                           |
| `curl https://<HOST_URI>/mcp/health` returns 404                                           | The chart renders the `/mcp` ingress path only when `mcp-server.install: true`. Reapply the CR and wait for the ingress controller to propagate the new path.                                                                                                                               |
| MCP logs don't appear in Datadog                                                           | Confirm `mcp-server.datadog.enabled: true`, `mcp-server.datadog.mode: "agent"`, and that the Datadog Agent DaemonSet collects pod stdout. Search Datadog with the configured `service` and `env` values.                                                                                    |
| MCP logs include more user-supplied text than expected                                     | Set `mcp-server.privacy.logLevel` to `"standard"` or `"strict"`. Use `"strict"` when identifiers such as entity, project, run, or user names should not remain in plaintext logs.                                                                                                           |
| `wandb-mcp-server` pod in `ImagePullBackOff` in an air-gapped or mirrored cluster          | Mirror the image to your registry and override `mcp-server.image.repository` in your CR, the same pattern used for other W\&B component images in air-gapped installs. See [Deploy on Air-Gapped Kubernetes](/platform/hosting/self-managed/on-premises-deployments/kubernetes-airgapped/). |

## Environment-specific considerations

Kubernetes is the same whether it runs on-premises or in the cloud. The main differences are in naming and managed services (for example, MySQL compared to RDS, or S3 compared to on-premises object storage). This section covers considerations that vary by environment.

### On-premises and bare metal

When deploying on on-premises or bare-metal Kubernetes, pay attention to the following.

#### Load balancer configuration

On-premises Kubernetes clusters typically require manual load balancer configuration. Options include:

* **External load balancer**: Configure an existing hardware or software load balancer, such as F5 or HAProxy.
* **Nginx Ingress Controller**: Deploy nginx-ingress-controller with NodePort or host networking.
* **MetalLB**: For bare-metal Kubernetes clusters, MetalLB provides load balancer services.

For detailed load balancer configuration examples, see the [Reference Architecture networking section](/platform/hosting/self-managed/ref-arch#networking).

#### Persistent storage

Ensure your Kubernetes cluster has a StorageClass configured for persistent volumes. W\&B components might require persistent storage for caching and temporary data.

Common on-premises storage options include:

* NFS-based storage classes
* Ceph/Rook storage
* Local persistent volumes
* Enterprise storage solutions such as NetApp or Pure Storage

#### DNS and certificate management

For on-premises deployments, complete the following tasks:

* Configure internal DNS records to point to your W\&B hostname.
* Provision SSL/TLS certificates from your internal Certificate Authority (CA).
* If using self-signed certificates, configure the operator to trust your CA certificate.

See the [SSL/TLS requirements](/platform/hosting/self-managed/requirements#ssl-tls) for certificate configuration details.

#### OpenShift deployments

W\&B fully supports deployment on OpenShift Kubernetes clusters. OpenShift deployments require additional security context configurations due to OpenShift's stricter security policies.

For OpenShift-specific configuration details, see [OpenShift Kubernetes clusters](#openshift-kubernetes-clusters). For OpenShift examples in air-gapped environments, see [Deploy on Air-Gapped Kubernetes](/platform/hosting/self-managed/on-premises-deployments/kubernetes-airgapped#openshift-configuration).

#### Object storage for on-premises and S3-compatible

After provisioning your object storage bucket (see [Object storage provisioning](/platform/hosting/data-security/secure-storage-connector)), configure it in your W\&B Custom Resource.

**AWS S3 (on-premises)**

For on-premises AWS S3 (through Outposts or compatible storage):

```yaml theme={null}
bucket:
  kmsKey: <kms key arn>  # Optional KMS key for encryption
  name: <bucket name>    # Example: wandb
  path: ""               # Keep as empty string
  provider: s3
  region: <region>       # Example: us-east-1
```

**S3-compatible storage such as MinIO, Ceph, or NetApp**

For S3-compatible storage systems:

```yaml theme={null}
bucket:
  kmsKey: null
  name: <s3 endpoint>    # Example: s3.example.com:9000
  path: <bucket name>    # Example: wandb
  provider: s3
  region: <region>       # Example: us-east-1
```

To enable TLS for S3-compatible storage, append `?tls=true` to the bucket path:

```yaml theme={null}
bucket:
  path: "wandb?tls=true"
```

<Warning>
  The certificate must be trusted. Self-signed certificates require additional configuration. See the [SSL/TLS requirements](/platform/hosting/self-managed/requirements#ssl-tls) for details.
</Warning>

**Important considerations for on-premises object storage**

When running your own object storage, consider:

1. **Storage capacity and performance**: Monitor disk capacity carefully. Average W\&B usage results in tens to hundreds of gigabytes. Heavy usage can result in petabytes of storage consumption.
2. **Fault tolerance**: At minimum, use RAID arrays for physical disks. For S3-compatible storage, use distributed or highly available configurations.
3. **Availability**: Configure monitoring to ensure the storage remains available.

**MinIO considerations**

<Warning>
  MinIO Open Source is in [maintenance mode](https://github.com/minio/minio) with no active development. Pre-compiled binaries are no longer provided, and only critical security fixes are considered case-by-case. For production deployments, W\&B recommends using managed object storage services or [MinIO Enterprise (AIStor)](https://min.io/product/aistor).
</Warning>

Enterprise alternatives for on-premises object storage include:

* [Amazon S3 on Outposts](https://aws.amazon.com/s3/outposts/)
* [NetApp StorageGRID](https://www.netapp.com/data-storage/storagegrid/)
* MinIO Enterprise (AIStor)
* [Dell ObjectScale](https://www.dell.com/en-us/shop/cty/sf/objectscale)

If you are using an existing MinIO deployment or MinIO Enterprise, you can create a bucket using the MinIO client:

```bash theme={null}
mc config host add local http://$MINIO_HOST:$MINIO_PORT "$MINIO_ACCESS_KEY" "$MINIO_SECRET_KEY" --api s3v4
mc mb --region=us-east-1 local/wandb-files
```

### Public cloud with Terraform

For full infrastructure-plus-application deployment on AWS, Google Cloud, or Azure, see [Deploy with Terraform on public cloud](#deploy-with-terraform-on-public-cloud).

## Deploy with Terraform on public cloud

<Note>
  W\&B recommends fully managed deployment options such as [W\&B Multi-tenant Cloud](/platform/hosting/hosting-options/multi_tenant_cloud) or [W\&B Dedicated Cloud](/platform/hosting/hosting-options/dedicated-cloud) deployment types. Fully managed services require little or no configuration.
</Note>

W\&B provides Terraform modules for deploying the platform on public cloud providers. These modules automate the provisioning of infrastructure and installation of W\&B Server, so you can stand up a complete environment without manually creating each cloud resource.

Before you start, W\&B recommends that you choose one of the [remote backends](https://developer.hashicorp.com/terraform/language/backend) available for Terraform to store the [State File](https://developer.hashicorp.com/terraform/language/state). The State File is the necessary resource to roll out upgrades or make changes in your deployment without recreating all components.

Select your cloud provider:

<Tabs>
  <Tab title="AWS">
    W\&B recommends using the [W\&B Server AWS Terraform Module](https://registry.terraform.io/modules/wandb/wandb/aws/latest) to deploy the platform on AWS.

    The Terraform Module deploys the following mandatory components:

    * Load Balancer
    * AWS Identity & Access Management (IAM)
    * AWS Key Management System (KMS)
    * Amazon Aurora MySQL
    * Amazon VPC
    * Amazon S3
    * Amazon Route53
    * Amazon Certificate Manager (ACM)
    * Amazon Elastic Load Balancing (ALB)
    * Amazon Secrets Manager

    Optional components include:

    * Elastic Cache for Redis
    * SQS

    ### Prerequisite permissions

    The account that runs Terraform must be able to create all components listed in the preceding section and have permission to create **IAM Policies** and **IAM Roles** and assign roles to resources.

    ### General steps

    The steps in this section are common for any deployment option.

    1. Prepare the development environment.
       * Install [Terraform](https://developer.hashicorp.com/terraform/tutorials/aws-get-started/install-cli)
       * W\&B recommends creating a Git repository for version control.

    2. Create the `terraform.tfvars` file.

       Customize the `tvfars` file content according to the installation type. The minimum recommended content looks like the following example.

       ```bash theme={null}
       namespace                  = "wandb"
       license                    = "xxxxxxxxxxyyyyyyyyyyyzzzzzzz"
       subdomain                  = "wandb-aws"
       domain_name                = "wandb.ml"
       zone_id                    = "xxxxxxxxxxxxxxxx"
       allowed_inbound_cidr       = ["0.0.0.0/0"]
       allowed_inbound_ipv6_cidr  = ["::/0"]
       eks_cluster_version        = "1.29"
       ```

       Define variables in your `tvfars` file before you deploy because the `namespace` variable is a string that prefixes all resources created by Terraform.

       The combination of `subdomain` and `domain` forms the FQDN for your W\&B instance. In the preceding example, the W\&B FQDN is `wandb-aws.wandb.ml` and the DNS `zone_id` is where Terraform creates the FQDN record.

       Both `allowed_inbound_cidr` and `allowed_inbound_ipv6_cidr` also require setting. In the module, this is a mandatory input. The following example permits access from any source to the W\&B installation.

    3. Create the file `versions.tf`.

       This file contains the Terraform and Terraform provider versions required to deploy W\&B in AWS:

       ```bash theme={null}
       provider "aws" {
         region = "eu-central-1"

         default_tags {
           tags = {
             GithubRepo = "terraform-aws-wandb"
             GithubOrg  = "wandb"
             Enviroment = "Example"
             Example    = "PublicDnsExternal"
           }
         }
       }
       ```

       Refer to the [Terraform Official Documentation](https://registry.terraform.io/providers/hashicorp/aws/latest/docs#provider-configuration) to configure the AWS provider.

       W\&B recommends that you also add the [remote backend configuration](https://developer.hashicorp.com/terraform/language/backend) mentioned at the beginning of this documentation.

    4. Create the file `variables.tf`

       For every option configured in the `terraform.tfvars` Terraform requires a correspondent variable declaration.

       ```hcl theme={null}
       variable "namespace" {
         type        = string
         description = "Name prefix used for resources"
       }

       variable "domain_name" {
         type        = string
         description = "Domain name used to access instance."
       }

       variable "subdomain" {
         type        = string
         default     = null
         description = "Subdomain for accessing the Weights & Biases UI."
       }

       variable "license" {
         type = string
       }

       variable "zone_id" {
         type        = string
         description = "Domain for creating the Weights & Biases subdomain on."
       }

       variable "allowed_inbound_cidr" {
        description = "CIDRs allowed to access wandb-server."
        nullable    = false
        type        = list(string)
       }

       variable "allowed_inbound_ipv6_cidr" {
        description = "CIDRs allowed to access wandb-server."
        nullable    = false
        type        = list(string)
       }

       variable "eks_cluster_version" {
        description = "EKS cluster kubernetes version"
        nullable    = false
        type        = string
       }
       ```

    ### Recommended deployment

    This is the most straightforward deployment option configuration that creates all mandatory components and installs in the Kubernetes Cluster the latest version of W\&B.

    1. Create the `main.tf`

       In the same directory where you created the files in the General Steps, create a file `main.tf` with the following content:

       ```hcl theme={null}
       module "wandb_infra" {
         source  = "wandb/wandb/aws"
         version = "~>7.0"

         namespace   = var.namespace
         domain_name = var.domain_name
         license     = var.license
         subdomain   = var.subdomain
         zone_id     = var.zone_id

         allowed_inbound_cidr           = var.allowed_inbound_cidr
         allowed_inbound_ipv6_cidr      = var.allowed_inbound_ipv6_cidr

         public_access                  = true
         external_dns                   = true
         kubernetes_public_access       = true
         kubernetes_public_access_cidrs = ["0.0.0.0/0"]
         eks_cluster_version            = var.eks_cluster_version
       }

        data "aws_eks_cluster" "eks_cluster_id" {
          name = module.wandb_infra.cluster_name
        }

        data "aws_eks_cluster_auth" "eks_cluster_auth" {
          name = module.wandb_infra.cluster_name
        }

        provider "kubernetes" {
          host                   = data.aws_eks_cluster.eks_cluster_id.endpoint
          cluster_ca_certificate = base64decode(data.aws_eks_cluster.eks_cluster_id.certificate_authority.0.data)
          token                  = data.aws_eks_cluster_auth.eks_cluster_auth.token
        }


        provider "helm" {
          kubernetes {
            host                   = data.aws_eks_cluster.eks_cluster_id.endpoint
            cluster_ca_certificate = base64decode(data.aws_eks_cluster.eks_cluster_id.certificate_authority.0.data)
            token                  = data.aws_eks_cluster_auth.eks_cluster_auth.token
          }
        }

        output "url" {
          value = module.wandb_infra.url
        }

        output "bucket" {
          value = module.wandb_infra.bucket_name
        }
       ```

    2. Deploy W\&B

       To deploy W\&B, execute the following commands:

       ```bash theme={null}
       terraform init
       terraform apply -var-file=terraform.tfvars
       ```

    ### Enable Redis

    To use Redis to cache SQL queries and speed up the application response when loading metrics, add the option `create_elasticache_subnet = true` to the `main.tf` file:

    ```hcl theme={null}
    module "wandb_infra" {
      source  = "wandb/wandb/aws"
      version = "~>7.0"

      namespace   = var.namespace
      domain_name = var.domain_name
      subdomain   = var.subdomain
      zone_id     = var.zone_id
      create_elasticache_subnet = true
    }
    [...]
    ```

    ### Enable message broker (queue)

    To enable an external message broker using SQS, add the option `use_internal_queue = false` to the `main.tf` file:

    <Note>
      This is optional because W\&B includes an embedded broker. This option doesn't bring a performance improvement.
    </Note>

    ```hcl theme={null}
    module "wandb_infra" {
      source  = "wandb/wandb/aws"
      version = "~>7.0"

      namespace   = var.namespace
      domain_name = var.domain_name
      subdomain   = var.subdomain
      zone_id     = var.zone_id
      use_internal_queue = false

    [...]
    }
    ```

    ### Additional resources

    * [AWS Terraform Module documentation](https://registry.terraform.io/modules/wandb/wandb/aws/latest)
    * [AWS Terraform Module source code](https://github.com/wandb/terraform-aws-wandb)
    * [Migrate to operator-based AWS Terraform modules](#migrate-to-operator-based-aws-terraform-modules)
  </Tab>

  <Tab title="Google Cloud">
    W\&B recommends using the [W\&B Server Google Cloud Terraform Module](https://registry.terraform.io/modules/wandb/wandb/google/latest) to deploy the platform on Google Cloud.

    The module documentation lists all available options.

    Before you start, W\&B recommends that you choose one of the [remote backends](https://developer.hashicorp.com/terraform/language/backend/remote) available for Terraform to store the [State File](https://developer.hashicorp.com/terraform/language/state). The State File is the necessary resource to roll out upgrades or make changes in your deployment without recreating all components.

    The Terraform Module deploys the following mandatory components:

    * VPC
    * Cloud SQL for MySQL
    * Cloud Storage Bucket
    * Google Kubernetes Engine
    * Memorystore for Redis
    * KMS Crypto Key
    * Load Balancer

    Optional components include:

    * Pub/Sub messages system

    ### Prerequisite permissions

    The account that runs Terraform must have the role `roles/owner` in the Google Cloud project used.

    ### General steps

    The steps in this section are common for any deployment option.

    1. Prepare the development environment.
       * Install [Terraform](https://developer.hashicorp.com/terraform/tutorials/aws-get-started/install-cli).
       * W\&B recommends creating a Git repository for your code, but you can keep your files locally.
       * Create a project in [Google Cloud Console](https://console.cloud.google.com/).
       * Authenticate with Google Cloud (make sure to [install gcloud](https://cloud.google.com/sdk/docs/install) first) using `gcloud auth application-default login`.

    2. Create the `terraform.tfvars` file.

       Customize the `tvfars` file content according to the installation type. The minimum recommended content looks like the following example.

       ```bash theme={null}
       project_id  = "wandb-project"
       region      = "europe-west2"
       zone        = "europe-west2-a"
       namespace   = "wandb"
       license     = "xxxxxxxxxxyyyyyyyyyyyzzzzzzz"
       subdomain   = "wandb-gcp"
       domain_name = "wandb.ml"
       ```

       You must decide the values for these variables before deployment. The `namespace` variable is a string that prefixes all resources created by Terraform.

       The combination of `subdomain` and `domain` forms the FQDN that configures W\&B. In the preceding example, the W\&B FQDN is `wandb-gcp.wandb.ml`.

    3. Create the file `variables.tf`.

       For every option configured in the `terraform.tfvars` Terraform requires a correspondent variable declaration.

       ```hcl theme={null}
       variable "project_id" {
         type        = string
         description = "Project ID"
       }

       variable "region" {
         type        = string
         description = "Google region"
       }

       variable "zone" {
         type        = string
         description = "Google zone"
       }

       variable "namespace" {
         type        = string
         description = "Namespace prefix used for resources"
       }

       variable "domain_name" {
         type        = string
         description = "Domain name for accessing the Weights & Biases UI."
       }

       variable "subdomain" {
         type        = string
         description = "Subdomain for access the Weights & Biases UI."
       }

       variable "license" {
         type        = string
         description = "W&B License"
       }
       ```

    ### Recommended deployment

    This is the most straightforward deployment option configuration that creates all mandatory components and installs in the Kubernetes Cluster the latest version of W\&B.

    1. Create the `main.tf`

       In the same directory where you created the files in the General Steps, create a file `main.tf` with the following content:

       ```hcl theme={null}
       provider "google" {
        project = var.project_id
        region  = var.region
        zone    = var.zone
       }

       provider "google-beta" {
        project = var.project_id
        region  = var.region
        zone    = var.zone
       }

       data "google_client_config" "current" {}

       provider "kubernetes" {
         host                   = "https://${module.wandb.cluster_endpoint}"
         cluster_ca_certificate = base64decode(module.wandb.cluster_ca_certificate)
         token                  = data.google_client_config.current.access_token
       }

       provider "helm" {
         kubernetes {
           host                   = "https://${module.wandb.cluster_endpoint}"
           cluster_ca_certificate = base64decode(module.wandb.cluster_ca_certificate)
           token                  = data.google_client_config.current.access_token
         }
       }

       # Spin up all required services
       module "wandb" {
         source  = "wandb/wandb/google"
         version = "~> 10.0"

         namespace   = var.namespace
         license     = var.license
         domain_name = var.domain_name
         subdomain   = var.subdomain
       }

       # You'll want to update your DNS with the provisioned IP address
       output "url" {
         value = module.wandb.url
       }

       output "address" {
         value = module.wandb.address
       }

       output "bucket_name" {
         value = module.wandb.bucket_name
       }
       ```

    2. Deploy W\&B.

       To deploy W\&B, execute the following commands:

       ```bash theme={null}
       terraform init
       terraform apply -var-file=terraform.tfvars
       ```

    ### Enable Redis

    To use Redis to cache SQL queries and speed up the application response when loading metrics, add the option `create_redis = true` to the `main.tf` file:

    ```hcl theme={null}
    [...]

    module "wandb" {
      source  = "wandb/wandb/google"
      version = "~> 10.0"

      namespace    = var.namespace
      license      = var.license
      domain_name  = var.domain_name
      subdomain    = var.subdomain
      create_redis = true
    }
    [...]
    ```

    ### Enable message broker (queue)

    To enable an external message broker using Pub/Sub, add the option `use_internal_queue = false` to the `main.tf` file:

    <Note>
      This is optional because W\&B includes an embedded broker. This option doesn't bring a performance improvement.
    </Note>

    ```hcl theme={null}
    [...]

    module "wandb" {
      source  = "wandb/wandb/google"
      version = "~> 10.0"

      namespace          = var.namespace
      license            = var.license
      domain_name        = var.domain_name
      subdomain          = var.subdomain
      use_internal_queue = false
    }

    [...]
    ```

    ### Additional resources

    * [Google Cloud Terraform Module documentation](https://registry.terraform.io/modules/wandb/wandb/google/latest)
    * [Google Cloud Terraform Module source code](https://github.com/wandb/terraform-google-wandb)
  </Tab>

  <Tab title="Azure">
    W\&B recommends using the [W\&B Server Azure Terraform Module](https://registry.terraform.io/modules/wandb/wandb/azurerm/latest) to deploy the platform on Azure.

    The module documentation lists all available options.

    The Terraform Module deploys the following mandatory components:

    * Azure Resource Group
    * Azure Virtual Network (VPC)
    * Azure MySQL Flexible Server
    * Azure Storage Account and Blob Storage
    * Azure Kubernetes Service
    * Azure Application Gateway

    Optional components include:

    * Azure Cache for Redis
    * Azure Event Grid

    ### Prerequisite permissions

    The simplest way to configure the AzureRM provider is through the [Azure CLI](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/guides/azure_cli). For automation, you can also use an [Azure Service Principal](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/guides/service_principal_client_secret).

    Regardless of the authentication method used, the account that runs Terraform must be able to create all components listed in the preceding section.

    ### General steps

    The steps in this section are common for any deployment option.

    1. Prepare the development environment.
       * Install [Terraform](https://developer.hashicorp.com/terraform/tutorials/aws-get-started/install-cli)
       * W\&B recommends creating a Git repository for your code, but you can keep your files locally.

    2. Create the `terraform.tfvars` file.

       Customize the `tvfars` file content according to the installation type. The minimum recommended content looks like the following example.

       ```bash theme={null}
        namespace     = "wandb"
        wandb_license = "xxxxxxxxxxyyyyyyyyyyyzzzzzzz"
        subdomain     = "wandb-azure"
        domain_name   = "wandb.ml"
        location      = "westeurope"
       ```

       You must decide the values for these variables before deployment. The `namespace` variable is a string that prefixes all resources created by Terraform.

       The combination of `subdomain` and `domain` forms the FQDN that configures W\&B. In the preceding example, the W\&B FQDN is `wandb-azure.wandb.ml`.

    3. Create the file `versions.tf`.

       This file contains the Terraform and Terraform provider versions required to deploy W\&B in Azure:

       ```bash theme={null}
         terraform {
        required_version = "~> 1.3"

        required_providers {
          azurerm = {
            source  = "hashicorp/azurerm"
            version = "~> 3.17"
          }
        }
         }
       ```

       Refer to the [Terraform Official Documentation](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs) to configure the Azure provider.

       W\&B recommends that you also add the [remote backend configuration](https://developer.hashicorp.com/terraform/language/backend) mentioned at the beginning of this documentation.

    4. Create the file `variables.tf`

       For every option configured in the `terraform.tfvars` Terraform requires a correspondent variable declaration.

       ```bash theme={null}
           variable "namespace" {
             type        = string
             description = "String used for prefix resources."
           }

           variable "location" {
             type        = string
             description = "Azure Resource Group location"
           }

           variable "domain_name" {
             type        = string
             description = "Domain for accessing the Weights & Biases UI."
           }

           variable "subdomain" {
             type        = string
             default     = null
             description = "Subdomain for accessing the Weights & Biases UI. Default creates record at Route53 Route."
           }

           variable "license" {
             type        = string
             description = "Your wandb/local license"
           }
       ```

    ### Recommended deployment

    This is the most straightforward deployment option configuration that creates all mandatory components and installs in the Kubernetes Cluster the latest version of W\&B.

    1. Create the `main.tf`

       In the same directory where you created the files in the General Steps, create a file `main.tf` with the following content:

       ```bash theme={null}
         provider "azurerm" {
        features {}
         }

         provider "kubernetes" {
        host                   = module.wandb.cluster_host
        cluster_ca_certificate = base64decode(module.wandb.cluster_ca_certificate)
        client_key             = base64decode(module.wandb.cluster_client_key)
        client_certificate     = base64decode(module.wandb.cluster_client_certificate)
         }

         provider "helm" {
        kubernetes {
          host                   = module.wandb.cluster_host
          cluster_ca_certificate = base64decode(module.wandb.cluster_ca_certificate)
          client_key             = base64decode(module.wandb.cluster_client_key)
          client_certificate     = base64decode(module.wandb.cluster_client_certificate)
        }
         }

         # Spin up all required services
         module "wandb" {
        source  = "wandb/wandb/azurerm"
        version = "~> 1.2"

        namespace   = var.namespace
        location    = var.location
        license     = var.license
        domain_name = var.domain_name
        subdomain   = var.subdomain

        deletion_protection = false

        tags = {
          "Example" : "PublicDns"
        }
         }

         output "address" {
        value = module.wandb.address
         }

         output "url" {
        value = module.wandb.url
         }
       ```

    2. Deploy W\&B

       To deploy W\&B, execute the following commands:

       ```bash theme={null}
       terraform init
       terraform apply -var-file=terraform.tfvars
       ```

    ### Enable Redis

    To use Redis to cache SQL queries and speed up the application response when loading metrics, add the option `create_redis = true` to the `main.tf` file:

    ```bash theme={null}
    # Spin up all required services
    module "wandb" {
      source  = "wandb/wandb/azurerm"
      version = "~> 1.2"

      namespace   = var.namespace
      location    = var.location
      license     = var.license
      domain_name = var.domain_name
      subdomain   = var.subdomain

      create_redis = true
      [...]
    }
    ```

    ### Enable message broker (queue)

    To enable an external message broker using Azure Event Grid, add the option `use_internal_queue = false` to the `main.tf` file:

    <Note>
      This is optional because W\&B includes an embedded broker. This option doesn't bring a performance improvement.
    </Note>

    ```bash theme={null}
    # Spin up all required services
    module "wandb" {
      source  = "wandb/wandb/azurerm"
      version = "~> 1.2"

      namespace   = var.namespace
      location    = var.location
      license     = var.license
      domain_name = var.domain_name
      subdomain   = var.subdomain

      use_internal_queue = false
      [...]
    }
    ```

    ### Additional resources

    * [Azure Terraform Module documentation](https://registry.terraform.io/modules/wandb/wandb/azurerm/latest)
    * [Azure Terraform Module source code](https://github.com/wandb/terraform-azurerm-wandb)
  </Tab>
</Tabs>

### Other deployment options

You can combine multiple deployment options by adding all configurations to the same file. Each Terraform module provides several options that can be combined with the standard options and the minimal configuration found in the recommended deployment section.

Refer to the module documentation for your cloud provider for the full list of available options:

* [AWS Module documentation](https://registry.terraform.io/modules/wandb/wandb/aws/latest)
* [Google Cloud Module documentation](https://registry.terraform.io/modules/wandb/wandb/google/latest)
* [Azure Module documentation](https://registry.terraform.io/modules/wandb/wandb/azurerm/latest)

## Access the W\&B management console

The W\&B Kubernetes Operator comes with a management console where you can review deployment status, view component metrics, and adjust operator-level settings. It's available at `${HOST_URI}/console`, for example `https://wandb.company-name.com/console`.

You can log in to the management console in two ways:

<Tabs>
  <Tab title="Option 1 (Recommended)">
    1. Open the W\&B application in the browser and log in. Log in to the W\&B application with `${HOST_URI}/`, for example `https://wandb.company-name.com/`
    2. Access the console. Click the icon in the top right corner and then click **System console**. Only users with admin privileges can see the **System console** entry.

       <Frame>
         <img alt="System console access" />
       </Frame>
  </Tab>

  <Tab title="Option 2">
    <Note>
      W\&B recommends you access the console using the following steps only if Option 1 doesn't work.
    </Note>

    1. Open the console application in your browser. Open the URL described in the preceding section, which redirects you to the login screen:
       <Frame>
         <img alt="Direct system console access" />
       </Frame>
    2. Retrieve the password from the Kubernetes secret that the installation generates:
       ```shell theme={null}
       kubectl get secret wandb-password -o jsonpath='{.data.password}' | base64 -d
       ```
       Copy the password.
    3. Log in to the console. Paste the copied password, then click **Login**.
  </Tab>
</Tabs>

## Update the W\&B Kubernetes Operator

This section describes how to update the W\&B Kubernetes Operator itself. Update the operator periodically so that you get bug fixes and new reconciliation features.

<Note>
  * Updating the W\&B Kubernetes Operator doesn't update the W\&B Server application.
  * If you use a Helm chart that doesn't use the W\&B Kubernetes Operator, see the [migration instructions](#migrate-self-managed-instances-to-wb-operator) before following the steps in this section to update the W\&B Operator.
</Note>

Copy and paste the following code snippets into your terminal.

1. Update the repo with [`helm repo update`](https://helm.sh/docs/helm/helm_repo_update/):
   ```shell theme={null}
   helm repo update
   ```

2. Update the Helm chart with [`helm upgrade`](https://helm.sh/docs/helm/helm_upgrade/):
   ```shell theme={null}
   helm upgrade operator wandb/operator -n wandb-cr --reuse-values
   ```

## Update the W\&B Server application

You no longer need to update W\&B Server application if you use the W\&B Kubernetes Operator.

The operator automatically updates your W\&B Server application when a new version of the software of W\&B is released.

## Migrate self-managed instances to W\&B Operator

The following section describes how to migrate from self-managing your own W\&B Server installation to using the W\&B Operator to do this for you. Migrating lets the operator handle reconciliation and W\&B Server upgrades automatically, so you no longer have to coordinate manifest changes or Helm upgrades for the application. The migration process depends on how you installed W\&B Server:

<Note>
  The W\&B Operator is the default and recommended installation method for W\&B Server. Reach out to [Customer Support](mailto:support@wandb.com) or your W\&B team if you have any questions.
</Note>

* If you used the official W\&B Cloud Terraform Modules, navigate to the appropriate documentation and follow the steps there:
  * [AWS](#migrate-to-operator-based-aws-terraform-modules)
  * [Google Cloud](#migrate-to-operator-based-google-cloud-terraform-modules)
  * [Azure](#migrate-to-operator-based-azure-terraform-modules)
* If you used the [W\&B Non-Operator Helm chart](https://github.com/wandb/helm-charts/tree/main/charts/wandb), see [Migrate to operator-based Helm chart](#migrate-to-operator-based-helm-chart).
* If you used the [W\&B Non-Operator Helm chart with Terraform](https://registry.terraform.io/modules/wandb/wandb/kubernetes/latest), see [Migrate to operator-based Terraform Helm chart](#migrate-to-operator-based-terraform-helm-chart).
* If you created the Kubernetes resources with manifests, see [Migrate to operator-based Helm chart](#migrate-to-operator-based-helm-chart).

### Migrate to operator-based AWS Terraform modules

For a detailed description of the migration process, see the [operator-wandb chart documentation](https://github.com/wandb/helm-charts/tree/main/charts/operator-wandb).

### Migrate to operator-based Google Cloud Terraform modules

Reach out to [Customer Support](mailto:support@wandb.com) or your W\&B team if you have any questions or need assistance.

### Migrate to operator-based Azure Terraform modules

Reach out to [Customer Support](mailto:support@wandb.com) or your W\&B team if you have any questions or need assistance.

### Migrate to operator-based Helm chart

Follow these steps to migrate to the operator-based Helm chart:

1. Get the current W\&B configuration. If you deployed W\&B with a non-operator-based version of the Helm chart, export the values like this:
   ```shell theme={null}
   helm get values wandb
   ```
   If you deployed W\&B with Kubernetes manifests, export the values like this:
   ```shell theme={null}
   kubectl get deployment wandb -o yaml
   ```
   You now have all the configuration values you need for the next step.

2. Create a file called `operator.yaml`. Follow the format described in the [Configuration Reference](#configuration-reference-for-wb-operator). Use the values from step 1.

3. Scale the current deployment to 0 pods. This step stops the current deployment.
   ```shell theme={null}
   kubectl scale --replicas=0 deployment wandb
   ```

4. Update the Helm chart repo:
   ```shell theme={null}
   helm repo update
   ```

5. Install the new Helm chart:
   ```shell theme={null}
   helm upgrade --install operator wandb/operator -n wandb-cr --create-namespace
   ```

6. Configure the new Helm chart and trigger W\&B application deployment. Apply the new configuration.
   ```shell theme={null}
   kubectl apply -f operator.yaml
   ```
   The deployment takes a few minutes to complete.

7. Verify the installation. Make sure that everything works by following the steps in [Verify the installation](#verify-the-installation).

8. Remove the old installation. Uninstall the old Helm chart or delete the resources that you created with manifests.

### Migrate to operator-based Terraform Helm chart

Follow these steps to migrate to the operator-based Helm chart:

1. Prepare Terraform config. Replace the Terraform code from the old deployment in your Terraform config with the code described in [Deploy W\&B with Helm Terraform module](#deploy-wb-with-helm-terraform-module). Set the same variables as before. Do not change the `.tfvars` file if you have one.
2. Execute Terraform run. Execute `terraform init`, `terraform plan`, and `terraform apply`.
3. Verify the installation. Make sure that everything works by following the steps in [Verify the installation](#verify-the-installation).
4. Remove the old installation. Uninstall the old Helm chart or delete the resources that you created with manifests.

## Configuration reference for W\&B Server

This section is a reference for the configuration options that you set in your `WeightsAndBiases` custom resource. Use it to look up the YAML schema for a specific subsystem (for example, MySQL, Redis, ingress, or OIDC) as you build or update your `operator.yaml` file.

This section describes the configuration options for W\&B Server application. The application receives its configuration as custom resource definition named [WeightsAndBiases](#how-it-works). Some configuration options are exposed with the following configuration. You must set others as environment variables.

The documentation has two lists of environment variables: [basic](/platform/hosting/env-vars/) and [advanced](/platform/hosting/iam/advanced_env_vars/). Only use environment variables if the configuration option that you need is not exposed using the Helm chart.

### Basic example

This example defines the minimum set of values required for W\&B. For a more realistic production example, see [Complete example](#complete-example).

This YAML file defines the desired state of your W\&B deployment, including the version, environment variables, external resources like databases, and other necessary settings.

```yaml theme={null}
apiVersion: apps.wandb.com/v1
kind: WeightsAndBiases
metadata:
  labels:
    app.kubernetes.io/name: weightsandbiases
    app.kubernetes.io/instance: wandb
  name: wandb
  namespace: default
spec:
  values:
    global:
      host: https://<HOST_URI>
      license: eyJhbGnUzaH...j9ZieKQ2x5GGfw
      bucket:
        <details depend on the provider>
      mysql:
        <redacted>
    ingress:
      annotations:
        <redacted>
```

Find the full set of values in the [W\&B Helm repository](https://github.com/wandb/helm-charts/blob/main/charts/operator-wandb/values.yaml). **Change only those values you need to override**.

### Complete example

This example configuration deploys W\&B to Google Cloud Anthos using Google Cloud Storage:

```yaml theme={null}
apiVersion: apps.wandb.com/v1
kind: WeightsAndBiases
metadata:
  labels:
    app.kubernetes.io/name: weightsandbiases
    app.kubernetes.io/instance: wandb
  name: wandb
  namespace: default
spec:
  values:
    global:
      host: https://abc-wandb.sandbox-gcp.wandb.ml
      bucket:
        name: abc-wandb-moving-pipefish
        provider: gcs
      mysql:
        database: wandb_local
        host: 10.218.0.2
        name: wandb_local
        password: 8wtX6cJHizAZvYScjDzZcUarK4zZGjpV
        port: 3306
        user: wandb
      redis:
        host: redis.example.com
        port: 6379
        password: password
      api:
        enabled: true
      glue:
        enabled: true
      executor:
        enabled: true
      license: eyJhbGnUzaHgyQjQyQWhEU3...ZieKQ2x5GGfw
    ingress:
      annotations:
        ingress.gcp.kubernetes.io/pre-shared-cert: abc-wandb-cert-creative-puma
        kubernetes.io/ingress.class: gce
        kubernetes.io/ingress.global-static-ip-name: abc-wandb-operator-address
```

### Host

```yaml theme={null}
 # Provide the FQDN with protocol
global:
  # example host name, replace with your own
  host: https://wandb.example.com
```

### Object storage (bucket)

**AWS**

```yaml theme={null}
global:
  bucket:
    provider: "s3"
    name: ""
    kmsKey: ""
    region: ""
```

**Google Cloud**

```yaml theme={null}
global:
  bucket:
    provider: "gcs"
    name: ""
```

**Azure**

```yaml theme={null}
global:
  bucket:
    provider: "az"
    name: ""
    secretKey: ""
```

**Other providers (Minio, Ceph, and other S3-compatible storage)**

For other S3 compatible providers, set the bucket configuration as follows:

```yaml theme={null}
global:
  bucket:
    # Example values, replace with your own
    provider: s3
    name: storage.example.com
    kmsKey: null
    path: wandb
    region: default
    accessKey: 5WOA500...P5DK7I
    secretKey: HDKYe4Q...JAp1YyjysnX
```

For S3-compatible storage hosted outside of AWS, `kmsKey` must be `null`.

To reference `accessKey` and `secretKey` from a secret:

```yaml theme={null}
global:
  bucket:
    # Example values, replace with your own
    provider: s3
    name: storage.example.com
    kmsKey: null
    path: wandb
    region: default
    secret:
      secretName: bucket-secret
      accessKeyName: ACCESS_KEY
      secretKeyName: SECRET_KEY
```

### MySQL

```yaml theme={null}
global:
   mysql:
     # Example values, replace with your own
     host: db.example.com
     port: 3306
     database: wandb_local
     user: wandb
     password: 8wtX6cJH...ZcUarK4zZGjpV 
```

To reference the `password` from a secret:

```yaml theme={null}
global:
   mysql:
     # Example values, replace with your own
     host: db.example.com
     port: 3306
     database: wandb_local
     user: wandb
     passwordSecret:
       name: database-secret
       passwordKey: MYSQL_WANDB_PASSWORD
```

### License

```yaml theme={null}
global:
  # Example license, replace with your own
  license: eyJhbGnUzaHgyQjQy...VFnPS_KETXg1hi
```

To reference the `license` from a secret:

```yaml theme={null}
global:
  licenseSecret:
    name: license-secret
    key: CUSTOMER_WANDB_LICENSE
```

### Ingress

See [How to identify the Kubernetes ingress class](#how-to-identify-the-kubernetes-ingress-class).

**Without TLS**

```yaml theme={null}
global:
# IMPORTANT: Ingress is on the same level in the YAML as 'global' (not a child)
ingress:
  class: ""
```

**With TLS**

Create a secret that contains the certificate

```console theme={null}
kubectl create secret tls wandb-ingress-tls --key wandb-ingress-tls.key --cert wandb-ingress-tls.crt
```

Reference the secret in the ingress configuration

```yaml theme={null}
global:
# IMPORTANT: Ingress is on the same level in the YAML as 'global' (not a child)
ingress:
  class: ""
  annotations:
    {}
    # kubernetes.io/ingress.class: nginx
    # kubernetes.io/tls-acme: "true"
  tls: 
    - secretName: wandb-ingress-tls
      hosts:
        - <HOST_URI>
```

For Nginx, you might have to add the following annotation:

```yaml theme={null}
ingress:
  annotations:
    nginx.ingress.kubernetes.io/proxy-body-size: 0
```

### Custom Kubernetes service accounts

Specify custom Kubernetes service accounts to run the W\&B pods.

The following snippet creates a service account as part of the deployment with the specified name:

```yaml theme={null}
app:
  serviceAccount:
    name: custom-service-account
    create: true

parquet:
  serviceAccount:
    name: custom-service-account
    create: true

global:
  ...
```

The subsystems "app" and "parquet" run under the specified service account. The other subsystems run under the default service account.

If the service account already exists on the cluster, set `create: false`:

```yaml theme={null}
app:
  serviceAccount:
    name: custom-service-account
    create: false

parquet:
  serviceAccount:
    name: custom-service-account
    create: false
    
global:
  ...
```

You can specify service accounts on different subsystems such as app, parquet, console, and others:

```yaml theme={null}
app:
  serviceAccount:
    name: custom-service-account
    create: true

console:
  serviceAccount:
    name: custom-service-account
    create: true

global:
  ...
```

The service accounts can be different between the subsystems:

```yaml theme={null}
app:
  serviceAccount:
    name: custom-service-account
    create: false

console:
  serviceAccount:
    name: another-custom-service-account
    create: true

global:
  ...
```

### External Redis

```yaml theme={null}
redis:
  install: false

global:
  redis:
    host: ""
    port: 6379
    password: ""
    parameters: {}
    caCert: ""
```

To reference the `password` from a secret:

```console theme={null}
kubectl create secret generic redis-secret --from-literal=redis-password=supersecret
```

Reference it in the following configuration:

```yaml theme={null}
redis:
  install: false

global:
  redis:
    host: redis.example
    port: 9001
    auth:
      enabled: true
      secret: redis-secret
      key: redis-password
```

### LDAP

<Warning>
  LDAP configuration support in the current Helm chart is limited. Contact W\&B Support or your AISE for assistance configuring LDAP.
</Warning>

Configure LDAP by setting environment variables in `global.extraEnv`:

```yaml theme={null}
global:
  extraEnv:
    LDAP_ADDRESS: ldaps://ldap.company.example.com
    LDAP_BASE_DN: cn=accounts,dc=company,dc=example,dc=com
    LDAP_USER_BASE_DN: cn=users,cn=accounts,dc=company,dc=example,dc=com
    LDAP_GROUP_BASE_DN: cn=groups,cn=accounts,dc=company,dc=example,dc=com
    LDAP_BIND_DN: uid=ldapbind,cn=sysaccounts,cn=etc,dc=company,dc=example,dc=com
    LDAP_BIND_PW: ********************
    LDAP_ATTRIBUTES: email=mail,name=cn
    LDAP_TLS_ENABLE: "true"
    LDAP_LOGIN: "true"
    LDAP_USER_OBJECT_CLASS: user
    LDAP_GROUP_OBJECT_CLASS: group
```

<accordion title="Legacy LDAP configuration">
  This legacy approach is no longer recommended. This section is provided for reference.

  **Without TLS**

  ```yaml theme={null}
  global:
    ldap:
      enabled: true
      # LDAP server address including "ldap://" or "ldaps://"
      host:
      # LDAP search base to use for finding users
      baseDN:
      # LDAP user to bind with (if not using anonymous bind)
      bindDN:
      # Secret name and key with LDAP password to bind with (if not using anonymous bind)
      bindPW:
      # LDAP attribute for email and group ID attribute names as comma separated string values.
      attributes:
      # LDAP group allow list
      groupAllowList:
      # Enable LDAP TLS
      tls: false
  ```

  **With TLS**

  The LDAP TLS cert configuration requires a config map pre-created with the certificate content.

  To create the config map you can use the following command:

  ```console theme={null}
  kubectl create configmap ldap-tls-cert --from-file=certificate.crt
  ```

  And use the config map in the YAML like the following example.

  ```yaml theme={null}
  global:
    ldap:
      enabled: true
      # LDAP server address including "ldap://" or "ldaps://"
      host:
      # LDAP search base to use for finding users
      baseDN:
      # LDAP user to bind with (if not using anonymous bind)
      bindDN:
      # Secret name and key with LDAP password to bind with (if not using anonymous bind)
      bindPW:
      # LDAP attribute for email and group ID attribute names as comma separated string values.
      attributes:
      # LDAP group allow list
      groupAllowList:
      # Enable LDAP TLS
      tls: true
      # ConfigMap name and key with CA certificate for LDAP server
      tlsCert:
        configMap:
          name: "ldap-tls-cert"
          key: "certificate.crt"
  ```
</accordion>

### OIDC SSO

```yaml theme={null}
global: 
  auth:
    sessionLengthHours: 720
    oidc:
      clientId: ""
      secret: ""
      # Only include if your IdP requires it.
      authMethod: ""
      issuer: ""
```

`authMethod` is optional.

### SMTP

```yaml theme={null}
global:
  email:
    smtp:
      host: ""
      port: 587
      user: ""
      password: ""
```

### Environment variables

```yaml theme={null}
global:
  extraEnv:
    GLOBAL_ENV: "example"
```

### Custom certificate authority

`customCACerts` is a list and can take many certificates. Certificate authorities specified in `customCACerts` only apply to the W\&B Server application.

```yaml theme={null}
global:
  customCACerts:
  - |
    -----BEGIN CERTIFICATE-----
    MIIBnDCCAUKgAwIBAg.....................fucMwCgYIKoZIzj0EAwIwLDEQ
    MA4GA1UEChMHSG9tZU.....................tZUxhYiBSb290IENBMB4XDTI0
    MDQwMTA4MjgzMFoXDT.....................oNWYggsMo8O+0mWLYMAoGCCqG
    SM49BAMCA0gAMEUCIQ.....................hwuJgyQRaqMI149div72V2QIg
    P5GD+5I+02yEp58Cwxd5Bj2CvyQwTjTO4hiVl1Xd0M0=
    -----END CERTIFICATE-----
  - |
    -----BEGIN CERTIFICATE-----
    MIIBxTCCAWugAwIB.......................qaJcwCgYIKoZIzj0EAwIwLDEQ
    MA4GA1UEChMHSG9t.......................tZUxhYiBSb290IENBMB4XDTI0
    MDQwMTA4MjgzMVoX.......................UK+moK4nZYvpNpqfvz/7m5wKU
    SAAwRQIhAIzXZMW4.......................E8UFqsCcILdXjAiA7iTluM0IU
    aIgJYVqKxXt25blH/VyBRzvNhViesfkNUQ==
    -----END CERTIFICATE-----
```

CA certificates can also be stored in a ConfigMap:

```yaml theme={null}
global:
  caCertsConfigMap: custom-ca-certs
```

The ConfigMap must look like this:

```yaml theme={null}
apiVersion: v1
kind: ConfigMap
metadata:
  name: custom-ca-certs
data:
  ca-cert1.crt: |
    -----BEGIN CERTIFICATE-----
    ...
    -----END CERTIFICATE-----
  ca-cert2.crt: |
    -----BEGIN CERTIFICATE-----
    ...
    -----END CERTIFICATE-----
```

<Note>
  If using a ConfigMap, each key in the ConfigMap must end with `.crt` (for example, `my-cert.crt` or `ca-cert1.crt`). This naming convention is required for `update-ca-certificates` to parse and add each certificate to the system CA store.
</Note>

### Custom security context

Each W\&B component supports custom security context configurations of the following form:

```yaml theme={null}
pod:
  securityContext:
    runAsNonRoot: true
    runAsUser: 1001
    runAsGroup: 0
    fsGroup: 1001
    fsGroupChangePolicy: Always
    seccompProfile:
      type: RuntimeDefault
container:
  securityContext:
    capabilities:
      drop:
        - ALL
    readOnlyRootFilesystem: false
    allowPrivilegeEscalation: false 
```

<Note>
  The only valid value for `runAsGroup:` is `0`. Any other value is an error.
</Note>

For example, to configure the application pod, add a section `app` to your configuration:

```yaml theme={null}
global:
  ...
app:
  pod:
    securityContext:
      runAsNonRoot: true
      runAsUser: 1001
      runAsGroup: 0
      fsGroup: 1001
      fsGroupChangePolicy: Always
      seccompProfile:
        type: RuntimeDefault
  container:
    securityContext:
      capabilities:
        drop:
          - ALL
      readOnlyRootFilesystem: false
      allowPrivilegeEscalation: false 
```

The same concept applies to `console`, `weave`, `weave-trace`, and `parquet`.

## Configuration reference for W\&B Operator

This section describes configuration options for W\&B Kubernetes Operator (`wandb-controller-manager`). The operator receives its configuration in the form of a YAML file.

By default, the W\&B Kubernetes Operator doesn't need a configuration file. Create a configuration file if required. For example, you might need a configuration file to specify custom certificate authorities, deploy in an air gap environment, and so forth.

Find the full list of spec customization [in the Helm repository](https://github.com/wandb/helm-charts/blob/main/charts/operator/values.yaml).

### Custom CA

A custom certificate authority (`customCACerts`) is a list and can take many certificates. Those certificate authorities, when added, only apply to the W\&B Kubernetes Operator (`wandb-controller-manager`).

```yaml theme={null}
customCACerts:
- |
  -----BEGIN CERTIFICATE-----
  MIIBnDCCAUKgAwIBAg.....................fucMwCgYIKoZIzj0EAwIwLDEQ
  MA4GA1UEChMHSG9tZU.....................tZUxhYiBSb290IENBMB4XDTI0
  MDQwMTA4MjgzMFoXDT.....................oNWYggsMo8O+0mWLYMAoGCCqG
  SM49BAMCA0gAMEUCIQ.....................hwuJgyQRaqMI149div72V2QIg
  P5GD+5I+02yEp58Cwxd5Bj2CvyQwTjTO4hiVl1Xd0M0=
  -----END CERTIFICATE-----
- |
  -----BEGIN CERTIFICATE-----
  MIIBxTCCAWugAwIB.......................qaJcwCgYIKoZIzj0EAwIwLDEQ
  MA4GA1UEChMHSG9t.......................tZUxhYiBSb290IENBMB4XDTI0
  MDQwMTA4MjgzMVoX.......................UK+moK4nZYvpNpqfvz/7m5wKU
  SAAwRQIhAIzXZMW4.......................E8UFqsCcILdXjAiA7iTluM0IU
  aIgJYVqKxXt25blH/VyBRzvNhViesfkNUQ==
  -----END CERTIFICATE-----
```

CA certificates can also be stored in a ConfigMap:

```yaml theme={null}
caCertsConfigMap: custom-ca-certs
```

The ConfigMap must look like this:

```yaml theme={null}
apiVersion: v1
kind: ConfigMap
metadata:
  name: custom-ca-certs
data:
  ca-cert1.crt: |
    -----BEGIN CERTIFICATE-----
    ...
    -----END CERTIFICATE-----
  ca-cert2.crt: |
    -----BEGIN CERTIFICATE-----
    ...
    -----END CERTIFICATE-----
```

<Note>
  Each key in the ConfigMap must end with `.crt` (for example, `my-cert.crt` or `ca-cert1.crt`). This naming convention is required for `update-ca-certificates` to parse and add each certificate to the system CA store.
</Note>

## FAQ

### Purpose and role of each pod

A W\&B Server deployment includes the following pods:

* **`wandb-app`**: the core of W\&B, including the GraphQL API and frontend application. It powers most of the W\&B platform's functionality.
* **`wandb-console`**: the administration console, accessed through `/console`.
* **`wandb-otel`**: the OpenTelemetry agent, which collects metrics and logs from resources at the Kubernetes layer for display in the administration console.
* **`wandb-prometheus`**: the Prometheus server, which captures metrics from various components for display in the administration console.
* **`wandb-parquet`**: a backend microservice separate from the `wandb-app` pod that exports database data to object storage in Parquet format.
* **`wandb-weave`**: another backend microservice that loads query tables in the UI and supports various core app features.
* **`wandb-weave-trace`**: a framework for tracking, experimenting with, evaluating, deploying, and improving LLM-based applications. The framework is accessed through the `wandb-app` pod.

### How to get the W\&B Operator Console password

See [Access the W\&B management console](#access-the-wb-management-console).

### How to access the W\&B Operator Console if Ingress doesn't work

Execute the following command on a host that can reach the Kubernetes cluster:

```console theme={null}
kubectl port-forward svc/wandb-console 8082
```

Access the console in the browser with `https://localhost:8082/` console.

For how to get the password (Option 2), see [Access the W\&B management console](#access-the-wb-management-console).

### How to view W\&B Server logs

The application pod is named **wandb-app-xxx**.

```console theme={null}
kubectl get pods
kubectl logs wandb-XXXXX-XXXXX
```

### How to identify the Kubernetes ingress class

You can get the ingress class installed in your cluster by running

```console theme={null}
kubectl get ingressclass
```


# Rate limits
Source: https://docs.wandb.ai/platform/hosting/self-managed/rate-limits

Optional rate limits on Self-Managed instances for stability

You can configure rate limits on your W\&B Self-Managed instance to maintain stability and prevent one user or workload from affecting others. These limits are optional. If you don't set them, the instance doesn't enforce any limits.

## Default limits and notification

The following default limits help maintain platform stability:

| Limit                            | Default | Scope   |
| -------------------------------- | ------- | ------- |
| Filestream requests per second   | 500     | Project |
| Filestream ingestion per second  | 120 MB  | Project |
| Filestream requests per second   | 2       | Run     |
| Run creation requests per second | 80      | Project |

When a limit is exceeded, the W\&B SDK returns HTTP response `429`, and the message `HTTP 429: rate limited exceeded` appears in the SDK logs.

* Filesystem rate limits never cause logging to crash or fail. When the SDK receives a `429` response on a filestream request, it will back off and retry the rate-limited request as-is, while subsequent updates accumulate.

* Run creation rate limits block further training.

```
HTTP 429: rate limit exceeded
```

W\&B recommends these defaults for typical production workloads. If your workloads consistently exceed these limits, you can adjust them (or leave them unset) according to your instance size and shared usage. For configuration details, see [Deploy W\&B with Kubernetes Operator](/platform/hosting/self-managed/operator) or contact [W\&B support](mailto:support@wandb.com).


# Reference architecture
Source: https://docs.wandb.ai/platform/hosting/self-managed/ref-arch

Review the reference architecture for self-managed W&B deployments covering Kubernetes, MySQL, object storage, and networking.

This page describes a reference architecture for a W\&B deployment and outlines the recommended infrastructure and resources to support a production deployment of the platform. Use it as a planning guide to size, provision, and integrate the components required for a reliable Self-Managed installation.

This page is intended for platform engineers, site reliability engineers, and infrastructure administrators who deploy and operate W\&B on their own infrastructure.

Depending on your chosen deployment environment for W\&B, different services can help to enhance the resiliency of your deployment.

For instance, major cloud providers offer managed database services that help to reduce the complexity of database configuration, maintenance, high availability, and resilience.

This reference architecture addresses common deployment scenarios and shows how you can integrate your W\&B deployment with cloud vendor services for performance and reliability.

## Before you start

Running any application in production comes with its own set of challenges, and W\&B is no exception. Although W\&B aims to streamline the process, complexities may arise depending on your architecture and design decisions. Typically, managing a production deployment involves overseeing components including hardware, operating systems, networking, storage, security, the W\&B platform itself, and other dependencies. This responsibility extends to both the initial setup of the environment and its ongoing maintenance.

Consider carefully whether a Self-Managed approach with W\&B is suitable for your team and your requirements.

A strong understanding of how to run and maintain production-grade application is an important prerequisite before you deploy Self-Managed W\&B. If your team needs assistance, the W\&B Professional Services team and partners offer support for implementation and optimization.

To learn more about managed solutions for running W\&B instead of managing it yourself, refer to [W\&B Multi-tenant Cloud](/platform/hosting/hosting-options/multi_tenant_cloud) and [W\&B Dedicated Cloud](/platform/hosting/hosting-options/dedicated-cloud).

## Infrastructure

A W\&B deployment consists of an application layer and a storage layer. The following diagram shows how these layers fit together, and the subsections that follow describe each one.

<Frame>
  <img alt="W&B infrastructure diagram" />
</Frame>

### Application layer

The application layer consists of a multi-node Kubernetes cluster, with resilience against node failures. The Kubernetes cluster runs and maintains the W\&B pods.

### Storage layer

The storage layer consists of a MySQL database and object storage. The MySQL database stores metadata and the object storage stores artifacts such as models and datasets.

## Infrastructure requirements

The following sections detail requirements for a W\&B deployment, including Kubernetes cluster details, MySQL, Redis, object storage, software versions, networking, DNS, load balancer and ingress, SSL/TLS, and supported CPU architectures. Confirm that your environment meets each of these requirements before you begin a deployment.

### Kubernetes

W\&B deploys the W\&B Server application as a [Kubernetes Operator](/platform/hosting/self-managed/operator) that deploys multiple pods. For this reason, W\&B requires a Kubernetes cluster with:

* A fully configured and functioning ingress controller.
* The capability to provision Persistent Volumes.

W\&B supports deployment on [OpenShift Kubernetes clusters](https://www.redhat.com/en/technologies/cloud-computing/openshift) in cloud, on-premises, and air-gapped environments. For specific configuration instructions, see the [OpenShift section](/platform/hosting/self-managed/operator#openshift-kubernetes-clusters) in the Operator guide.

### MySQL

W\&B stores metadata in a MySQL database. The database's performance and storage requirements depend on the shapes of the model parameters and related metadata. For example, the database grows in size as you track more training runs, and load on the database increases based on queries in run tables, user workspaces, and reports.

**W\&B strongly recommends using managed database services** (such as AWS RDS Aurora MySQL, Google Cloud SQL for MySQL, or Azure Database for MySQL) for production deployments. Managed services provide automated backups, monitoring, high availability, and patching, and reduce operational complexity. See the [Cloud provider instance recommendations](#cloud-provider-instance-recommendations) section for specific service recommendations.

If you choose to deploy a self-managed MySQL database, consider the following:

* **Backups**: Periodically back up the database to a separate facility. W\&B recommends daily backups with at least 1 week of retention.
* **Performance**: The database requires fast storage hardware, such as SSD or accelerated NAS.
* **Monitoring**: The database requires adequate CPU resources. Monitor the database server's CPU load. If CPU usage is sustained at > 90% of the system for more than 5 minutes, consider adding CPU capacity.
* **Availability**: To meet your availability and durability requirements, W\&B recommends configuring a hot standby deployment on a separate machine. The standby streams all updates in real time from the primary deployment and is ready to fail over if the primary server crashes, becomes corrupted, or experiences sustained downtime.

#### MySQL topology

For production, a managed MySQL service is the simplest path to high availability because the cloud provider handles failover, backups, and patching. Use the provider's high availability option, for example, Aurora Multi-AZ on AWS.

If you run self-managed MySQL, use a primary database with a hot standby that receives a real-time replication stream and can take over on failure. W\&B doesn't support a multi-primary topology or read-only replicas for the application database.

#### MySQL database creation

For instructions to manually create the MySQL database and user, see the [bare-metal guide MySQL database section](/platform/hosting/self-managed/operator#mysql-database).

#### MySQL configuration parameters

These parameters tune MySQL for the write patterns and schema changes that W\&B performs at scale. If you're running your own MySQL instance, configure MySQL with these settings:

```ini theme={null}
binlog_format = 'ROW'
binlog_row_image = 'MINIMAL'
innodb_flush_log_at_trx_commit = 1
innodb_online_alter_log_max_size = 268435456
max_prepared_stmt_count = 1048576
sort_buffer_size = '67108864'
sync_binlog = 1
```

W\&B has validated these settings for performance and reliability.

### Redis

W\&B depends on a single-node Redis 7.x deployment that W\&B components use for job queuing and data caching. For convenience during testing and development of proofs of concept, W\&B Self-Managed includes a local Redis deployment that isn't appropriate for production deployments.

W\&B can connect to a Redis instance in the following environments:

* [AWS Elasticache](https://aws.amazon.com/elasticache/).
* [Google Cloud Memory Store](https://cloud.google.com/memorystore?hl=en).
* [Azure Cache for Redis](https://azure.microsoft.com/en-us/products/cache).
* Redis deployment hosted in your cloud or on-premises infrastructure.

### Object storage

W\&B requires object storage with pre-signed URL and CORS support, deployed in one of:

* [CoreWeave AI Object Storage](https://docs.coreweave.com/products/storage/object-storage) is an S3-compatible object storage service optimized for AI workloads.
* [Amazon S3](https://aws.amazon.com/s3/) is an object storage service that provides scalability, data availability, security, and performance.
* [Google Cloud Storage](https://cloud.google.com/storage) is a managed service for storing unstructured data at scale.
* [Azure Blob Storage](https://azure.microsoft.com/en-us/products/storage/blobs) is a cloud-based object storage solution for storing unstructured data like text, binary data, images, videos, and logs.
* S3-compatible storage such as [MinIO Enterprise (AIStor)](https://www.min.io/product/aistor), [NetApp StorageGRID](https://www.netapp.com/data-storage/storagegrid/), or other enterprise-grade solutions hosted in your cloud or on-premises infrastructure.

### Versions

| Software   | Minimum version                                                                                                                 |
| ---------- | ------------------------------------------------------------------------------------------------------------------------------- |
| Kubernetes | v1.34 or newer ([Supported Kubernetes versions](https://kubernetes.io/releases/patch-releases/))                                |
| Helm       | v3.x                                                                                                                            |
| MySQL      | v8.0.x is required, v8.0.32 or newer; v8.0.44 or newer is recommended.<br />Aurora MySQL 3.x releases, must be v3.05.2 or newer |
| Redis      | v7.x                                                                                                                            |

### Networking

For a networked deployment, egress to these endpoints is required during *both* installation and runtime:

* [https://deploy.wandb.ai](https://deploy.wandb.ai)
* [https://charts.wandb.ai](https://charts.wandb.ai)
* [https://quay.io](https://quay.io) (used for Prometheus images)

<Note>
  Additional container registries may be required depending on your deployment configuration:

  * `https://gcr.io` is needed when deploying Bufstream and etcd for Weave online evaluations.
</Note>

To learn about air-gapped deployments, refer to [Kubernetes operator for air-gapped instances](/platform/hosting/self-managed/on-premises-deployments/kubernetes-airgapped).

Access to W\&B and to the object storage is required for the training infrastructure and for each system that tracks the needs of experiments.

### DNS

The fully qualified domain name (FQDN) of the W\&B deployment must resolve to the IP address of the ingress or load balancer using an `A` record.

### Load balancer and ingress

The W\&B Kubernetes Operator can expose services using a Kubernetes ingress controller, which routes to service endpoints based on URL paths with different ports. The ingress controller must be accessible by all machines that execute machine learning payloads or access the service through web browsers.

#### Ingress controller requirements

Your Kubernetes cluster must have an `IngressClass` available. Common ingress controller options include:

* [Nginx Ingress Controller](https://kubernetes.github.io/ingress-nginx/).
* [Istio](https://istio.io).
* [Traefik](https://traefik.io/).
* Cloud provider ingress controllers (AWS ALB, GCP Ingress, and Azure Application Gateway).

#### W\&B service routing

The W\&B Operator routes requests automatically to multiple backend services based on path:

| Path        | Service             | Default port | Purpose                            |
| ----------- | ------------------- | ------------ | ---------------------------------- |
| `/`         | `wandb-app`         | 8080         | Main web application UI            |
| `/api`      | `wandb-api`         | 8081         | API service                        |
| `/graphql`  | `wandb-api`         | 8081         | GraphQL API endpoint               |
| `/graphql2` | `wandb-api`         | 8081         | GraphQL API v2 endpoint            |
| `/console`  | `wandb-console`     | 8082         | System Console                     |
| `/traces`   | `wandb-weave-trace` | 8722         | Weave tracing service (if enabled) |

#### Example ingress configuration

The following shows an example ingress resource created by the W\&B Operator:

```yaml theme={null}
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: wandb
  namespace: wandb
  annotations:
    nginx.ingress.kubernetes.io/proxy-body-size: "0"
spec:
  ingressClassName: nginx
  rules:
  - host: wandb.example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: wandb-app
            port:
              number: 8080
      - path: /api
        pathType: Prefix
        backend:
          service:
            name: wandb-api
            port:
              number: 8081
      - path: /graphql
        pathType: Prefix
        backend:
          service:
            name: wandb-api
            port:
              number: 8081
      - path: /graphql2
        pathType: Prefix
        backend:
          service:
            name: wandb-api
            port:
              number: 8081
      - path: /console
        pathType: Prefix
        backend:
          service:
            name: wandb-console
            port:
              number: 8082
  tls:
  - hosts:
    - wandb.example.com
    secretName: wandb-tls
```

<Note>
  The W\&B Operator creates and manages the ingress configuration automatically. You typically don't need to create ingress resources manually. Make sure your cluster has a functioning ingress controller and the appropriate `IngressClass` configured.
</Note>

### SSL/TLS

W\&B requires a valid signed SSL/TLS certificate for secure communication between clients and the server. SSL/TLS termination must occur on the ingress/load balancer. The W\&B Server application does not terminate SSL or TLS connections.

**Important**: W\&B does not support self-signed certificates and custom CAs. Using self-signed certificates will cause challenges for users and is not supported.

If possible, using a service like [Let's Encrypt](https://letsencrypt.org) is a great way to provide trusted certificates to your load balancer. Services like Caddy and Cloudflare manage SSL for you.

If your security policies require SSL communication within your trusted networks, consider using a tool like Istio and [side car containers](https://istio.io/latest/docs/reference/config/networking/sidecar/).

### Supported CPU architectures

W\&B runs on Intel and AMD 64-bit architecture. ARM isn't supported.

## Deployment method

After your infrastructure meets the preceding requirements, choose how to install W\&B and provision the underlying resources. The following sections describe the recommended deployment method and the recommended approach for infrastructure provisioning.

### W\&B Kubernetes Operator with Helm

The recommended installation method for W\&B Self-Managed uses the **W\&B Kubernetes Operator**, deployed through Helm. This approach provides:

* Automated updates and management of W\&B components.
* Simplified configuration and deployment.
* Support for all deployment scenarios (cloud, on-premises, and air-gapped).

For detailed installation instructions, see:

* [Deploy W\&B Platform On-premises](/platform/hosting/self-managed/operator) - Primary installation guide.
* [Kubernetes operator for air-gapped instances](/platform/hosting/self-managed/on-premises-deployments/kubernetes-airgapped) - For disconnected environments.

### Infrastructure provisioning

Terraform is the recommended way to provision infrastructure for W\&B production deployments. With Terraform, you define the required resources, their references to other resources, and their dependencies. W\&B provides Terraform modules for the major cloud providers. For details, refer to [Deploy W\&B Server within Self-Managed cloud accounts](/platform/hosting/hosting-options/self-managed#deploy-wb-server-within-Self-Managed-cloud-accounts).

## Sizing

Use the following guidelines as a starting point when planning a deployment. W\&B recommends that you monitor all components of a deployment closely and that you make adjustments based on observed usage patterns. Continue to monitor production deployments over time and make adjustments as needed to maintain performance.

When you plan capacity, you size two core components: a Kubernetes cluster for the W\&B Operator workload and a MySQL database for metadata. Recommendations vary by **environment** (Test/Dev or Production) and, for Kubernetes only, by **product mix** (Models only, Weave only, or Models and Weave). W\&B recommends starting with a minimum of 3 worker nodes for both Test/Dev and Production, and enabling cluster autoscaling in Production.

The following sections give per-node sizing recommendations for the Kubernetes cluster and the MySQL database.

### Kubernetes sizing

<Tabs>
  <Tab title="Models only">
    | Environment | CPU     | Memory | Disk   |
    | ----------- | ------- | ------ | ------ |
    | Test/Dev    | 2 cores | 16 GB  | 100 GB |
    | Production  | 8 cores | 64 GB  | 100 GB |

    Numbers are per Kubernetes worker node.
  </Tab>

  <Tab title="Weave only">
    | Environment | CPU      | Memory | Disk   |
    | ----------- | -------- | ------ | ------ |
    | Test/Dev    | 4 cores  | 32 GB  | 100 GB |
    | Production  | 12 cores | 96 GB  | 100 GB |

    Numbers are per Kubernetes worker node.
  </Tab>

  <Tab title="Models and Weave">
    | Environment | CPU      | Memory | Disk   |
    | ----------- | -------- | ------ | ------ |
    | Test/Dev    | 4 cores  | 32 GB  | 100 GB |
    | Production  | 16 cores | 128 GB | 100 GB |

    Numbers are per Kubernetes worker node.
  </Tab>
</Tabs>

### MySQL sizing

These recommendations don't vary by product mix. For topology and availability guidance, see [MySQL topology](#mysql-topology) under [MySQL](#mysql).

| Environment | CPU     | Memory | Disk   |
| ----------- | ------- | ------ | ------ |
| Test/Dev    | 2 cores | 16 GB  | 100 GB |
| Production  | 8 cores | 64 GB  | 500 GB |

Numbers are per MySQL node.

## Cloud provider instance recommendations

After you determine the per-node CPU, memory, and disk requirements from the preceding sizing tables, use the following recommendations to pick specific cloud provider instance types and managed services that meet those requirements. These recommendations apply to each node of a Self-Managed deployment of W\&B in cloud infrastructure.

<Tabs>
  <Tab title="AWS">
    **Recommended managed services**

    * **Kubernetes**: Amazon EKS
    * **MySQL**: Amazon RDS Aurora
    * **Object storage**: Amazon S3

    | Environment | K8s (Models only) | K8s (Weave only) | K8s (Models\&Weave) | MySQL          |
    | ----------- | ----------------- | ---------------- | ------------------- | -------------- |
    | Test/Dev    | r6i.large         | r6i.xlarge       | r6i.xlarge          | db.r6g.large   |
    | Production  | r6i.2xlarge       | r6i.4xlarge      | r6i.4xlarge         | db.r6g.2xlarge |
  </Tab>

  <Tab title="Google Cloud">
    **Recommended managed services**

    * **Kubernetes**: Google Kubernetes Engine (GKE)
    * **MySQL**: Google Cloud SQL for MySQL
    * **Object storage**: Google Cloud Storage (GCS)

    | Environment | K8s (Models only) | K8s (Weave only) | K8s (Models\&Weave) | MySQL           |
    | ----------- | ----------------- | ---------------- | ------------------- | --------------- |
    | Test/Dev    | n2-highmem-2      | n2-highmem-4     | n2-highmem-4        | db-n1-highmem-2 |
    | Production  | n2-highmem-8      | n2-highmem-16    | n2-highmem-16       | db-n1-highmem-8 |
  </Tab>

  <Tab title="Azure">
    **Recommended managed services**

    * **Kubernetes**: Azure Kubernetes Service (AKS)
    * **MySQL**: Azure Database for MySQL
    * **Object storage**: Azure Blob Storage

    | Environment | K8s (Models only) | K8s (Weave only)  | K8s (Models\&Weave) | MySQL                  |
    | ----------- | ----------------- | ----------------- | ------------------- | ---------------------- |
    | Test/Dev    | Standard\_E2\_v5  | Standard\_E4\_v5  | Standard\_E4\_v5    | MO\_Standard\_E2ds\_v4 |
    | Production  | Standard\_E8\_v5  | Standard\_E16\_v5 | Standard\_E16\_v5   | MO\_Standard\_E8ds\_v4 |
  </Tab>
</Tabs>


# Self-Managed infrastructure requirements
Source: https://docs.wandb.ai/platform/hosting/self-managed/requirements

Infrastructure and software requirements for W&B Self-Managed deployments

This page describes the infrastructure and software requirements for deploying W\&B Self-Managed. It's intended for platform and infrastructure engineers planning a Self-Managed installation. Review these requirements before you begin your deployment to confirm that your environment can support W\&B Server.

<Note>
  W\&B recommends fully managed deployment options such as [W\&B Multi-tenant Cloud](/platform/hosting/hosting-options/multi_tenant_cloud) or [W\&B Dedicated Cloud](/platform/hosting/hosting-options/dedicated-cloud) deployment types. W\&B fully managed services are straightforward and secure to use, and require minimal to no configuration.
</Note>

For complete architectural guidance, see the [reference architecture](/platform/hosting/self-managed/ref-arch/).

## Software version requirements

| Software   | Minimum version                                                                                                                 |
| ---------- | ------------------------------------------------------------------------------------------------------------------------------- |
| Kubernetes | v1.34 or newer ([Supported Kubernetes versions](https://kubernetes.io/releases/patch-releases/))                                |
| Helm       | v3.x                                                                                                                            |
| MySQL      | v8.0.x is required, v8.0.32 or newer; v8.0.44 or newer is recommended.<br />Aurora MySQL 3.x releases, must be v3.05.2 or newer |
| Redis      | v7.x                                                                                                                            |

## Hardware requirements

**CPU Architecture**: W\&B runs on Intel (x86) CPU architecture only. ARM is not supported.

**Sizing**: For CPU, memory, and disk sizing recommendations for Kubernetes nodes and MySQL, see the [Sizing section](/platform/hosting/self-managed/ref-arch/#sizing) in the reference architecture. Requirements vary based on whether you're running Models, Weave, or both.

For detailed sizing recommendations based on your use case (Models only, Weave only, or both), see the [reference architecture sizing section](/platform/hosting/self-managed/ref-arch/#sizing).

## Kubernetes

W\&B Server is deployed as a [Kubernetes Operator](/platform/hosting/self-managed/operator/) that manages multiple pods. Your Kubernetes cluster must meet the following requirements:

* **Version**: See the preceding [Software version requirements](#software-version-requirements) section.
* **Ingress controller**: A fully configured and functioning ingress controller (Nginx, Istio, Traefik, or cloud provider ingress).
* **Persistent volumes**: Capability to provision persistent volumes.
* **CPU architecture**: Intel or AMD 64-bit (ARM is not supported).

W\&B supports deployment on [OpenShift Kubernetes clusters](https://www.redhat.com/en/technologies/cloud-computing/openshift) in cloud, on-premises, and air-gapped environments. For specific configuration instructions, see the [OpenShift section](/platform/hosting/self-managed/operator/#openshift-kubernetes-clusters) in the Operator guide.

For complete Kubernetes requirements, including load balancer and ingress configuration, see the [reference architecture Kubernetes section](/platform/hosting/self-managed/ref-arch/#kubernetes).

## MySQL database

W\&B requires an external MySQL database.

For production, W\&B strongly recommends using managed database services:

* [AWS RDS Aurora MySQL](https://aws.amazon.com/rds/aurora/)
* [Google Cloud SQL for MySQL](https://cloud.google.com/sql/mysql)
* [Azure Database for MySQL](https://azure.microsoft.com/en-us/products/mysql/)

Managed database services provide automated backups, monitoring, high availability, patching, and reduce operational overhead.

See the [reference architecture](/platform/hosting/self-managed/ref-arch/#mysql) for complete MySQL requirements, including sizing recommendations and configuration parameters. For database creation SQL, see the [bare-metal guide](/platform/hosting/self-managed/operator/#mysql-database). For questions about your deployment's database configuration, contact [support](mailto:support@wandb.com) or your AISE.

W\&B recommends using managed database services such as AWS RDS Aurora MySQL, Google Cloud SQL for MySQL, or Azure Database for MySQL for production deployments. Managed services provide automated backups, monitoring, high availability, and patching, and they reduce operational complexity.

### MySQL configuration parameters

If you're running your own MySQL instance, configure MySQL with the following settings for compatibility with W\&B Server:

```ini theme={null}
binlog_format = 'ROW'
binlog_row_image = 'MINIMAL'
innodb_flush_log_at_trx_commit = 1
innodb_online_alter_log_max_size = 268435456
max_prepared_stmt_count = 1048576
sort_buffer_size = '67108864'
sync_binlog = 1
```

W\&B has validated these settings for performance and reliability.

### Database creation

If you aren't using a managed MySQL service that provisions the database automatically, follow these instructions to manually create the MySQL database and user that W\&B Server uses:

Create a database and a user with the following SQL commands. Replace `SOME_PASSWORD` with a secure password of your choice:

```sql theme={null}
CREATE USER 'wandb_local'@'%' IDENTIFIED BY 'SOME_PASSWORD';
CREATE DATABASE wandb_local CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;
GRANT ALL ON wandb_local.* TO 'wandb_local'@'%' WITH GRANT OPTION;
```

For additional considerations, including backups, performance, monitoring, and availability, see the [reference architecture MySQL section](/platform/hosting/self-managed/ref-arch/#mysql).

## Redis

W\&B Server uses Redis for caching and background job coordination.

W\&B depends on a single-node Redis 7.x deployment used by W\&B's components for job queuing and data caching. For convenience during testing and development of proofs of concept, W\&B Self-Managed includes a local Redis deployment that is not appropriate for production deployments.

For production deployments, W\&B can connect to a Redis instance in the following environments:

* [AWS Elasticache](https://aws.amazon.com/elasticache/)
* [Google Cloud Memory Store](https://cloud.google.com/memorystore?hl=en)
* [Azure Cache for Redis](https://azure.microsoft.com/en-us/products/cache)
* Redis deployment hosted in your cloud or on-premise infrastructure

W\&B can connect to a Redis instance in any of the following environments:

* [AWS Elasticache](https://aws.amazon.com/pm/elasticache/).
* [Google Cloud Memory Store](https://cloud.google.com/memorystore?hl=en).
* [Azure Cache for Redis](https://azure.microsoft.com/en-us/products/cache).
* Redis deployment hosted in your cloud or on-premises infrastructure.

## Object storage

W\&B Server requires an object storage bucket to store artifacts, media, and run data.

W\&B requires object storage with pre-signed URL and CORS support.

**Recommended storage providers:**

* [Amazon S3](https://aws.amazon.com/s3/): Object storage service offering industry-leading scalability, data availability, security, and performance.
* [Google Cloud Storage](https://cloud.google.com/storage): Managed service for storing unstructured data at scale.
* [Azure Blob Storage](https://azure.microsoft.com/en-us/products/storage/blobs): Cloud-based object storage solution for storing massive amounts of unstructured data.
* [CoreWeave AI Object Storage](https://docs.coreweave.com/products/storage/object-storage): High-performance, S3-compatible object storage service optimized for AI workloads.
* Enterprise S3-compatible storage: [MinIO Enterprise (AIStor)](https://www.min.io/product/aistor), [NetApp StorageGRID](https://www.netapp.com/data-storage/storagegrid/), or other enterprise-grade solutions

<Note>
  MinIO Open Source is in [maintenance mode](https://github.com/minio/minio) with no active development or pre-compiled binaries. For production deployments, W\&B recommends using managed object storage services or enterprise S3-compatible solutions such as MinIO Enterprise (AIStor).
</Note>

For detailed bucket provisioning instructions including IAM policies, CORS configuration, and access setup, see the [Bring Your Own Bucket (BYOB) guide](/platform/hosting/data-security/secure-storage-connector).

See the [reference architecture object storage section](/platform/hosting/self-managed/ref-arch/#object-storage) for complete requirements.

### Provision your storage bucket

Before configuring W\&B, provision your object storage bucket with proper IAM policies, CORS configuration, and access credentials.

**See the [Bring Your Own Bucket (BYOB) guide](/platform/hosting/data-security/secure-storage-connector) for detailed step-by-step provisioning instructions for:**

* Amazon S3 (including IAM policies and bucket policies)
* Google Cloud Storage (including PubSub notifications)
* Azure Blob Storage (including managed identities)
* CoreWeave AI Object Storage
* S3-compatible storage (MinIO Enterprise, NetApp StorageGRID, and other enterprise solutions)

### Configure W\&B to use your bucket

After you provision your bucket, you must configure W\&B to use it through the Operator's Helm values so that W\&B Server can read from and write to the bucket. See the [Operator object storage configuration section](/platform/hosting/self-managed/operator/#object-storage-bucket) for details.

## Networking

Networking configuration exposes W\&B Server to users and machine-learning workloads. The following sections describe DNS, load balancer, and ingress requirements.

For a networked deployment, egress to these endpoints is required during *both* installation and runtime:

* [https://deploy.wandb.ai](https://deploy.wandb.ai)
* [https://charts.wandb.ai](https://charts.wandb.ai)
* [https://quay.io](https://quay.io) (used for Prometheus images)

<Note>
  Additional container registries may be required depending on your deployment configuration:

  * `https://gcr.io` is needed when deploying Bufstream and etcd for Weave online evaluations.
</Note>

To learn about air-gapped deployments, refer to [Kubernetes operator for air-gapped instances](/platform/hosting/self-managed/on-premises-deployments/kubernetes-airgapped).

Access to W\&B and to the object storage is required for the training infrastructure and for each system that tracks the needs of experiments.

### DNS

The fully qualified domain name (FQDN) of the W\&B deployment must resolve to the IP address of the ingress or load balancer using an A record.

### Load balancer and ingress

The W\&B Kubernetes Operator exposes services using a Kubernetes ingress controller, which routes to service endpoints based on URL paths. The ingress controller must be accessible by all machines that execute machine-learning payloads or access the service through web browsers.

For detailed load balancer options, ingress controller requirements, and configuration examples, see the [reference architecture load balancer section](/platform/hosting/self-managed/ref-arch/#load-balancer-and-ingress).

## SSL/TLS

W\&B requires a valid signed SSL/TLS certificate for secure communication between clients and the server. SSL/TLS termination must occur on the ingress/load balancer. The W\&B Server application does not terminate SSL or TLS connections.

**Important**: W\&B does not support self-signed certificates and custom CAs. Using self-signed certificates will cause challenges for users and is not supported.

If possible, using a service like [Let's Encrypt](https://letsencrypt.org) is a great way to provide trusted certificates to your load balancer. Services like Caddy and Cloudflare manage SSL for you.

If your security policies require SSL communication within your trusted networks, consider using a tool like Istio and [side car containers](https://istio.io/latest/docs/reference/config/networking/sidecar/).

## License

All Self-Managed deployments require a valid W\&B Server license. Without a license, W\&B Server can't start.

You need a W\&B license to deploy W\&B Self-Managed.

1. If you do not already have a W\&B account, create one.
2. If you need an enterprise trial license with support for important security and other enterprise-friendly capabilities, [submit a request](https://wandb.ai/site/for-enterprise/self-hosted-trial) or reach out to your W\&B team.
3. Otherwise, open the [Deploy Manager](https://deploy.wandb.ai/deploy) to generate a free trial license. The URL redirects you to a **Get a License for W\&B Local** form. Provide the following information:
   * The owner of the license
   * The deployment type
   * A name and optional description for the instance
4. Click **Generate License Key**.

A page displays with an overview of your deployment along with the license associated with the instance.

## Next steps

After you confirm that your infrastructure meets these requirements, proceed to the deployment guide that matches your environment:

* **Cloud and on-premises deployments**: See [Deploy W\&B with Kubernetes Operator](/platform/hosting/self-managed/operator) for Helm and Terraform deployment options.
* **Air-gapped deployments**: See [Deploy on Air-Gapped Kubernetes](/platform/hosting/self-managed/on-premises-deployments/kubernetes-airgapped) for disconnected environments.
* **All deployment methods**: See [Deploy with Kubernetes Operator](/platform/hosting/self-managed/operator) for the core operator deployment guide.


# Update W&B license and version
Source: https://docs.wandb.ai/platform/hosting/server-upgrade-process

Guide for updating W&B version and license across different installation methods.

This guide shows W\&B Server administrators how to update the W\&B Server version and license key for an existing self-managed deployment. Keeping your server and license current ensures access to the latest features, fixes, and continued entitlement to use W\&B Server.

Update your W\&B Server version and license with the same method you used to install W\&B Server. The following table lists how to update your license and version based on different deployment methods.

| Release type                        | Description                                                                                                                                                                                                                                                                                             |
| ----------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| [Terraform](#update-with-terraform) | W\&B supports three public Terraform modules for cloud deployment: [AWS](https://registry.terraform.io/modules/wandb/wandb/aws/latest), [Google Cloud](https://registry.terraform.io/modules/wandb/wandb/google/latest), and [Azure](https://registry.terraform.io/modules/wandb/wandb/azurerm/latest). |
| [Helm](#update-with-helm)           | Use the [Helm Chart](https://github.com/wandb/helm-charts) to install W\&B into an existing Kubernetes cluster.                                                                                                                                                                                         |

## Update with Terraform

If you deployed W\&B Server with one of the W\&B-maintained Terraform modules, use Terraform to update both your license key and the W\&B version in place. The following table lists W\&B-managed Terraform modules by cloud platform.

| Cloud provider | Terraform module                                                                                 |
| -------------- | ------------------------------------------------------------------------------------------------ |
| AWS            | [AWS Terraform module](https://registry.terraform.io/modules/wandb/wandb/aws/latest)             |
| Google Cloud   | [Google Cloud Terraform module](https://registry.terraform.io/modules/wandb/wandb/google/latest) |
| Azure          | [Azure Terraform module](https://registry.terraform.io/modules/wandb/wandb/azurerm/latest)       |

1. Navigate to the W\&B-maintained Terraform module for your cloud provider. See the preceding table to find the Terraform module that matches your cloud provider.

2. In your Terraform configuration, update `wandb_version` and `license` in your Terraform `wandb_app` module configuration:

   ```hcl theme={null}
   module "wandb_app" {
       source  = "wandb/wandb/[CLOUD-SPECIFIC-MODULE]"
       version = "new_version"
       license       = "new_license_key" # Your new license key
       wandb_version = "new_wandb_version" # Desired W&B version
       ...
   }
   ```

3. Apply the Terraform configuration with `terraform plan` and `terraform apply`.
   ```bash theme={null}
   terraform init
   terraform apply
   ```

4. Optional: If you use a `terraform.tfvars` or other `.tfvars` file, update or create one with the new W\&B version and license key.
   ```bash theme={null}
   terraform plan -var-file="terraform.tfvars"
   ```
   From your Terraform workspace directory, apply the configuration:
   ```bash theme={null}
   terraform apply -var-file="terraform.tfvars"
   ```

After Terraform applies the change, your deployment runs the specified W\&B version and uses the updated license key.

## Update with Helm

<Warning>
  The `wandb` Helm chart is deprecated and no longer supported. It deployed a single pod and has been replaced by the [W\&B Kubernetes Operator](/platform/hosting/self-managed/operator). If you're still using this chart, follow the [migration guide](/platform/hosting/self-managed/operator#migrate-self-managed-instances-to-wb-operator) to move to the operator.
</Warning>

Two Helm-based update paths are available: update from your existing Helm values file, or set the new license and image tag directly on the upgrade command. The following sections describe each approach.

### Update W\&B with spec

Use this approach when you manage your Helm configuration in a tracked `*.yaml` values file.

1. Specify a new version by modifying the `image.tag` or `license` values, or both, in your Helm chart `*.yaml` configuration file:

   ```yaml theme={null}
   license: 'new_license'
   image:
     repository: wandb/local
     tag: 'new_version'
   ```

2. Update the Helm repository and upgrade the W\&B release using your values file:

   ```bash theme={null}
   helm repo update
   helm upgrade --namespace=wandb --create-namespace \
     --install wandb wandb/wandb --version ${chart_version} \
     -f ${wandb_install_spec.yaml}
   ```

### Update license and version directly

Use this approach to update the license and image tag without editing a values file, and reuse your existing Helm release configuration.

1. Set the new license key and image tag as environment variables:

   ```bash theme={null}
   export LICENSE='new_license'
   export TAG='new_version'
   ```

2. Upgrade your Helm release, merging the new values with the existing configuration:

   ```bash theme={null}
   helm repo update
   helm upgrade --namespace=wandb --create-namespace \
     --install wandb wandb/wandb --version ${chart_version} \
     --reuse-values --set license=$LICENSE --set image.tag=$TAG
   ```

For more information, see the [upgrade guide](https://github.com/wandb/helm-charts/blob/main/upgrade) in the public repository.

## Update with admin UI

Use the admin UI to rotate your license key from within the W\&B App, without changing your deployment configuration. This method only works for updating licenses that aren't set with an environment variable in the W\&B server container, typically in self-managed Docker installations. This method updates the license only. It doesn't change the running W\&B Server version.

1. Obtain a new license from the [W\&B Deployment Page](https://deploy.wandb.ai/), ensuring it matches the correct organization and deployment ID for the deployment you want to upgrade.
2. Access the **License** page in the W\&B App. Click **Settings** > **License** or browse to `HOST_URL/console/settings/license`, where `HOST_URL` is your W\&B Server host URL.
3. Navigate to the license management section.
4. Enter the new license key and save your changes.


# Manage secrets
Source: https://docs.wandb.ai/platform/secrets

Overview of W&B secrets, how they work, and how to get started using them.

W\&B Secret Manager lets you securely and centrally store, manage, and inject *secrets*, which are sensitive strings such as access tokens, bearer tokens, API keys, or passwords. W\&B features can read team secret values, which removes the need to paste them or store them in code, training scripts, or plain-text automation configuration. This page is for W\&B Admins who need to create, rotate, delete, or manage access to team secrets used by webhook automations, Weave Playground, sandboxes, and LLM evaluation jobs.

Secrets live in each team's Secret Manager, in the **Team secrets** section of the [team settings](/platform/app/settings-page/teams/).

<Note>
  * Only W\&B Admins can create, edit, or delete a secret.
  * Secrets are included as a core part of W\&B, including in [W\&B Server deployments](/platform/hosting/) that you host in Azure, Google Cloud, or AWS. Connect with your W\&B account team to discuss how you can use secrets in W\&B if you use a different deployment type.
  * In W\&B Server, you are responsible for configuring security measures that satisfy your security needs.

    * W\&B strongly recommends that you store secrets in a W\&B instance of a cloud provider's secrets manager from AWS, Google Cloud, or Azure, which include advanced security capabilities.

    * W\&B recommends against using a Kubernetes cluster as the backend of your secrets store. Use a cluster only if you can't use a W\&B instance of a cloud secrets manager (AWS, Google Cloud, or Azure) and you understand how to prevent the security vulnerabilities that can occur.
</Note>

## Where team secrets are used

You can use team secrets in W\&B in multiple contexts. After you [add a secret](#add-a-secret), a feature like W\&B Automations can access the secret by name.

* **Webhook automations**: When an automation sends an HTTP request to a [webhook](/models/automations/create-automations/webhook/), you can attach team secrets for authentication headers and for values referenced in the payload. You can scope automations to a [project](/models/automations/automation-events/#project) or a [Registry](/models/automations/automation-events/#registry). Registry-scoped automations that call a webhook use the same team webhooks and team secrets as project-scoped webhook automations.
* **Weave Playground**: Supply provider credentials as named team secrets. See [Add provider credentials and information](/weave/guides/tools/playground#add-provider-credentials-and-information).
* **Sandboxes**: Securely provide team secrets to your sandboxes to make them available as environment variables. See [Secrets in sandboxes](/sandboxes/secrets).
* **LLM evaluation jobs**: Some benchmarks need API keys or tokens stored as team secrets. See the [Evaluation benchmark catalog](/models/launch/evaluations).

## Add a secret

Add a secret when you want to make a sensitive value available to W\&B features without exposing it in code or configuration. After you complete these steps, the secret is available by name to the team features described in [Where team secrets are used](#where-team-secrets-are-used).

To add a secret:

1. If an external service gives you a token or API key, obtain that value through that service's normal flow. If necessary, save the sensitive string securely, such as in a password manager, before you paste it into W\&B Secret Manager. Saving a backup matters because, after creation, W\&B no longer reveals the secret's value.
2. Sign in to W\&B and go to the team's **Settings** page.
3. In the **Team Secrets** section, click **New secret**.
4. Provide a name for the secret, using letters, numbers, and underscores (`_`).
5. Paste the sensitive string into the **Secret** field.
6. Click **Add secret**.

When you configure a webhook for an automation, select which team secrets the webhook may use. For field names, access tokens, and payload variables, see [Create a webhook automation](/models/automations/create-automations/webhook).

<Note>
  After you create a secret, you can access that secret in a [webhook automation's payload](/models/automations/create-automations/webhook/) using the format `${SECRET_NAME}`.
</Note>

## Rotate a secret

Rotate a secret when its value changes (for example, when an upstream credential is regenerated or when you suspect the existing value has been compromised). Because W\&B doesn't reveal a secret's current value after creation, rotation is also the way to replace a value you no longer have a copy of.

To rotate a secret and update its value:

1. Click the pencil icon in the secret's row to open the secret's details.
2. Set **Secret** to the new value. Optionally, click **Reveal secret** to verify the new value.
3. Click **Add secret**. The secret's value updates and no longer resolves to the previous value.

<Note>
  After a secret is created or updated, you can no longer reveal its current value. Instead, rotate the secret to a new value.
</Note>

Rotating or replacing a secret can affect every feature that still expects the old value. Before you rely on the new value everywhere, update [webhook automations](/models/automations/create-automations/webhook), [sandboxes](/sandboxes/secrets) that inject the secret, [evaluation jobs](/models/launch/evaluations), [Weave Playground](/weave/guides/tools/playground#add-provider-credentials-and-information), or other consumers.

## Delete a secret

Delete a secret when no team feature uses it. Because deletion is immediate and permanent, confirm that no automations, sandboxes, or other consumers still reference the secret before you proceed (see [Manage access to secrets](#manage-access-to-secrets)).

To delete a secret:

1. Click the trash icon in the secret's row.
2. Read the confirmation dialog, then click **Delete**. The secret is deleted immediately and permanently.

## Manage access to secrets

You can reference a team's secrets by name in [webhook automations](/models/automations/create-automations/webhook), [Weave Playground](/weave/guides/tools/playground#add-provider-credentials-and-information), [sandboxes](/sandboxes/secrets), [LLM evaluation jobs](/models/launch/evaluations), and other team-scoped features that select secrets by name.

Before you remove a secret, update or remove every automation, job, sandbox configuration, or Playground flow that uses it so they don't stop working.


# W&B Pricing
Source: https://docs.wandb.ai/pricing





# Serverless Inference
Source: https://docs.wandb.ai/product-inference





# W&B Models
Source: https://docs.wandb.ai/product-models





# Serverless Sandboxes
Source: https://docs.wandb.ai/product-sandboxes





# Serverless Training
Source: https://docs.wandb.ai/product-serverless-training





# W&B Weave
Source: https://docs.wandb.ai/product-weave





# W&B Platform Security
Source: https://docs.wandb.ai/security





# Console logs
Source: https://docs.wandb.ai/models/app/console-logs

View and debug console log messages including info, warnings, and errors from your W&B experiment runs.

When you run an experiment, you might notice messages printed to your console. W\&B captures console logs and displays them in the W\&B App. Use these messages to debug and monitor the behavior of your experiment.

The following sections describe how to view, configure, search, filter, download, and copy console logs for your runs.

## View console logs

Access console logs for a run in the W\&B App to inspect messages produced during the run.

1. Navigate to your project in the W\&B App.
2. Select a run within the **Runs** table.
3. Click the **Logs** tab in the project sidebar.

<Note>
  W\&B stores a maximum of 100,000 lines of your logs for a run. In the W\&B App, a maximum of 10,000 lines of your logs display at once. To view all stored lines, scroll through the logs to display older lines.
</Note>

## Types of console logs

W\&B captures three types of console logs and adds a prefix to indicate each log's severity. The prefix helps you scan logs and identify the messages most relevant to debugging. The following table summarizes each type, ordered from most to least severe.

| Severity | Prefix    | Description                                                             | Example                                                         |
| -------- | --------- | ----------------------------------------------------------------------- | --------------------------------------------------------------- |
| Error    | `ERROR`   | Serious issues that might prevent the run from completing successfully. | `ERROR Failed to save notebook.`                                |
| Warning  | `WARNING` | Potential issues that don't stop execution.                             | `WARNING Found .wandb file, not streaming tensorboard metrics.` |
| Info     | `wandb:`  | Updates about the run's progress and status.                            | `wandb: Starting Run: abc123`                                   |

## Console log settings

To control which types of console output W\&B captures and displays, pass a `wandb.Settings` object to `wandb.init()` when you initialize a run. The relevant parameters are `show_errors`, `show_warnings`, `show_info`, and `silent`. For details on each parameter and its default value, see the [`wandb.Settings` reference](/models/ref/python/experiments/settings).

The following example shows how to configure these settings:

```python theme={null}
import wandb

settings = wandb.Settings(
    show_errors=True,  # Show error messages in the W&B App
    silent=False,      # Disable all W&B console output
    show_warnings=True # Show warning messages in the W&B App
)

with wandb.init(settings=settings) as run:
    # Your training code here
    run.log({"accuracy": 0.95})
```

## Custom logging

If you already have your own logging setup, you can continue to use it alongside W\&B. W\&B captures console logs from your application, but it doesn't interfere with your own logging setup. You can use Python's built-in `print()` function or the `logging` module to log messages.

```python theme={null}
import wandb

with wandb.init(project="my-project") as run:
    for i in range(100, 1000, 100):
        # Logs to W&B and prints to console
        run.log({"epoch": i, "loss": 0.1 * i})
        print(f"epoch: {i} loss: {0.1 * i}")
```

The console logs look similar to the following:

```text theme={null}
1 epoch:  100 loss: 1.3191105127334595
2 epoch:  200 loss: 0.8664389848709106
3 epoch:  300 loss: 0.6157898902893066
4 epoch:  400 loss: 0.4961796700954437
5 epoch:  500 loss: 0.42592573165893555
6 epoch:  600 loss: 0.3771176040172577
7 epoch:  700 loss: 0.3393910825252533
8 epoch:  800 loss: 0.3082585036754608
9 epoch:  900 loss: 0.28154927492141724
```

## Timestamps

W\&B automatically adds timestamps to each console log entry. This lets you track when each log message was generated.

To show or hide timestamps in the console logs, select the **Timestamp visible** drop-down list on the console logs page.

## Search console logs

To quickly locate relevant entries, use the search bar on the console logs page to filter logs by keywords. You can search for specific terms, labels, or error messages.

## Filter with custom labels

<Warning>
  Parameters prefixed by `x_` (such as `x_label`) are in public preview. Create a [GitHub issue in the W\&B repository](https://github.com/wandb/wandb) to provide feedback.
</Warning>

You can filter console logs based on the labels you pass as arguments for `x_label` in `wandb.Settings`. Enter the label in the search bar on the console logs page.

```python theme={null}
import wandb

# Initialize a run in the primary node
with wandb.init(
    entity="[ENTITY-NAME]",
    project="[PROJECT-NAME]",
    settings=wandb.Settings(
        x_label="[CUSTOM-LABEL]"  # (Optional) Custom label for filtering logs
    )
) as run:
    # Your code here
```

## Download console logs

To save logs locally for offline analysis or sharing, download console logs for a run in the W\&B App:

1. Navigate to your project in the W\&B App.
2. Select a run within the **Runs** table.
3. Click the **Logs** tab in the project sidebar.
4. Click the download button on the console logs page.

## Copy console logs

To paste logs into another tool or message, copy console logs for a run in the W\&B App:

1. Navigate to your project in the W\&B App.
2. Select a run within the **Runs** table.
3. Click the **Logs** tab in the project sidebar.
4. Click the copy button on the console logs page.


# Manage workspace, section, and panel settings
Source: https://docs.wandb.ai/models/app/features/cascade-settings

Manage workspace, section, and panel settings in the W&B App, including layout and line plot configuration options.

A workspace page has three different setting levels: workspaces, sections, and panels. [Workspace settings](#workspace-settings) apply to the entire workspace. [Section settings](#section-settings) apply to all panels within a section. [Panel settings](#panel-settings) apply to individual panels.

This page describes how to configure each level of settings so that you can control your workspace's layout, organize panels, and customize how data is displayed. The following sections describe each setting level in turn.

## Workspace settings

Workspace settings apply to all sections and all panels within those sections. You can edit two types of workspace settings: [Workspace layout](#workspace-layout-options) and [Line plots](#line-plots-options). **Workspace layouts** determine the structure of the workspace, while **Line plots** settings control the default settings for line plots in the workspace.

To edit settings that apply to the overall structure of this workspace:

1. Navigate to your project workspace.
2. Click the gear icon next to the **New report** button to view the workspace settings.
3. Choose **Workspace layout** to change the workspace's layout, or choose **Line plots** to configure default settings for line plots in the workspace.

<Frame>
  <img alt="Workspace settings gear icon" />
</Frame>

The workspace settings panel opens, where you can apply the changes described in the following sections.

<Note>
  After customizing your workspace, you can use *workspace templates* to create new workspaces with the same settings. Refer to [Workspace templates](/models/track/workspaces/#workspace-templates).
</Note>

### Workspace layout options

Configure a workspace's layout to define the overall structure of the workspace. This includes sectioning logic and panel organization.

<Frame>
  <img alt="Workspace layout options" />
</Frame>

The workspace layout options page shows whether the workspace generates panels automatically or manually. To adjust a workspace's panel generation mode, refer to [Panels](/models/app/features/panels/).

This table describes each workspace layout option.

| Workspace setting                     | Description                                                                                                                                             |
| ------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Hide empty sections during search** | Hide sections that don't contain any panels when searching for a panel.                                                                                 |
| **Sort panels alphabetically**        | Sort panels in your workspaces alphabetically.                                                                                                          |
| **Section organization**              | Remove all existing sections and panels and repopulate them with new section names. Groups the newly populated sections either by first or last prefix. |

<Note>
  W\&B suggests that you organize sections by grouping the first prefix rather than grouping by the last prefix. Grouping by the first prefix can result in fewer sections and better performance.
</Note>

### Line plots options

Set global defaults and custom rules for line plots in a workspace by modifying the **Line plots** workspace settings.

<Frame>
  <img alt="Line plot settings" />
</Frame>

You can edit two main settings within **Line plots** settings: **Data** and **Display preferences**. The **Data** tab contains the following settings:

| Line plot setting                | Description                                                                                                                                             |
| -------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **X axis**                       | The scale of the x-axis in line plots. The x-axis is set to **Step** by default. See the following table for the list of x-axis options.                |
| **Range**                        | Minimum and maximum settings to display for the x-axis.                                                                                                 |
| **Smoothing**                    | Change the smoothing on the line plot. For more information about smoothing, see [Smooth line plots](/models/app/features/panels/line-plot/smoothing/). |
| **Outliers**                     | Rescale to exclude outliers from the default plot min and max scale.                                                                                    |
| **Point aggregation method**     | Improve data visualization accuracy and performance. For more information, see [Point aggregation](/models/app/features/panels/line-plot/sampling/).    |
| **Max number of runs or groups** | Limit the number of runs or groups displayed on the line plot.                                                                                          |

Besides **Step**, the x-axis supports the following options:

| X axis option               | Description                                                                                                                                                                           |
| --------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Relative Time (Wall)**    | Timestamp since the process starts. For example, suppose you start a run and resume that run the next day. If you then log something, the recorded point is 24 hours.                 |
| **Relative Time (Process)** | Timestamp inside the running process. For example, suppose you start a run and let it continue for 10 seconds. The next day you resume that run. The point is recorded as 10 seconds. |
| **Wall Time**               | Minutes elapsed since the start of the first run on the graph.                                                                                                                        |
| **Step**                    | Increments each time you call `wandb.Run.log()`.                                                                                                                                      |

<Note>
  For information about how to edit an individual line plot, see [Edit line panel settings](/models/app/features/panels/line-plot/#edit-line-panel-settings) in Line plots.
</Note>

Within the **Display preferences** tab, you can toggle the following settings:

| Display preference                                       | Description                                            |
| -------------------------------------------------------- | ------------------------------------------------------ |
| **Remove legends from all panels**                       | Remove the panel's legend.                             |
| **Display colored run names in tooltips**                | Show the runs as colored text within the tooltip.      |
| **Only show highlighted run in companion chart tooltip** | Display only highlighted runs in chart tooltip.        |
| **Number of runs shown in tooltips**                     | Display the number of runs in the tooltip.             |
| **Display full run names on the primary chart tooltip**  | Display the full name of the run in the chart tooltip. |

## Section settings

Section settings apply to all panels within that section. Within a workspace section you can sort panels, rearrange panels, and rename the section name.

To modify section settings, select the **action (<Icon icon="ellipsis" />)** menu in the upper right corner of a section.

<Frame>
  <img alt="Section settings menu" />
</Frame>

From the dropdown, you can edit the following settings that apply to the entire section:

| Section setting      | Description                                                             |
| -------------------- | ----------------------------------------------------------------------- |
| **Rename a section** | Rename the name of the section.                                         |
| **Sort panels A-Z**  | Sort panels within a section alphabetically.                            |
| **Rearrange panels** | Select and drag a panel within a section to manually order your panels. |

The following animation demonstrates how to rearrange panels within a section:

<Frame>
  <img alt="Rearranging panels" />
</Frame>

<Note>
  Besides the settings described in the preceding table, you can also edit how sections appear in your workspaces such as **Add section below**, **Add section above**, **Delete section**, and **Add section to report**.
</Note>

## Panel settings

Customize an individual panel's settings to compare multiple lines on the same plot, calculate custom axes, rename labels, and more. To edit a panel's settings:

1. Hover over the panel you want to edit.
2. Select the pencil icon that appears.

<Frame>
  <img alt="Panel edit icon" />
</Frame>

3. Within the modal that appears, you can edit settings related to the panel's data, display preferences, and more.

<Frame>
  <img alt="Panel settings modal" />
</Frame>

Your changes apply to that panel only and take effect when you save them. For a complete list of settings you can apply to a panel, see [Edit line panel settings](/models/app/features/panels/line-plot/#edit-line-panel-settings).


# Custom charts overview
Source: https://docs.wandb.ai/models/app/features/custom-charts

Create custom charts in W&B projects with Vega visualizations

Custom charts let you visualize logged data exactly how you want, beyond the defaults W\&B provides. Use them when you need to plot relationships, distributions, or model evaluation metrics that the built-in panels don't cover, such as precision-recall curves, custom histograms, or overlays of multiple experiments.

Create custom charts in your W\&B project. Log arbitrary tables of data and visualize them exactly how you want. Control details of fonts, colors, and tooltips with [Vega](https://vega.github.io/vega/).

* Code: Try an example [Colab notebook](https://tiny.cc/custom-charts).
* Video: Watch a [walkthrough video](https://www.youtube.com/watch?v=3-N9OV6bkSM).
* Example: Quick Keras and Sklearn [demo notebook](https://colab.research.google.com/drive/1g-gNGokPWM2Qbc8p1Gofud0_5AoZdoSD?usp=sharing).

<Frame>
  <img alt="Supported charts from vega.github.io/vega" />
</Frame>

## How it works

1. **Log data**: From your script, log [config](/models/track/config/) and summary data.
2. **Customize the chart**: Pull in logged data with a [GraphQL](https://graphql.org) query. Visualize the results of your query with [Vega](https://vega.github.io/vega/), a visualization grammar.
3. **Log the chart**: Call your own preset from your script with `wandb.plot_table()`.

<Frame>
  <img alt="PR and ROC curves" />
</Frame>

If you don't see the expected data, the column you're looking for might not be logged in the selected runs. Save your chart, go back out to the runs table, and verify selected runs using the **eye** icon.

## Log charts from a script

The following sections describe two ways to log charts directly from your training script: built-in chart presets that cover common visualizations, and custom presets that let you reuse your own Vega specifications.

### Built-in presets

W\&B has several built-in chart presets that you can log directly from your script. These include line plots, scatter plots, bar charts, histograms, PR curves, and ROC curves.

<Tabs>
  <Tab title="Line plot">
    `wandb.plot.line()`

    Log a custom line plot, a list of connected and ordered points (x,y) on arbitrary axes x and y.

    ```python theme={null}
    with wandb.init() as run:
      data = [[x, y] for (x, y) in zip(x_values, y_values)]
      table = wandb.Table(data=data, columns=["x", "y"])
      run.log(
          {
              "my_custom_plot_id": wandb.plot.line(
                  table, "x", "y", title="Custom Y vs X Line Plot"
              )
          }
      )
    ```

    A line plot logs curves on any two dimensions. If you plot two lists of values against each other, the number of values in the lists must match exactly (for example, each point must have an x and a y).

    <Frame>
      <img alt="Custom line plot" />
    </Frame>

    [See an example report](https://wandb.ai/wandb/plots/reports/Custom-Line-Plots--VmlldzoyNjk5NTA) or [try an example Google Colab notebook](https://tiny.cc/custom-charts).
  </Tab>

  <Tab title="Scatter plot">
    `wandb.plot.scatter()`

    Log a custom scatter plot, a list of points (x, y) on a pair of arbitrary axes x and y.

    ```python theme={null}
    with wandb.init() as run:
      data = [[x, y] for (x, y) in zip(class_x_prediction_scores, class_y_prediction_scores)]
      table = wandb.Table(data=data, columns=["class_x", "class_y"])
      run.log({"my_custom_id": wandb.plot.scatter(table, "class_x", "class_y")})
    ```

    You can use this to log scatter points on any two dimensions. Note that if you're plotting two lists of values against each other, the number of values in the lists must match exactly (for example, each point must have an x and a y).

    <Frame>
      <img alt="Scatter plot" />
    </Frame>

    [See an example report](https://wandb.ai/wandb/plots/reports/Custom-Scatter-Plots--VmlldzoyNjk5NDQ) or [try an example Google Colab notebook](https://tiny.cc/custom-charts).
  </Tab>

  <Tab title="Bar chart">
    `wandb.plot.bar()`

    Log a custom bar chart (a list of labeled values as bars):

    ```python theme={null}
    with wandb.init() as run:
      data = [[label, val] for (label, val) in zip(labels, values)]
      table = wandb.Table(data=data, columns=["label", "value"])
      run.log(
          {
              "my_bar_chart_id": wandb.plot.bar(
                  table, "label", "value", title="Custom Bar Chart"
              )
          }
      )
    ```

    You can use this to log arbitrary bar charts. Note that the number of labels and values in the lists must match exactly (for example, each data point must have both).

    <Frame>
      <img alt="Demo bar plot" />
    </Frame>

    [See an example report](https://wandb.ai/wandb/plots/reports/Custom-Bar-Charts--VmlldzoyNzExNzk) or [try an example Google Colab notebook](https://tiny.cc/custom-charts).
  </Tab>

  <Tab title="Histogram">
    `wandb.plot.histogram()`

    Log a custom histogram (sort a list of values into bins by count or frequency of occurrence). For example, suppose you have a list of prediction confidence scores (`scores`) and want to visualize their distribution:

    ```python theme={null}
    with wandb.init() as run:
      data = [[s] for s in scores]
      table = wandb.Table(data=data, columns=["scores"])
      run.log({"my_histogram": wandb.plot.histogram(table, "scores", title=None)})
    ```

    You can use this to log arbitrary histograms. Note that `data` is a list of lists, intended to support a 2D array of rows and columns.

    <Frame>
      <img alt="Custom histogram" />
    </Frame>

    [See an example report](https://wandb.ai/wandb/plots/reports/Custom-Histograms--VmlldzoyNzE0NzM) or [try an example Google Colab notebook](https://tiny.cc/custom-charts).
  </Tab>

  <Tab title="PR curve">
    `wandb.plot.pr_curve()`

    Create a [Precision-Recall curve](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.precision_recall_curve.html#sklearn.metrics.precision_recall_curve) in one line:

    ```python theme={null}
    with wandb.init() as run:
      plot = wandb.plot.pr_curve(ground_truth, predictions, labels=None, classes_to_plot=None)

      run.log({"pr": plot})
    ```

    You can log this whenever your code has access to:

    * A model's predicted scores (`predictions`) on a set of examples.
    * The corresponding ground truth labels (`ground_truth`) for those examples.
    * `labels` (Optional): A list of the labels or class names (`labels=["cat", "dog", "bird"...]` if label index 0 means cat, 1 = dog, 2 = bird, and so on).
    * `classes_to_plot` (Optional): A subset (still in list format) of the labels to visualize in the plot.

    <Frame>
      <img alt="Precision-recall curves" />
    </Frame>

    [See an example report](https://wandb.ai/wandb/plots/reports/Plot-Precision-Recall-Curves--VmlldzoyNjk1ODY) or [try an example Google Colab notebook](https://colab.research.google.com/drive/1mS8ogA3LcZWOXchfJoMrboW3opY1A8BY?usp=sharing).
  </Tab>

  <Tab title="ROC curve">
    `wandb.plot.roc_curve()`

    Create an [ROC curve](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.roc_curve.html#sklearn.metrics.roc_curve) in one line:

    ```python theme={null}
    with wandb.init() as run:
      # ground_truth is a list of true labels, predictions is a list of predicted scores
      ground_truth = [0, 1, 0, 1, 0, 1]
      predictions = [0.1, 0.4, 0.35, 0.8, 0.7, 0.9]

      # Create the ROC curve plot
      # labels is an optional list of class names, classes_to_plot is an optional subset of those labels to visualize
      plot = wandb.plot.roc_curve(
          ground_truth, predictions, labels=None, classes_to_plot=None
      )

      run.log({"roc": plot})
    ```

    You can log this whenever your code has access to:

    * A model's predicted scores (`predictions`) on a set of examples.
    * The corresponding ground truth labels (`ground_truth`) for those examples.
    * `labels` (Optional): A list of the labels or class names (`labels=["cat", "dog", "bird"...]` if label index 0 means cat, 1 = dog, 2 = bird, and so on).
    * `classes_to_plot` (Optional): A subset (still in list format) of these labels to visualize on the plot.

    <Frame>
      <img alt="ROC curve" />
    </Frame>

    [See an example report](https://wandb.ai/wandb/plots/reports/Plot-ROC-Curves--VmlldzoyNjk3MDE) or [try an example Google Colab notebook](https://colab.research.google.com/drive/1_RMppCqsA8XInV_jhJz32NCZG6Z5t1RO?usp=sharing).
  </Tab>
</Tabs>

### Custom presets

Tweak a built-in preset, or create a new preset, then save the chart. Use the chart ID to log data to that custom preset directly from your script. [Try an example Google Colab notebook](https://tiny.cc/custom-charts).

```python theme={null}
# Create a table with the columns to plot
table = wandb.Table(data=data, columns=["step", "height"])

# Map from the table's columns to the chart's fields
fields = {"x": "step", "value": "height"}

# Use the table to populate the new custom chart preset
# To use your own saved chart preset, change the vega_spec_name
my_custom_chart = wandb.plot_table(
    vega_spec_name="carey/new_chart",
    data_table=table,
    fields=fields,
)
```

<Frame>
  <img alt="Custom chart presets" />
</Frame>

## Log data

Before you can visualize anything in a custom chart, your script must log the underlying data in a format the chart editor can query. You can log the following data types from your script and use them in a custom chart:

* **Config**: Initial settings of your experiment (your independent variables). This includes any named fields you've logged as keys to `wandb.Run.config` at the start of your training. For example, `wandb.Run.config.learning_rate = 0.0001`.
* **Summary**: Single values logged during training (your results or dependent variables). For example, `wandb.Run.log({"val_acc" : 0.8})`. If you write to this key multiple times during training through `wandb.Run.log()`, the summary is set to the final value of that key.
* **History**: The full time series of the logged scalar is available to the query through the `history` field.
* **summaryTable**: If you need to log a list of multiple values, use a `wandb.Table()` to save that data, then query it in your custom panel.
* **historyTable**: If you need to see the history data, then query `historyTable` in your custom chart panel. Each time you call `wandb.Table()` or log a custom chart, you're creating a new table in history for that step.

### Log a custom table

Use `wandb.Table()` to log your data as a 2D array. Typically, each row of this table represents one data point, and each column denotes the relevant fields or dimensions for each data point that you want to plot. As you configure a custom panel, the whole table is accessible through the named key passed to `wandb.Run.log()` (`custom_data_table` in the following example). The individual fields are accessible through the column names (`x`, `y`, and `z`). You can log tables at multiple time steps throughout your experiment. The maximum size of each table is 10,000 rows. [Try an example Google Colab notebook](https://tiny.cc/custom-charts).

```python theme={null}
with wandb.init() as run:
  # Logging a custom table of data
  my_custom_data = [[x1, y1, z1], [x2, y2, z2]]
  run.log(
      {"custom_data_table": wandb.Table(data=my_custom_data, columns=["x", "y", "z"])}
  )
```

## Customize the chart

After you log data, build a chart in the W\&B app by choosing which logged values to pull in and how to render them. Add a new custom chart to get started, then edit the query to select data from your visible runs. The query uses [GraphQL](https://graphql.org) to fetch data from the config, summary, and history fields in your runs.

### Build the GraphQL query

The custom chart editor runs a GraphQL query over the runs you selected in the project workspace or report. In the query editor, add the fields you need. You can pick from `config`, `summary`, `history`, `summaryTable`, and `historyTable` so you don't need to write the query from scratch for most cases.

Each source in the query maps to different logged data:

* **Config** pulls [run configuration](/models/track/config/) values (hyperparameters and other settings).
* **Summary** pulls [summary](/models/track/log/) values. By default, the summary for a key logged with `wandb.Run.log()` holds the last value written for that key. To use a different aggregate, call `wandb.Run.define_metric(..., summary=...)` with `"min"`, `"max"`, `"mean"`, `"best"`, or `"none"`. To set a value directly, assign `wandb.Run.summary["key"] = value`.
* **History** pulls scalar time series from run history (for example, `loss` or `accuracy` at each step). Use **history** when you need the full curve, not only the final number.
* **`summaryTable`** loads a [`wandb.Table`](/models/ref/python/data-types/table) from the run summary. Use it when the table you care about is stored as a single snapshot on the run (for example, one confusion matrix you log once at the end).
* **`historyTable`** loads a [`wandb.Table`](/models/ref/python/data-types/table) from run history. Each time you log a table with `wandb.Run.log()`, you add another step to run history that includes that table. Use **`historyTable`** when the table changes over time or when you want to enable the step selector in the custom chart editor (see [How do you show a step slider in a custom chart?](/support/models/articles/how-do-you-show-a-step-slider-in-a-custo)).

For **`summaryTable`** and **`historyTable`**, set **`tableKey`** to the dictionary key you used inside `wandb.Run.log()`, not to a column name inside the [`wandb.Table`](/models/ref/python/data-types/table).

The following examples cover common cases:

* Plot columns from a table you log at each step (for example, a PR curve): add **`historyTable`**, set **`tableKey`** to your logged key (for example, `pr_curve`), then map table columns in **Chart fields**. See the [custom charts tutorial](/models/app/features/custom-charts/walkthrough).
* Plot columns from a table that lives in a summary (for example, class scores for a composite histogram): add **`summaryTable`**, set **`tableKey`** to that key (the tutorial uses `class_scores`). See [Bonus: composite histograms](/models/app/features/custom-charts/walkthrough#bonus-composite-histograms).
* Plot a scalar metric over training steps: add the metric from **history**. If you only add it from **summary**, the chart shows a single value per run.

**Chart field names**

After the query runs, **Chart fields** lists columns you can bind into the Vega spec. Names often begin with `runSets_` and reflect the selected query fields. Choose them from the dropdowns next to each `${field:...}` placeholder instead of typing them manually.

If a column never appears, confirm the key exists on the selected runs, open the run page to verify how the data was logged, and check whether **`summaryTable`** or **`historyTable`** matches that logging pattern.

Custom charts use this GraphQL-based panel query. [Query panels](/models/app/features/panels/query-panels) use a different expression language and are documented separately.

<Frame>
  <img alt="Custom chart creation" />
</Frame>

### Custom visualizations

Select a **Chart** in the upper right corner to start with a default preset. Next, select **Chart fields** to map the data you're pulling in from the query to the corresponding fields in your chart.

The following image shows how to select a metric and then map it into the bar chart fields.

<Frame>
  <img alt="Creating a custom bar chart" />
</Frame>

### Edit Vega

Click **Edit** at the top of the panel to go into [Vega](https://vega.github.io/vega/) edit mode. Here you can define a [Vega specification](https://vega.github.io/vega/docs/specification/) that creates an interactive chart in the UI. You can change any aspect of the chart. For example, you can change the title, pick a different color scheme, or show curves as a series of points instead of as connected lines. You can also make changes to the data itself, such as using a Vega transform to bin an array of values into a histogram. The panel preview updates interactively, so you can see the effect of your changes as you edit the Vega spec or query. For more information, see the [Vega documentation and tutorials](https://vega.github.io/vega/).

**Field references**

To pull data into your chart from W\&B, add template strings of the form `"${field:<field-name>}"` anywhere in your Vega spec. This creates a dropdown in the **Chart fields** area on the right side, which you can use to select a query result column to map into Vega.

To set a default value for a field, use this syntax: `"${field:<field-name>:<placeholder text>}"`

### Save chart presets

Save a preset to reuse the same Vega definition across panels and projects instead of recreating it each time. Apply changes to a specific visualization panel with the button at the bottom of the modal. Alternatively, you can save the Vega spec to reuse it elsewhere in your project. To save the reusable chart definition, click **Save as** at the top of the Vega editor and give your preset a name.

## Reports and guides

The following reports show end-to-end examples and deeper explorations of custom charts in practice.

* [The W\&B Machine Learning Visualization IDE](https://wandb.ai/wandb/posts/reports/The-W-B-Machine-Learning-Visualization-IDE--VmlldzoyNjk3Nzg)
* [Visualizing NLP Attention Based Models](https://wandb.ai/kylegoyette/gradientsandtranslation2/reports/Visualizing-NLP-Attention-Based-Models-Using-Custom-Charts--VmlldzoyNjg2MjM)
* [Visualizing The Effect of Attention on Gradient Flow](https://wandb.ai/kylegoyette/gradientsandtranslation/reports/Visualizing-The-Effect-of-Attention-on-Gradient-Flow-Using-Custom-Charts--VmlldzoyNjg1NDg)
* [Logging arbitrary curves](https://wandb.ai/stacey/presets/reports/Logging-Arbitrary-Curves--VmlldzoyNzQyMzA)

## Common use cases

Custom charts are useful when the default panels can't represent what you need to show. For example:

* Customizing bar plots with error bars
* Showing model validation metrics that require custom x-y coordinates (like precision-recall curves)
* Overlaying data distributions from two different models or experiments as histograms
* Showing changes in a metric through snapshots at multiple points during training
* Creating a unique visualization not yet available in W\&B and sharing it with others


# Tutorial: Use custom charts
Source: https://docs.wandb.ai/models/app/features/custom-charts/walkthrough

Tutorial of using the custom charts feature in the W&B UI

Custom charts let you control both the data loaded into a panel and how it's visualized. This tutorial walks you through logging data, building a query, customizing the chart's Vega specification, and saving the result for reuse. It's intended for users who want to go beyond the default chart types and tailor visualizations to specific data.

By the end of this tutorial, you'll have a working custom chart in your project that you can save as a preset and a bonus composite histogram you can adapt to your own data.

## Log data to W\&B

Before you can visualize anything in a custom chart, your run needs to log the data you want to display. Use [wandb.Run.config](/models/track/config/) for single points set at the beginning of training, like hyperparameters. Use [wandb.Run.log()](/models/track/log/) for multiple points over time, and log custom 2D arrays with `wandb.Table()`. We recommend logging up to 10,000 data points per logged key.

```python theme={null}
with wandb.init() as run: 

  # Logging a custom table of data
  my_custom_data = [[x1, y1, z1], [x2, y2, z2]]
  run.log(
    {"custom_data_table": wandb.Table(data=my_custom_data, columns=["x", "y", "z"])}
  )
```

[Try a quick example notebook](https://bit.ly/custom-charts-colab) to log the data tables, and in the next step you'll set up custom charts. See what the resulting charts look like in the [live report](https://app.wandb.ai/demo-team/custom-charts/reports/Custom-Charts--VmlldzoyMTk5MDc).

With your data logged, you're ready to pull it into a custom chart panel.

## Create a query

A query tells the custom chart which logged data to load. After you've logged data to visualize, go to your project page and click the **`+`** button to add a new panel, then select **Custom Chart**. You can follow along in the [custom charts demo workspace](https://app.wandb.ai/demo-team/custom-charts).

<Frame>
  <img alt="Blank custom chart" />
</Frame>

### Add a query

1. Click `summary` and select `historyTable` to set up a new query pulling data from the run history.
2. Type in the key where you logged the `wandb.Table()`. In the previous code snippet, it was `custom_data_table`. In the [example notebook](https://bit.ly/custom-charts-colab), the keys are `pr_curve` and `roc_curve`. For more information about **`summaryTable`**, **`historyTable`**, and **`tableKey`**, see [Build the GraphQL query](/models/app/features/custom-charts#build-the-graphql-query).

### Set Vega fields

With the query in place, you can map your logged columns to the chart's visual encodings using the Vega fields dropdown menus:

<Frame>
  <img alt="Pulling in columns from the query results to set Vega fields" />
</Frame>

* **x-axis:** runSets\_historyTable\_r (recall)
* **y-axis:** runSets\_historyTable\_p (precision)
* **color:** runSets\_historyTable\_c (class label)

## Customize the chart

The default visualization is a good starting point, but you can edit the Vega spec to change the chart type, add titles, and refine the appearance. To switch from a scatter plot to a line plot, click **Edit** to change the Vega spec for this built-in chart. Follow along in the [custom charts demo workspace](https://app.wandb.ai/demo-team/custom-charts).

<Frame>
  <img alt="Custom chart selection" />
</Frame>

Update the Vega spec to customize the visualization:

* Add titles for the plot, legend, x-axis, and y-axis (set "title" for each field).
* Change the value of "mark" from "point" to "line".
* Remove the unused "size" field.

<Frame>
  <img alt="PR curve Vega spec" />
</Frame>

Save a preset to reuse the same chart configuration on other panels in the project. To save this as a preset that you can use elsewhere in this project, click **Save as** at the top of the page. Here's what the result looks like, along with an ROC curve.

<Frame>
  <img alt="PR curve chart" />
</Frame>

## Bonus: composite histograms

This section shows how to build a more advanced custom chart, a composite histogram that overlays two distributions in the same view, to demonstrate what's possible after you're comfortable editing the Vega spec.

Histograms can visualize numerical distributions to help you understand larger datasets. Composite histograms show multiple distributions across the same bins, letting you compare two or more metrics across different models or across different classes within your model. For a semantic segmentation model detecting objects in driving scenes, you might compare the effectiveness of optimizing for accuracy versus Intersection over Union (IoU), or you might want to know how well different models detect cars (large, common regions in the data) versus traffic signs (much smaller, less common regions). In the [demo Colab](https://bit.ly/custom-charts-colab), you can compare the confidence scores for two of the ten classes of living things.

<Frame>
  <img alt="Composite histogram" />
</Frame>

To create your own version of the custom composite histogram panel:

1. Create a new custom chart panel in your workspace or report (by adding a **Custom Chart** visualization). Click **Edit** in the top right to modify the Vega spec starting from any built-in panel type.
2. Replace that built-in Vega spec with the [starter code for a composite histogram in Vega](https://gist.github.com/staceysv/9bed36a2c0c2a427365991403611ce21). You can modify the main title, axis titles, input domain, and any other details directly in this Vega spec [using Vega syntax](https://vega.github.io/). For example, you can change the colors or add a third histogram.
3. Modify the query in the right panel to load the correct data from your wandb logs. Add the field `summaryTable` and set the corresponding `tableKey` to `class_scores` to fetch the `wandb.Table` logged by your run. This lets you populate the two histogram bin sets (`red_bins` and `blue_bins`) through the dropdown menus with the columns of the `wandb.Table` logged as `class_scores`. For example, you can choose the `animal` class prediction scores for the red bins and `plant` for the blue bins.
4. You can keep making changes to the Vega spec and query until the plot you see in the preview rendering matches what you want. After you're done, click **Save as** in the top and give your custom plot a name so you can reuse it. Then click **Apply from panel library** to finish your plot.

Here's an example result from a brief experiment: training on only 1,000 examples for one epoch yields a model that's confident that most images are not plants and uncertain about which images might be animals.

<Frame>
  <img alt="Chart configuration" />
</Frame>

<Frame>
  <img alt="Chart result" />
</Frame>


# Panels
Source: https://docs.wandb.ai/models/app/features/panels

Use and customize workspace panels to visualize your logged data

Use workspace panel visualizations to explore your [logged data](/models/ref/python/experiments/run.md/#method-runlog) by key, visualize the relationships between hyperparameters and output metrics, and more. This page describes how to choose a workspace mode, add and configure panels, organize them into sections, and share them with collaborators.

## Workspace modes

W\&B projects support two different workspace modes. The icon next to the workspace name shows its mode.

| Icon                                   | Workspace mode                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| -------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| <img alt="automated workspace icon" /> | **Automated workspaces** automatically generate panels for all keys logged in the project. Choose an automatic workspace:<ul><li>To get started quickly by visualizing all available data for the project.</li><li>For smaller projects that log fewer keys.</li><li>For more broad analysis.</li></ul>If you delete a panel from an automatic workspace, you can use [Quick add](#quick-add) to recreate it.                                                                   |
| <img alt="manual workspace icon" />    | **Manual workspaces** start as blank slates and display only the panels you intentionally add. Choose a manual workspace:<ul><li>When you care mainly about a fraction of the keys logged in the project.</li><li>For more focused analysis.</li><li>To improve the performance of a workspace by avoiding loading panels that are less useful to you.</li></ul>Use [Quick add](#quick-add) to quickly populate a manual workspace and its sections with useful visualizations. |

To change how a workspace generates panels, [reset the workspace](#reset-a-workspace).

<Note>
  **Undo changes to your workspace**

  To undo changes to your workspace, click the Undo button (arrow that points left) or press **Cmd+Z** (macOS) or **Ctrl+Z** (Windows or Linux).
</Note>

## Reset a workspace

Reset a workspace to change its mode or clear its panels to start fresh.

To reset a workspace:

1. At the top of the workspace, click the **action (<Icon icon="ellipsis" />)** menu.
2. Click **Reset workspace**.

## Configure the workspace layout

Workspace layout settings control how panels and sections are organized and displayed across the whole workspace.

To configure the workspace layout, click **Settings** near the top of the workspace, then click **Workspace layout**. The following settings are available:

* **Hide empty sections during search** (turned on by default)
* **Sort panels alphabetically** (turned off by default)
* **Section organization** (grouped by first prefix by default). To modify this setting:
  1. Click the padlock icon.
  2. Choose how to group panels within a section.

To configure defaults for the workspace's line plots, refer to [Line plots](/models/app/features/panels/line-plot/).

### Configure a section's layout

To configure the layout of a section, click its gear icon, then click **Display preferences**. The following settings are available:

* **Turn on or off colored run names in tooltips** (turned on by default)
* **Only show highlighted run in companion chart tooltips** (turned off by default)
* **Number of runs shown in tooltips** (a single run, all runs, or **Default**)
* **Display full run names on the primary chart tooltip** (turned off by default)

## View a panel in full-screen mode

Full-screen mode gives a panel more space, which is useful for closer inspection of your data. In full-screen mode, the run selector displays.

To view a panel in full-screen mode:

1. Hover over the panel.
2. Click the panel's **action (<Icon icon="ellipsis" />)** menu, then click the full-screen button, which looks like a viewfinder or an outline showing the four corners of a square.
   <Frame>
     <img alt="Full-screen panel" />
   </Frame>

When you [share the panel](#share-a-panel) while viewing it in full-screen mode, the resulting link opens in full-screen mode automatically.

When you work in full-screen mode, use the following tips:

* To return to a panel's workspace from full-screen mode, click the left-pointing arrow at the top of the page.
* To navigate through a section's panels without exiting full-screen mode, use either the **Previous** and **Next** buttons below the panel or the left and right arrow keys.
* To reclaim more space for the panel, minimize the run selector with **Cmd+.** (macOS) or **Ctrl+.** (Windows or Linux).
* When you view an image from a [media panel](/models/app/features/panels/media) in full-screen mode, keyboard shortcuts can zoom in or out, reset zoom, or zoom to fit. See [Keyboard shortcuts](/models/app/keyboard-shortcuts#media-panels).

See [Keyboard shortcuts](/models/app/keyboard-shortcuts) for other full-screen and panel shortcuts.

## Add panels

The following sections describe ways to add panels to your workspace.

### Add a panel manually

Add panels to your workspace one at a time, either globally or at the section level.

1. To add a panel globally, click **Add panels** in the control bar near the panel search field.
2. To add a panel directly to a section instead, click the section's **action (<Icon icon="ellipsis" />)** menu, then click **+ Add panels**.
3. Select the type of panel to add, such as a chart. The panel's configuration details appear, with defaults selected.
4. Optional: Customize the panel and its display preferences. Configuration options depend on the type of panel you select. To learn more about the options for each type of panel, refer to the relevant section, such as [Line plots](/models/app/features/panels/line-plot/) or [Bar plots](/models/app/features/panels/bar-plot/).
5. Click **Apply**.

<Frame>
  <img alt="Demo of adding a panel" />
</Frame>

### Quick add panels

Use **Quick add** to add a panel automatically for each key you select, either globally or at the section level.

<Note>
  For an automated workspace with no deleted panels, the **Quick add** option isn't visible because the workspace already includes panels for all logged keys. You can use **Quick add** to re-add a panel that you deleted.
</Note>

To open **Quick add**, do one of the following:

* To use **Quick add** to add a panel globally, click **Add panels** in the control bar near the panel search field, then click **Quick add**.
* To use **Quick add** to add a panel directly to a section, click the section's **action (<Icon icon="ellipsis" />)** menu, click **Add panels**, then click **Quick add**.

A list of panels appears. Each panel with a checkmark is already included in the workspace. To add panels from the list:

* To add all available panels, click the **Add `[N]` panels** button at the top of the list. The **Quick Add** list closes and the new panels display in the workspace.
* To add an individual panel from the list, hover over the panel's row, then click **Add**. Repeat this step for each panel you want to add, then click the **X** at the top right to close the **Quick Add** list. The new panels display in the workspace.

Optional: Customize the panel's settings.

## Share a panel

Share a panel to send collaborators directly to a focused view of a specific visualization without navigating to the full workspace. To share a panel using a link, do one of the following:

* While viewing the panel in full-screen mode, copy the URL from the browser.
* Click the **action (<Icon icon="ellipsis" />)** menu, then click **Copy panel URL**.

Share the link with the user or team. When they access the link, the panel opens in [full-screen mode](#view-a-panel-in-full-screen-mode).

To return to a panel's workspace from full-screen mode, click the left-pointing arrow at the top of the page.

### Compose a panel's full-screen link programmatically

When you [create an automation](/models/automations/) or similar workflow, it can be useful to include the panel's full-screen URL. The following format shows a panel's full-screen URL. In the following example, replace the entity, project, panel, and section names in brackets.

```text theme={null}
https://wandb.ai/[ENTITY_NAME]/[PROJECT_NAME]?panelDisplayName=[PANEL_NAME]&panelSectionName=[SECTION_NAME]
```

If multiple panels in the same section have the same name, this URL opens the first panel with the name.

### Embed or share a panel on social media

To embed a panel in a website or share it on social media, the panel must be viewable by anyone with the link. If a project is private, only members of the project can view the panel. If the project is public, anyone with the link can view the panel.

To get the code to embed or share a panel on social media:

1. From the workspace, hover over the panel, then click its **action (<Icon icon="ellipsis" />)** menu.
2. Click the **Share** tab.
3. Change **Only those who are invited have access** to **Anyone with the link can view**. Otherwise, the choices in the next step aren't available.
4. Choose **Share on Twitter**, **Share on Reddit**, **Share on LinkedIn**, or **Copy embed link**.

### Email a panel report

To email a single panel as a stand-alone report:

1. Hover over the panel, then click the panel's **action (<Icon icon="ellipsis" />)** menu.
2. Click **Share panel in report**.
3. Select the **Invite** tab.
4. Enter an email address or username.
5. Optional: Change **can view** to **can edit**.
6. Click **Invite**. W\&B sends an email to the user with a clickable link to the report that contains only the panel you're sharing.

Unlike when you [share a panel](#share-a-panel), the recipient can't get to the workspace from this report.

## Manage panels

After you add panels to a workspace, you can edit, move, duplicate, or remove them to keep your visualizations organized and up to date.

### Edit a panel

To edit a panel:

1. Click its pencil icon.
2. Modify the panel's settings.
3. Optional: To change the panel to a different type, select the type and then configure the settings.
4. Click **Apply**.

### Move a panel

To move a panel to a different section, you can use the drag handle on the panel. To select the new section from a list instead:

1. Optional: Create a new section by clicking **Add section** after the last section.
2. Click the **action (<Icon icon="ellipsis" />)** menu for the panel.
3. Click **Move**, then select a new section.

You can also use the drag handle to rearrange panels within a section.

### Duplicate a panel

To duplicate a panel:

1. At the top of the panel, click the **action (<Icon icon="ellipsis" />)** menu.
2. Click **Duplicate**.

If desired, you can [customize](#edit-a-panel) or [move](#move-a-panel) the duplicated panel.

### Remove panels

To remove a panel:

1. Hover over the panel.
2. Click the **action (<Icon icon="ellipsis" />)** menu.
3. Click **Delete**.

To remove all panels from a manual workspace, click its **action (<Icon icon="ellipsis" />)** menu, then click **Clear all panels**.

To remove all panels from an automatic or manual workspace, you can [reset the workspace](#reset-a-workspace). Select **Automatic** to start with the default set of panels, or select **Manual** to start with an empty workspace with no panels.

## Manage sections

Sections group related panels together, making it easier to scan large workspaces. By default, sections in a workspace reflect the logging hierarchy of your keys. However, in a manual workspace, sections appear only after you start adding panels.

### Add a section

To add a section, click **Add section** after the last section.

To add a new section before or after an existing section, you can instead click the section's **action (<Icon icon="ellipsis" />)** menu, then click **New section below** or **New section above**.

<Warning>
  Don't name a section "Section". Because of a known limitation, panels in this section don't render until you rename the section.
</Warning>

### Manage a section's panels

Within a section, you can resize panels, control pagination, and delete individual panels. Sections with many panels are paginated by default. The default number of panels on a page depends on the panel's configuration and on the sizes of the panels in the section.

To resize a panel, hover over it, click the drag handle, and drag it to adjust the panel's size. Resizing one panel resizes all panels in the section.

If a section is paginated, you can customize the number of panels to show on a page:

1. At the top of the section, click **1 to `[X]` of `[Y]`**, where `[X]` is the number of visible panels and `[Y]` is the total number of panels.
2. Choose how many panels to show per page, up to 100.

To delete a panel from a section:

1. Hover over the panel, then click its **action (<Icon icon="ellipsis" />)** menu.
2. Click **Delete**.

If you reset a workspace to an automated workspace, all deleted panels appear again.

### Rename a section

To rename a section, click its **action (<Icon icon="ellipsis" />)** menu, then click **Rename section**.

<Warning>
  Don't name a section "Section". Because of a known limitation, panels in this section don't render until you rename the section.
</Warning>

### Delete a section

To delete a section, click the **action (<Icon icon="ellipsis" />)** menu, then click **Delete section**. This removes the section and its panels.


# Bar plots
Source: https://docs.wandb.ai/models/app/features/panels/bar-plot

Visualize metrics, customize axes, and compare categorical data as bars.

A bar plot presents categorical data with rectangular bars that you can plot vertically or horizontally. Use bar plots to visualize metrics, compare categorical data, and customize axes for your runs. Bar plots appear by default with `wandb.Run.log()` when all logged values are of length one.

<Frame>
  <img alt="Box and horizontal bar plots in W&B" />
</Frame>

Use chart settings to limit the maximum number of runs shown, group runs by any config, and rename labels.

<Frame>
  <img alt="Customized bar plot" />
</Frame>

## Customize bar plots

You can also create **Box** or **Violin** plots to combine many summary statistics into one chart type.

To create a box or violin plot:

1. Group runs through the runs table.
2. In the workspace, click **Add panel**.
3. Add a standard **Bar Chart** and select the metric to plot.
4. Under the **Grouping** tab, pick **Box** or **Violin** to plot either of these styles.

<Frame>
  <img alt="Customizing a bar plot with grouping options" />
</Frame>


# Save and diff code
Source: https://docs.wandb.ai/models/app/features/panels/code

Enable code saving, compare code across W&B runs with the code comparer, and capture Jupyter session history.

This page explains how to enable code saving so you can compare the code used across W\&B runs and review the cells executed in Jupyter sessions. Saving code makes it easier to reproduce experiments and understand how changes to your training code affect results.

By default, W\&B only saves the latest Git commit hash. You can turn on more code features to compare the code between your experiments in the UI.

Starting with `wandb` version 0.8.28, W\&B can save the code from your main training file where you call `wandb.init()`.

## Save library code

When you enable code saving, W\&B saves the code from the file that called `wandb.init()`. To save additional library code, you have three options.

### Call `log_code` after `wandb.init`

Call `wandb.Run.log_code(".")` after `wandb.init()`:

```python theme={null}
import wandb

with wandb.init() as run:
  run.log_code(".")
```

### Pass a settings object with `code_dir`

Pass a settings object to `wandb.init()` with `code_dir` set:

```python theme={null}
import wandb

wandb.init(settings=wandb.Settings(code_dir="."))
```

This captures all Python source code files in the current directory and all subdirectories as an [artifact](/models/ref/python/experiments/artifact). For more control over the types and locations of source code files that W\&B saves, see the [reference docs](/models/ref/python/experiments/run#log_code).

### Set code saving in the UI

In addition to setting code saving programmatically, you can configure defaults in the UI at the team or organization level. The following sections describe the team-level and organization-level settings.

#### Team

By default, W\&B disables code saving for all teams. Before you can turn it on for a team, an organization admin must turn it on for the organization. See the [Organization](#organization) section.

A team admin can open the team **Settings** page, go to the **Privacy** section, and configure **Enable code saving by default** for runs in that team. This option is available only when an organization admin hasn't enforced code saving restrictions for the whole organization. For navigation steps, see [Configure privacy settings for a team](/platform/hosting/privacy-settings#configure-privacy-settings-for-a-team).

#### Organization

An organization admin can open organization **Settings**, go to the **Privacy** section, and activate **Enforce default code saving restrictions** so code saving stays off by default for every team. While this enforcement is active, team admins can't turn on **Enable code saving by default** for a team. For the full list of organization controls, see [Enforce privacy settings for all teams](/platform/hosting/privacy-settings#enforce-privacy-settings-for-all-teams).

## Code comparer

The code comparer panel displays the code from different W\&B runs side by side in the workspace.

To compare code used in different W\&B runs:

1. Select the **Add panels** button in the top right corner of the page.
2. Expand the **TEXT AND CODE** dropdown and select **Code**.

<Frame>
  <img alt="Code comparer panel" />
</Frame>

## Jupyter session history

W\&B saves the history of code executed in your Jupyter notebook session. When you call `wandb.init()` inside of Jupyter, W\&B adds a hook to automatically save a Jupyter notebook that contains the history of code executed in your current session.

To view the saved notebook history for a run:

1. Navigate to the project workspace that contains your code.
2. Select the **Artifacts** tab in the project sidebar.
3. Expand the **code** artifact.
4. Select the **Files** tab.

<Frame>
  <img alt="Jupyter session history" />
</Frame>

This displays the cells that ran in your session along with any outputs created by calling IPython's `display` method. This lets you see exactly what code ran within Jupyter in a given run. When possible, W\&B also saves the most recent version of the notebook, which you find in the code directory as well.

<Frame>
  <img alt="Jupyter session output" />
</Frame>


# Line plots overview
Source: https://docs.wandb.ai/models/app/features/panels/line-plot

Visualize metrics, customize axes, and compare multiple lines on a plot

Line plots display by default for metrics logged with `wandb.Run.log()` over time. Line plots support plotting multiple metrics, calculating custom axes, and more.

This page shows how to create, configure, and manage line plots in a [workspace](/models/track/workspaces).

<Frame>
  <img alt="Line plot example" />
</Frame>

<Tip>
  For [runs](/models/runs) that execute on [CoreWeave Kubernetes Service (CKS)](https://docs.coreweave.com/products/cks) clusters, [CoreWeave Mission Control](https://www.coreweave.com/mission-control) can monitor your compute infrastructure when the integration is enabled. If an error occurs, W\&B populates infrastructure information onto your run's plots in your project's workspace. For prerequisites and details, see [Visualize CoreWeave infrastructure alerts](/models/runs/infrastructure-alerts).
</Tip>

## Add a line plot

The following sections describe how to create a line plot for a single metric or multiple metrics.

<Tabs>
  <Tab title="Single-metric line plot">
    In an [automatic workspace](/models/app/features/panels#workspace-modes), W\&B creates a single-metric line plot automatically for each logged metric. Follow these steps to re-add a line plot that was deleted from an automatic workspace, or to add a line plot to a manual workspace.

    1. Navigate to your workspace.
    2. To add a line plot globally, click **Add panels** in the control bar near the panel search field.

       To add a line plot directly to a section instead, click the section's **action (<Icon icon="ellipsis" />)** menu, then click **+ Add panels**.
    3. To add a single-metric plot with default settings, click **Quick panel builder**.
       1. In the **Single-key panels** tab, hover over a metric, then click **Add**. Repeat this step for each panel you want to add.
       2. Click **Create \[NUMBER] panels**.
    4. To add a custom line plot instead, click **Line plot**.
       1. Configure the line plot's data, grouping, and display preferences using the corresponding tabs. For details, see [Edit line plot settings](#edit-line-plot-settings).
       2. To add calculated expressions to the x-axis or y-axis, click **Expressions**. [JavaScript regular expressions](https://www.w3schools.com/js/js_regexp.asp) are supported.
    5. Select the type of panel to add, such as a chart. The panel's configuration details appear with selected defaults.
    6. Optionally, customize the panel and its display preferences. Configuration options depend on the type of panel you select. For more information about the options for each type of panel, see [Line plots](/models/app/features/panels/line-plot/) or [Bar plots](/models/app/features/panels/bar-plot/).
    7. Click **Apply**.
  </Tab>

  <Tab title="Multi-metric line plot">
    <Note>
      This feature is in preview, available by invitation only. To request enrollment, contact [support](mailto:support@wandb.com) or your AISE.
    </Note>

    In an [automatic workspace](/models/app/features/panels#workspace-modes), W\&B creates a single-metric line plot automatically for each logged metric. This section shows how to create a single line plot that shows multiple metrics together, defined by a JavaScript regular expression. You can optionally consolidate many single-metric plots into a single multi-metric plot. This can improve the performance of a workspace with many logged metrics, and can help you analyze the results of your runs.

    1. Navigate to your workspace.
    2. To add a line plot globally, click **Add panels** in the control bar near the panel search field.

       To add a line plot directly to a section instead, click the section's **action (<Icon icon="ellipsis" />)** menu, then click **+ Add panels**.
    3. Click **Quick panel builder**, then click the **Multi-metric panels** tab.
    4. In **Regex** enter an expression in [JavaScript regular expression](https://www.w3schools.com/js/js_regexp.asp) format. As you type, the UI updates to show which metrics match the expression. By default, the plot name shows the regular expression used by the plot. The plot includes lines for all metrics that match the expression, including metrics logged in the future.
    5. To optionally remove duplicate single-metric panels when the multi-metric plot is created, toggle **Clean up auto-generated panels**. A preview shows which panels are cleaned up. <Note>When this option is turned on, W\&B doesn't create a single-metric plot for a newly logged metric that matches the expression. Instead, it appears only in this multi-metric plot.</Note>
    6. Click **Create \[NUMBER] panels**.

    ### Multi-metric regular expressions

    Multi-metric line plots use [JavaScript regular expressions](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Guide/Regular_expressions) to match metric names. The following sections describe common use cases and provide more details about how the regular expressions work, such as how capture groups affect the panels that W\&B creates.

    #### Common use cases

    The following examples show some ways you can use multi-metric panels to help you analyze your experiment results.

    **Compare metrics across layers or model components**
    Instead of creating separate panels for each layer's metrics, you can view them together in a single panel. For example, if you log metrics with consistent naming, like `layer_0_loss`, `layer_1_loss`, and `layer_2_loss` in this Python example code, you can use the regex `layer_\d+_loss` to display all layer losses on one plot.

    ```python theme={null}
    with wandb.init(project="multi-layer-model") as run:
        for step in range(100):
            run.log({
                "layer_0_loss": loss_0,
                "layer_1_loss": loss_1,
                "layer_2_loss": loss_2,
                "step": step
            })
    ```

    **Group related metrics by prefix or suffix**
    Match all metrics that share a common naming pattern. For example:

    * `train_.*` matches all training metrics like `train_loss`, `train_accuracy`, `train_f1_score`.
    * `.*_accuracy` matches accuracy metrics across different datasets like `train_accuracy`, `val_accuracy`, `test_accuracy`.

    **Match specific metric variations**
    Use alternation to match only the metrics you want. For example, the non-capture group `(?:layer_0|layer_10)_loss` matches only the first and tenth layer losses, excluding intermediate layers.

    #### Capture groups

    Capture groups in your regular expression control how multi-metric panels are created. This behavior can be confusing if you're not expecting it.

    * **Capture groups create multiple panels**
      When your regular expression includes parentheses that form a capture group, the UI creates a separate panel for each unique value captured by that group.

      For example, the expression `(layer_0|layer_10)_loss` includes a capture group and will create two separate panels:

      1. One panel for metrics matching `layer_0`.
      2. One panel for metrics matching `layer_10`.

    * **Non-capturing groups keep metrics together**
      To match multiple alternatives without creating separate panels, use a non-capturing group with the `?:` syntax. The expression `(?:layer_0|layer_10)_loss` matches the same metrics as the previous example, but displays them together in a single panel.

    Here's the difference:

    * `(layer_0|layer_10)_loss` - Creates two panels, one for each layer.
    * `(?:layer_0|layer_10)_loss` - Creates one panel showing both layers together.

    This gives you flexibility to choose the approach that best fits your analysis needs. Use capture groups when you want to separate metrics into distinct panels. Use non-capturing groups when you want to compare metrics together on a single plot.
  </Tab>
</Tabs>

## Edit line plot settings

The following sections describe how to edit the settings for an individual line plot panel, all line plot panels in a section, or all line plot panels in a workspace. For details about line plot settings, see [Line plot reference](/models/app/features/panels/line-plot/reference).

### Individual line plot

A line plot's individual settings override the line plot settings for the section or the workspace. To customize a line plot:

1. Navigate to your workspace.
2. Hover your mouse over the panel, then click the gear icon.
3. Within the drawer that appears, select a tab to edit its settings.
4. Click **Apply**.

Line plot settings are organized into tabs:

* **Data**: Configure x-axis, y-axis, sampling method, smoothing, outliers, and chart type.
* **Grouping**: Configure whether and how to group and aggregate runs in the plot.
* **Chart**: Specify titles for the panel and axes, and configure legend visibility and position.
* **Legend**: Customize the appearance and content of the panel's legend.
* **Expressions**: Add custom calculated expressions for the axes.

For detailed information about each setting, see the [Line plot reference](/models/app/features/panels/line-plot/reference).

### All line plots in a section

To customize the default settings for all line plots in a section, overriding workspace settings for line plots:

1. Navigate to your workspace.
2. Click the section's gear icon to open its settings.
3. Within the drawer that appears, select the **Data** or **Display preferences** tabs to configure the default settings for the section. For details about each **Data** setting, see the [Line plot reference](/models/app/features/panels/line-plot/reference). For details about each display preference, see [Configure section layout](../#configure-section-layout).

### All line plots in a workspace

To customize the default settings for all line plots in a workspace:

1. Navigate to your workspace.
2. Click the workspace settings icon, which has a gear with the label **Settings**.
3. Click **Line plots**.
4. Within the drawer that appears, select the **Data** or **Display preferences** tabs to configure the default settings for the workspace.
   * For details about each **Data** setting, see the [Line plot reference](/models/app/features/panels/line-plot/reference).
   * For details about each **Display preferences** section, see [Workspace display preferences](../#configure-workspace-layout). At the workspace level, you can configure the default **Zooming** behavior for line plots. This setting controls whether to sync zooming across line plots with a matching x-axis key. Deactivated by default.

## Visualize average values on a plot

If you have several different experiments and you want to see the average of their values on a plot, you can use the Grouping feature in the table. Click **Group** above the run table and select **All** to show averaged values in your graphs.

The following image shows the graph before averaging, with one line per run:

<Frame>
  <img alt="Individual precision lines" />
</Frame>

The following image shows a graph that represents average values across runs using grouped lines.

<Frame>
  <img alt="Averaged precision lines" />
</Frame>

## Visualize NaN value on a plot

To track metrics that might sometimes be undefined, such as a loss that returns `NaN`, you can log them and W\&B renders them on the line plot. You can also plot `NaN` values including PyTorch tensors on a line plot with `wandb.Run.log()`. For example:

```python theme={null}
with wandb.init() as run:
    # Log a NaN value
    run.log({"test": float("nan")})
```

<Frame>
  <img alt="NaN value handling" />
</Frame>

## Compare multiple metrics on one chart

To compare multiple metrics from one or more runs side by side, add a **Run comparer** panel to your workspace.

<Frame>
  <img alt="Adding visualization panels" />
</Frame>

1. Navigate to your workspace.
2. Select the **Add panels** button in the top right corner of the page.
3. From the drawer that appears, expand the **Evaluation** dropdown.
4. Select **Run comparer**.

## Change the colors of the lines

If the default color of runs isn't helpful for comparison, W\&B provides two ways to change the colors: from the run table or from a chart's legend settings.

<Tabs>
  <Tab title="From the run table">
    Each run is given a random color by default upon initialization.

    <Frame>
      <img alt="Random colors given to runs" />
    </Frame>

    Upon clicking any of the colors, a color palette appears from which you can choose the color you want.

    <Frame>
      <img alt="The color palette" />
    </Frame>
  </Tab>

  <Tab title="From the chart legend settings">
    1. Navigate to your workspace.
    2. Hover your mouse over the panel whose settings you want to edit.
    3. Select the pencil icon that appears.
    4. Choose the **Legend** tab.

    <Frame>
      <img alt="Line plot legend settings" />
    </Frame>
  </Tab>
</Tabs>

## Visualize on different x axes

By default, line plots use training steps as the x-axis, but you can switch to a different x-axis to view your data from another perspective. If you want to see the absolute time that an experiment has taken, or see what day an experiment ran, you can switch the x-axis. The following example shows switching from steps to relative time and then to wall time.

<Frame>
  <img alt="X-axis time options" />
</Frame>

To use a custom x-axis, log the metric in the same call to `wandb.Run.log()` where you log the y-axis. For example:

```python theme={null}
with wandb.init() as run:
    for i in range(100):
        run.log({"accuracy": acc, "custom_x": i * 10})
```

For more details, see [Customize log axes](/models/track/log/customize-logging-axes#customize-log-axes).

## Zoom

To inspect a specific region of a line plot more closely, you can zoom in on both axes at once. Click and drag a rectangle to zoom vertically and horizontally at the same time. This changes the x-axis and y-axis zoom.

<Frame>
  <img alt="Plot zoom functionality" />
</Frame>

## Hide chart legend

If the chart legend is taking up space you want to use for the plot, you can turn it off. Turn off the legend in the line plot with this toggle:

<Frame>
  <img alt="Hide legend toggle" />
</Frame>

## Create a run metrics notification

Use [Automations](/models/automations/) to notify your team when a run metric meets a condition you specify. An automation can post to a Slack channel or run a webhook.

From a line plot, you can create a [run metrics notification](/models/automations/automation-events/#run-events) for the metric it shows:

1. Navigate to your workspace.
2. Hover over the panel, then click the bell icon.
3. Configure the automation using the basic or advanced configuration controls. For example, apply a run filter to limit the scope of the automation, or configure an absolute threshold.

Learn more about [Automations](/models/automations/).


# Line plot reference
Source: https://docs.wandb.ai/models/app/features/panels/line-plot/reference

Reference for line plot panel settings including x-axis, y-axis, smoothing, aggregation, and grouping options.

This page provides comprehensive details for line plot settings. For more details about working with line plots, see the [Line plots overview](/models/app/features/panels/line-plot).

## Data settings

Data settings control which metrics appear on the plot and how data points are sampled, smoothed, and aggregated.

### X-axis

<Frame>
  <img alt="Selecting X-Axis" />
</Frame>

Set the range of the x-axis to any integer or float value you logged with `wandb.Run.log()`.

Available time-based x-axis options:

* **Step**: Increments each time `wandb.Run.log()` is called. Reflects the number of training steps logged from your model. (Default)
* **Relative Time (Wall)**: Clock time since the process started. If you start a run, pause it for a day, then resume and log, that point appears at 24 hours.
* **Relative Time (Process)**: Time inside the running process. If you start a run, run for 10 seconds, pause for a day, then resume, that point appears at 10 seconds.
* **Wall Time**: Minutes elapsed since the start of the first run on the graph.
* **X range**: From the smallest to largest value of your x-axis by default. You can customize the minimum and maximum values.

### Y-axis

Set the y-axis variables to any integer or float value you logged with `wandb.Run.log()`. Specify a single value, array of values, or histogram of values. If you logged more than 1,500 points for a variable, W\&B samples down to 1,500 points.

<Note>
  Customize the color of your y-axis lines by changing the run color in the runs table.
</Note>

The **Y range** option defaults to the range from the smallest positive value of your metrics (inclusive of 0) to the largest value of your metrics. You can customize the minimum and maximum values.

### Point aggregation method

Choose the sampling mode for displaying data points:

* **Random sampling** (default): See [Random sampling](/models/app/features/panels/line-plot/sampling/#random-sampling).
* **Full fidelity**: See [Full fidelity](/models/app/features/panels/line-plot/sampling/#full-fidelity).

### Smoothing

Set the [smoothing coefficient](/support/models/articles/what-formula-do-you-use-for-your-smoothi) between 0 and 1, where 0 is no smoothing and 1 is maximum smoothing.

Available smoothing methods:

* **Time-weighted EMA** (default): an exponential moving average (EMA) technique for smoothing time series data by exponentially decaying the weight of previous points.
* **Running average**: replaces a point with the average of points in a window before and after the given x value.
* **Gaussian**: computes a weighted average of the points, where the weights correspond to a gaussian distribution with the standard deviation specified as the smoothing parameter.
* **No smoothing**

For comprehensive details, see [Smooth line plots](/models/app/features/panels/line-plot/smoothing).

### Ignore outliers

Rescale the plot to exclude outliers from the default plot min and max scale. The setting's impact depends on the plot's sampling mode:

* **Random sampling mode**: Ignoring outliers omits points below 5% and above 95% from the plot.
* **Full fidelity mode**: Ignoring outliers shows all points, condensed down to the last value in each bucket, and shades the area below 5% and above 95%.

### Max runs or groups

By default, plots include only the first 10 runs or groups of runs in the run list or run set. Change the sort order to control which runs or groups to display.

<Note>
  A workspace is limited to displaying a maximum of 1,000 runs, regardless of its configuration.
</Note>

### Chart type

Choose the plot style:

* **Line plot**
  <Frame>
    <img alt="Line plot style" />
  </Frame>
* **Area plot**
  <Frame>
    <img alt="Area plot style" />
  </Frame>
* **Percentage area plot**
  <Frame>
    <img alt="Percentage plot style" />
  </Frame>

Configure the chart type in the **Data** tab. Refer to [Individual line plot](/models/app/features/panels/line-plot#individual-line-plot).

## Grouping settings

Aggregate all runs by turning on grouping, or group over an individual variable. When you turn on grouping in the runs table, the groups automatically populate into the graph.

* **Group runs**: Turn on run grouping in the plot. Required to configure the shaded range in the plot.
* **Group by**: Optionally select a column. All runs with the same value in that column are grouped together.
* **Aggregation**: The value of the line on the graph. Options are mean, median, min, and max of the group.
* **Range**: Configure the shaded area of a full-fidelity line plot. Options are Min/Max, Std Dev, Std Err, or None.

## Chart settings

Configure titles and legend visibility:

* **Panel title**: Title displayed at the top of the panel.
* **X-axis title**: Label for the x-axis.
* **Y-axis title**: Label for the y-axis.
* **Legend**: Show or hide the legend, and configure its position.

## Legend settings

Customize the legend to show any config value logged and metadata from the runs, such as creation time or the user who created the run.

### Legend template

Define a template for the legend name.

1. Click the gear icon to open the plot settings.
2. Go to the **Display preferences** tab.
3. Expand **Advanced legend**, then specify the legend template.
4. Click **Apply**.

### Point-specific values

Set values inside `[[ ]]` to display point-specific values in the crosshair when hovering over a chart.

1. Click the gear icon to open the plot settings.
2. Go to the **Display preferences** tab.
3. At the bottom of the tab, configure point-specific values for one or more of the plot's metrics.
4. Click **Apply**.

Supported values inside `[[ ]]`:

| Value         | Meaning                                    |
| ------------- | ------------------------------------------ |
| `${x}`        | X value                                    |
| `${y}`        | Y value (including smoothing adjustment)   |
| `${original}` | Y value not including smoothing adjustment |
| `${mean}`     | Mean of grouped runs                       |
| `${stddev}`   | Standard deviation of grouped runs         |
| `${min}`      | Min of grouped runs                        |
| `${max}`      | Max of grouped runs                        |
| `${percent}`  | Percent of total (for stacked area charts) |

## Expressions

Formulate math expressions to create or transform line plots from your logged metrics, config values, and summary statistics. For example, you can compute the difference between two metrics, rescale a metric by a config value, or plot the logarithm of a metric. Expressions are evaluated at each step that you log.

The following sections show how to reference values in expressions and describe the operators and functions you can use in custom expressions. In a line plot, you can use an expression for the x-axis, y-axis, or both.

See [Example expressions](#example-expressions) for common use cases and example expressions.

### Create new line plots with expressions

1. Navigate to your project's workspace.
2. Click the **+ Add panel** button and select **Line plot**.
3. Click the **Data** tab. Select the data you want to plot on the line plot for both the x and y axes.
4. Click the **Expressions** tab.
5. In the **Y-axis** field or **X-axis** field, enter your expression.
6. Click **Apply** to save your settings and view the line plot.

### Transform existing line plots with expressions

1. Open the line plot you want to transform.
2. Click the gear icon in the upper right corner of the plot to edit the panel.
3. Click the **Expressions** tab.
4. In the **Y-axis** field or **X-axis** field, enter your expression.
5. Click **Apply** to save your settings and view the updated line plot.

### Referencing values

The following table describes how to reference logged metrics, config parameters, and summary statistics in line plot expressions.

| Type              | Syntax                 | Description                                              | Examples                                            |
| ----------------- | ---------------------- | -------------------------------------------------------- | --------------------------------------------------- |
| Metric            | `${metric_name}`       | Reference a logged metric by name.                       | `${val/accuracy}`, `${"accuracy"}`                  |
| Config parameter  | `${config:param_name}` | Reference config values using the `${config:}` prefix.   | `${config:lr}`, `${config:batch_size}`              |
| Summary statistic | `${summary:stat_name}` | Reference summary fields using the `${summary:}` prefix. | `${summary:final_accuracy}`, `${summary:best_loss}` |

Use `${...}` to escape any metric name, config parameter, or summary field that includes special characters or spaces such as `/`, `-`, or spaces.

For example, if you log a metric named `val/accuracy`, reference it as `${val/accuracy}` to avoid confusion with the division operator. If you log a config parameter named `dropout-rate`, reference it as `${config:dropout-rate}`. If you log a summary field named `best loss`, reference it as `${summary:best loss}`.

#### Nested configs

Access nested config values using dot notation and the following syntax:

```javascript theme={null}
${config:parent.child.grandchild}
```

Where `parent`, `child`, and `grandchild` are the keys in the nested config dictionary.

For example, suppose you log the following config with nested dictionaries:

```python theme={null}
config = {
    "model": {
        "type": "resnet",
        "layers": 50
    },
    "training": {
        "batch_size": 32,
        "learning_rate": 0.001
    }
}

with wandb.init(project="my-project", config=config) as run:
    ...
```

You can reference the model type with: `${config:model.type}`. Reference the batch size with: `${config:training.batch_size}`.

As another example, consider the following config with nested dictionaries:

```json Config parameters  theme={null}
config:
  optimizer:
    value:
      lr: 0.001
      weight_decay: 0.01
  model:
    value:
      hidden_size: 768
```

Reference the learning rate with `${config:optimizer.value.lr}`, the model's hidden size with `${config:model.value.hidden_size}`, or the weight decay with `${config:optimizer.value.weight_decay}`.

### Available operators

W\&B supports the following operators for line plot expressions:

| Category   | Operators                                         |
| ---------- | ------------------------------------------------- |
| Arithmetic | `+`, `-`, `*`, `/`, `%` (modulo), `**` (exponent) |
| Comparison | `==`, `!=`, `===`, `!==`, `<`, `>`, `<=`, `>=`    |
| Bitwise    | `\|`, `^`, `&`, `<<`, `>>`, `>>>`                 |
| Logical    | `\|\|` , `&&`                                     |

### Math constants and functions

All JavaScript math functions and constants are supported. For more details, see the [MDN Math documentation](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Math). The following tables summarize the most commonly used functions and constants for line plot expressions.

#### Math constants

| Constant  | Description             |
| --------- | ----------------------- |
| `e`       | Euler's number          |
| `pi`      | Pi                      |
| `ln2`     | Natural logarithm of 2  |
| `ln10`    | Natural logarithm of 10 |
| `log2e`   | Base-2 logarithm of e   |
| `log10e`  | Base-10 logarithm of e  |
| `sqrt2`   | Square root of 2        |
| `sqrt1_2` | Square root of 1/2      |

#### Arithmetic and statistical functions

The following table describes available arithmetic and statistical functions:

| Function       | Description                                    |
| -------------- | ---------------------------------------------- |
| abs(x)         | Absolute value                                 |
| ceil(x)        | Ceiling function (round up to nearest integer) |
| floor(x)       | Floor function (round down to nearest integer) |
| round(x)       | Round to nearest integer                       |
| min(x, y, ...) | Minimum value                                  |
| max(x, y, ...) | Maximum value                                  |
| sqrt(x)        | Square root                                    |

#### Logarithmic and exponential functions

The following table describes available logarithmic and exponential functions:

| Function  | Description                |
| --------- | -------------------------- |
| log(x)    | Natural logarithm (base e) |
| log10(x)  | Base-10 logarithm          |
| log2(x)   | Base-2 logarithm           |
| exp(x)    | Exponential function (e^x) |
| pow(x, y) | Power function (x^y)       |

#### Trigonometric functions

The following table describes available trigonometric functions:

| Function    | Description                   |
| ----------- | ----------------------------- |
| sin(x)      | Sine                          |
| cos(x)      | Cosine                        |
| tan(x)      | Tangent                       |
| asin(x)     | Arc sine (inverse sine)       |
| acos(x)     | Arc cosine (inverse cosine)   |
| atan(x)     | Arc tangent (inverse tangent) |
| atan2(y, x) | Two-argument arc tangent      |

#### Hyperbolic functions

The following table describes available hyperbolic functions:

| Function | Description        |
| -------- | ------------------ |
| sinh(x)  | Hyperbolic sine    |
| cosh(x)  | Hyperbolic cosine  |
| tanh(x)  | Hyperbolic tangent |

### Example expressions

The following are example expressions for line plot axes. These examples are for illustrative purposes. You can use any combination of operators and functions described in the previous sections to create complex expressions that suit your needs.

For the following examples, suppose your [summary metrics](/models/track/log/log-summary#log-summary-metrics) include `accuracy` and `loss` with the following values:

```json Summary metrics theme={null}
{
  "accuracy": 0.7829240801794489,
  "loss": 0.2194763318905079
}
```

And suppose your config looks like this:

```json Config parameters theme={null}
config = {
    "epochs": 100,
    "optimizer": {
        "value": {
            "lr": 0.001,
            "weight_decay": 0.01
        }
    }
}
```

Vertically shift the accuracy metric by adding a constant value. In the following example, shift the accuracy up by 1:

```javascript theme={null}
1 - accuracy
```

Vertically shift a metric by a constant value, in this case the learning rate (`lr`):

```javascript theme={null}
accuracy+${config:optimizer.value.lr}
```

Compute the sine of a metric. In the following example, compute the sine of the loss metric:

```javascript theme={null}
sin(loss)
```

Phase shift a metric by applying a sine function. For example, apply a sine function and phase shift the loss metric by 2:

```javascript theme={null}
sin(loss - 2)
```

Rescale the accuracy metric by a config parameter named `batch_size`:

```javascript theme={null}
${accuracy} / ${config:batch_size}
```

Compute the minimum of two metrics. In the following example, compute the minimum of loss and accuracy:

```javascript theme={null}
min(loss, accuracy)
```

Compute the square root of the sum of squares of two metrics. In the following example, compute the square root of the sum of squares of loss and accuracy:

```javascript theme={null}
sqrt(loss ** 2 + accuracy ** 2)
```

Compute the exponential of the loss. In the following example, compute the exponential of the loss metric:

```javascript theme={null}
sqrt(loss*100)+sqrt(loss*100000)
```

<Note>
  You can also refer to summary metric values with `${summary:metric_name}` syntax in expressions. For example:

  ```javascript theme={null}
  sqrt(${summary:loss}*100)+sqrt(${summary:loss}*100000)
  ```
</Note>

### Multi-metric panel expressions

Use a regular expression to create a single line plot that shows multiple metrics together (including matching metrics logged in the future). For detailed instructions, see [Add a line plot](/models/app/features/panels/line-plot#multi-metric-line-plot).

For example:

* Instead of creating separate panels for each layer's metrics, you can view them together in a single panel. For example, if you log metrics with consistent naming, like `layer_0_loss`, `layer_1_loss`, and `layer_2_loss`, you can use a regex like `layer_\d+_loss` to display all layer losses on one plot.
* Match all metrics that share a common naming pattern. For example:
  * `train_.*` matches all training metrics like `train_loss`, `train_accuracy`, `train_f1_score`
  * `.*_accuracy` matches accuracy metrics across different datasets like `train_accuracy`, `val_accuracy`, `test_accuracy`
* Use alternation to match only the metrics you want. For example, the non-capture group `(?:layer_0|layer_10)_loss` matches only the first and tenth layer losses, excluding intermediate layers.

#### Capture groups

Capture groups in your regular expression control how multi-metric panels are created. This behavior can be confusing if you're not expecting it.

* **Capture groups create multiple panels**
  When your regular expression includes parentheses that form a capture group, the UI creates a separate panel for each unique value captured by that group.

  For example, the expression `(layer_0|layer_10)_loss` includes a capture group and will create two separate panels:

  1. One panel for metrics matching `layer_0`.
  2. One panel for metrics matching `layer_10`.

* **Non-capturing groups keep metrics together**
  To match multiple alternatives without creating separate panels, use a non-capturing group with the `?:` syntax. The expression `(?:layer_0|layer_10)_loss` matches the same metrics as the previous example, but displays them together in a single panel.

Here's the difference:

* `(layer_0|layer_10)_loss` - Creates two panels, one for each layer.
* `(?:layer_0|layer_10)_loss` - Creates one panel showing both layers together.

This gives you flexibility to choose the approach that best fits your analysis needs. Use capture groups when you want to separate metrics into distinct panels. Use non-capturing groups when you want to compare metrics together on a single plot.


# Point aggregation
Source: https://docs.wandb.ai/models/app/features/panels/line-plot/sampling

Understand the two point aggregation modes for W&B line plots: full fidelity bucketed sampling and random sampling.

Use point aggregation methods within your line plots for improved data visualization accuracy and performance. This page explains the two available aggregation modes and how to configure each one, so you can choose the right tradeoff between detail and rendering speed for your workspace. Two types of point aggregation modes are available: [full fidelity](#full-fidelity) and [random sampling](#random-sampling). W\&B uses full fidelity mode by default.

## Full fidelity

Full fidelity is the default aggregation method and preserves the extreme values in your data. This section explains how full fidelity works, its main advantages, and how to turn it on for a single chart or an entire workspace.

When you use full fidelity mode, W\&B breaks the x-axis into dynamic buckets. The number of points per line adapts to chart size and the number of runs. It calculates the minimum and maximum values within each bucket (used for optional shading) and uses the last value in each bucket (not the average) to draw the primary line.

Full fidelity mode for point aggregation has three main advantages:

* Preserve extreme values and spikes: retain extreme values and spikes in your data.
* Configure how minimum and maximum points render: use the W\&B App to interactively decide whether you want to show extreme (min/max) values as a shaded area.
* Explore your data without losing data fidelity: W\&B recalculates x-axis bucket sizes when you zoom into specific data points. This helps ensure that you can explore your data without losing accuracy. W\&B caches previously computed aggregations to help reduce loading times, which is useful when you navigate through large datasets.

### Turn on full fidelity

W\&B uses full fidelity mode by default. To configure it manually, follow these steps:

<Tabs>
  <Tab title="All charts in a workspace">
    1. Navigate to your workspace.
    2. Select the gear icon on the top right corner of the screen next to the left of the **Add panels** button.
    3. From the UI slider that appears, select **Line plots**.
    4. Choose **Full fidelity** from the **Point aggregation** section.
    5. Configure the **Smoothing** algorithm and settings.
    6. Set **Aggregation** to **Mean**, **Min**, or **Max**.
    7. Click **Apply**.
  </Tab>

  <Tab title="Individual chart in a workspace">
    1. Navigate to your workspace.
    2. Select the **Workspace** icon on the left tab.
    3. Hover over the line plot panel you want to configure, then click the gear icon.
    4. Within the modal that appears, set **Point aggregation method** to **Full fidelity**.
    5. Configure the **Smoothing** algorithm and settings.
    6. Click **Apply**.
  </Tab>
</Tabs>

### Configure shading

Shading visualizes the variability within each bucket so you can see how much points spread around the line. The shaded areas of a full-fidelity line plot can show:

* **Min/Max**: For each x-axis point, shade the area between the minimum and maximum values. The shaded area shows all points from the lowest to the highest value in each bucket:
  ```math theme={null}
  \text{Min/Max Range} = [\min(x_1, x_2, \ldots, x_n),\ \max(x_1, x_2, \ldots, x_n)]
  ```
  where $x_1, x_2, \ldots, x_n$ are the values in a given bucket.

* **Standard deviation**: For each x-axis point, calculate the variability of the values using standard deviation, and shade the resulting area.
  ```math theme={null}
  SD = \sqrt{\frac{1}{n}\sum_{i=1}^{n}(x_i - \overline{x})^2}
  ```

* **Standard error**: For each x-axis point, calculate the likelihood of a sampling error by dividing the value by the square root of the sample size:
  ```math theme={null}
  SE = \frac{SD}{\sqrt{n}}
  ```

* **None**: No shading (the default).

The following image shows a blue line plot. The light blue shaded area represents the minimum and maximum values for each bucket.

<Frame>
  <img alt="Shaded confidence areas" />
</Frame>

To configure shading:

1. Navigate to your workspace.
2. Hover over a line plot, then click the gear icon.
3. In the **Data** tab, set **Point aggregation** to **Full fidelity** if necessary, then configure the smoothing algorithm.
4. In the **Grouping** tab, turn on **Group runs**. Optionally, set **Group by** to a run attribute.
5. Set **Agg** to **Mean** (default), **Min**, or **Max**.
6. Set **Range** to **Min/Max**, **Std Dev**, **Std Err**, or **None**.
7. Click **Apply**.

### Explore your data without losing data fidelity

Analyze specific regions of the dataset without missing critical points like extreme values or spikes. When you zoom in on a line plot, W\&B adjusts the bucket sizes used to calculate the minimum, maximum, and last value within each bucket.

<Frame>
  <img alt="Plot zoom functionality" />
</Frame>

W\&B uses dynamic binning to divide the x-axis into buckets. The number of points shown per line adapts to chart size and the number of runs: smaller chart dimensions or more runs can reduce the number of points per line so that the chart remains responsive and can display more lines. For each bucket, W\&B calculates the following values:

* **Minimum**: The lowest value in that bucket (used for shading).
* **Maximum**: The highest value in that bucket (used for shading).
* **Line value**: The last value in that bucket, used to draw the line.

W\&B plots values in buckets in a way that preserves full data representation and includes extreme values in every plot. The line is drawn from the last value in each bucket. When you zoom in far enough, full fidelity mode can render every data point without additional aggregation. The exact threshold depends on the current chart dimensions and number of runs.

To zoom in on a line plot, follow these steps:

1. Navigate to your W\&B project.
2. Select the **Workspace** icon on the left tab.
3. Optional: Add a line plot panel to your workspace or navigate to an existing line plot panel.
4. Click and drag to select a specific region to zoom in on.

<Note>
  **Line plot grouping and expressions**

  When you use Line Plot Grouping, W\&B applies the following based on the mode selected:

  * **Non-windowed sampling (grouping)**: Aligns points across runs on the x-axis. W\&B takes the average if multiple points share the same x-value. Otherwise, they appear as discrete points.
  * **Windowed sampling (grouping and expressions)**: Divides the x-axis either into 250 buckets or the number of points in the longest line (whichever is smaller). W\&B takes an average of points within each bucket.
  * **Full fidelity (grouping and expressions)**: Similar to non-windowed sampling, but fetches up to 500 points per run to balance performance and detail.
</Note>

## Random sampling

Random sampling is an alternative aggregation method that prioritizes rendering speed over data fidelity. Use it when you need faster chart performance and you don't need to preserve every extreme value.

Random sampling uses 1,500 randomly sampled points to render line plots. Because sampling drops points, spotting outliers or spikes becomes more difficult.

<Warning>
  Random sampling samples non-deterministically. As a result, random sampling sometimes excludes outliers or spikes in the data and therefore reduces data accuracy.
</Warning>

### Enable random sampling

By default, W\&B uses full fidelity mode. To enable random sampling, follow these steps:

<Tabs>
  <Tab title="All charts in a workspace">
    1. Navigate to your W\&B project.
    2. Select the **Workspace** icon on the left tab.
    3. Select the gear icon on the top right corner of the screen next to the left of the **Add panels** button.
    4. From the UI slider that appears, select **Line plots**.
    5. Choose **Random sampling** from the **Point aggregation** section.
  </Tab>

  <Tab title="Individual chart in a workspace">
    1. Navigate to your W\&B project.
    2. Select the **Workspace** icon on the left tab.
    3. Select the line plot panel you want to enable random sampling for.
    4. Within the modal that appears, select **Random sampling** from the **Point aggregation method** section.
  </Tab>
</Tabs>

### Access non-sampled data

If random sampling drops points you need, you can still access the complete, unsampled metric history programmatically. You can access the complete history of metrics logged during a run using the [W\&B Run API](/models/ref/python/public-api/runs). The following example demonstrates how to retrieve and process the loss values from a specific run:

```python theme={null}
# Initialize the W&B API
run = api.run("l2k2/examples-numpy-boston/i0wt6xua")

# Retrieve the history of the 'Loss' metric
history = run.scan_history(keys=["Loss"])

# Extract the loss values from the history
losses = [row["Loss"] for row in history]
```


# Smooth line plots
Source: https://docs.wandb.ai/models/app/features/panels/line-plot/smoothing

In line plots, use smoothing to see trends in noisy data.

Smoothing helps you spot trends in noisy line plots by reducing point-to-point variation, making the underlying signal easier to read. This page describes the smoothing algorithms W\&B supports, when each one is most useful, and how to control whether the original data remains visible.

W\&B supports several types of smoothing:

* [Time weighted exponential moving average (TWEMA) smoothing](#time-weighted-exponential-moving-average-twema-smoothing-default)
* [Gaussian smoothing](#gaussian-smoothing)
* [Running average](#running-average-smoothing)
* [Exponential moving average (EMA) smoothing](#exponential-moving-average-ema-smoothing)

To see these algorithms applied to real data, see this [interactive W\&B report](https://wandb.ai/carey/smoothing-example/reports/W-B-Smoothing-Features--Vmlldzo1MzY3OTc).

<Frame>
  <img alt="Comparison of various smoothing algorithms applied to a noisy line plot" />
</Frame>

## Time weighted exponential moving average (TWEMA) smoothing (default)

The time-weighted exponential moving average (TWEMA) smoothing algorithm is a technique for smoothing time series data by exponentially decaying the weight of previous points. For details about the technique, see [Exponential Smoothing](https://www.wikiwand.com/en/Exponential_smoothing). The range is 0 to 1. A debias term is added so that early values in the time series aren't biased towards zero.

The TWEMA algorithm takes the density of points on the line (the number of `y` values per unit of range on x-axis) into account. This allows consistent smoothing when displaying multiple lines with different characteristics simultaneously.

The following sample code shows how this works under the hood:

```javascript theme={null}
const smoothingWeight = Math.min(Math.sqrt(smoothingParam || 0), 0.999);
let lastY = yValues.length > 0 ? 0 : NaN;
let debiasWeight = 0;

return yValues.map((yPoint, index) => {
  const prevX = index > 0 ? index - 1 : 0;
  // VIEWPORT_SCALE scales the result to the chart's x-axis range
  const changeInX =
    ((xValues[index] - xValues[prevX]) / rangeOfX) * VIEWPORT_SCALE;
  const smoothingWeightAdj = Math.pow(smoothingWeight, changeInX);

  lastY = lastY * smoothingWeightAdj + yPoint;
  debiasWeight = debiasWeight * smoothingWeightAdj + 1;
  return lastY / debiasWeight;
});
```

To see this algorithm applied to live data, see the [TWEMA section of the interactive W\&B report](https://wandb.ai/carey/smoothing-example/reports/W-B-Smoothing-Features--Vmlldzo1MzY3OTc).

<Frame>
  <img alt="Line plot with TWEMA smoothing applied" />
</Frame>

## Gaussian smoothing

Gaussian smoothing (or Gaussian kernel smoothing) computes a weighted average of the points, where the weights correspond to a Gaussian distribution with the standard deviation specified as the smoothing parameter. W\&B calculates the smoothed value for every input `x` value, based on the points that occur both before and after it.

To see this algorithm applied to live data, see the [Gaussian smoothing section of the interactive W\&B report](https://wandb.ai/carey/smoothing-example/reports/W-B-Smoothing-Features--Vmlldzo1MzY3OTc#3.-gaussian-smoothing).

<Frame>
  <img alt="Line plot with Gaussian smoothing applied" />
</Frame>

## Running average smoothing

Running average is a smoothing algorithm that replaces a point with the average of points in a window before and after the given `x` value. See ["Boxcar Filter" on Wikipedia](https://en.wikipedia.org/wiki/Moving_average). The selected parameter for running average specifies the number of points to consider in the moving average.

If your points are spaced unevenly on the x-axis, use Gaussian smoothing instead, because a fixed-width window can produce misleading averages when point density varies.

To see this algorithm applied to live data, see the [running average section of the interactive W\&B report](https://wandb.ai/carey/smoothing-example/reports/W-B-Smoothing-Features--Vmlldzo1MzY3OTc#4.-running-average).

<Frame>
  <img alt="Line plot with running average smoothing applied" />
</Frame>

## Exponential moving average (EMA) smoothing

The exponential moving average (EMA) smoothing algorithm is a heuristic technique for smoothing time series data using the exponential window function. For details about the technique, see [Exponential Smoothing](https://www.wikiwand.com/en/Exponential_smoothing). The range is 0 to 1. A debias term is added so that early values in the time series aren't biased towards zero.

In most cases, EMA smoothing applies to a full scan of history, rather than bucketing first before smoothing. This typically produces more accurate smoothing.

In the following situations, EMA smoothing is applied after bucketing instead:

* Sampling
* Grouping
* Expressions
* Non-monotonic x-axes
* Time-based x-axes

The following sample code shows how this works under the hood:

```javascript theme={null}
  data.forEach(d => {
    const nextVal = d;
    last = last * smoothingWeight + (1 - smoothingWeight) * nextVal;
    numAccum++;
    debiasWeight = 1.0 - Math.pow(smoothingWeight, numAccum);
    smoothedData.push(last / debiasWeight);
```

To see this algorithm applied to live data, see the [EMA section of the interactive W\&B report](https://wandb.ai/carey/smoothing-example/reports/W-B-Smoothing-Features--Vmlldzo1MzY3OTc).

<Frame>
  <img alt="Line plot with EMA smoothing applied" />
</Frame>

## Hide original data

Compare the smoothed line to the raw data to judge how aggressively smoothing alters the signal. By default, the original unsmoothed data displays in the plot as a faint line in the background. Click **Show Original** to turn this off.

<Frame>
  <img alt="Toggling the display of the original unsmoothed data in a line plot" />
</Frame>


# Media panels
Source: https://docs.wandb.ai/models/app/features/panels/media

Add and configure media panels for images, video, audio, 3D objects, and point clouds in a W&B workspace.

A media panel visualizes [logged keys for media objects](/models/track/log/media/), including 3D objects, audio, images, video, or point clouds. This page shows how to add and manage media panels in a workspace.

<Frame>
  <img alt="Demo of a media panel" />
</Frame>

## Add a media panel

You can add a media panel using Quick Add (which applies the default configuration) or by adding it manually so you can configure it as you create it. In either case, you can add the panel globally to the workspace or to a specific section.

To add a media panel for a logged key using the default configuration, use Quick Add:

1. **Global**: Click **Add panels** in the control bar near the panel search field.
2. **Section**: Click the section's **action (<Icon icon="ellipsis" />)** menu, then click **Add panels**.
3. In the list of available panels, find the key for the panel, then click **Add**. Repeat this step for each media panel you want to add, then click the **X** at the top right to close the **Quick Add** list.
4. Optionally, [configure the panel](#configure-a-media-panel).

To add a media panel manually so that you can configure it during creation:

1. **Global**: Click **Add panels** in the control bar near the panel search field.
2. **Section**: Click the section's **action (<Icon icon="ellipsis" />)** menu, then click **Add panels**.
3. Click the **Media** section to expand it.
4. Select the type of media the panel visualizes, 3D objects, images, video, or audio. The panel configuration screen displays. Configure the panel, then click **Apply**. Refer to [Configure a media panel](#configure-a-media-panel).

## Configure a media panel

Panels for all media types have the same options, so the following steps apply regardless of which media type you're working with.

When you add a media panel manually, its configuration page opens after you select the type of media. To update the configuration for an existing panel, hover over the panel, then click the gear icon that appears at the top right. The following sections describe the settings available in each tab.

### Overlays

This tab appears for images and point clouds logged with segmentation masks or bounding boxes. Use it to:

* Search and filter overlays by name.
* Customize overlay colors.

### Sync

This tab appears in the workspace and section settings, and provides the following settings:

* **Sync slider by key**: Configure whether the step sliders for videos in the section move in sync.
* **Autoplay videos**: Configure whether videos start playing when the page loads.
* **Loop videos**: Configure whether videos in the section restart automatically and play continuously until stopped. Not customizable at the section level. Appears only if the workspace has video media panels.

### Display

Customize the panel's overall appearance and behavior:

* Configure the panel's title.
* Select the media keys to visualize.
* Customize the panel's slider and playback behavior.
  * Configure the slider key, which defaults to **Step**.
  * Set **Stride length** to the number of steps to advance for each click of the slider.
  * Turn on or off **Snap to existing step**. If it's turned on, the stepper advances to the next existing step after **Stride length**. Otherwise, it advances by **Stride length** even if that doesn't align with an existing step.
* **Images**: Turn on or off smoothing.
* **Videos**: Configure the video seek step interval, which controls how far forward or backward to skip when using the left and right arrow keys.
* **3d objects**: Configure the background color and point color.

### Layout

Customize the display of the panel's individual items:

* Optionally limit the **Max runs to include** in the panel.
* Optionally specify a **Media display limit** to limit the number of media items to include per run.
* Set **Panel mode**:
  * **Gallery** (default): Specify the number of columns and whether the grid shows a single item per run (default), step, or index.
  * **Grid**: Specify the number of columns, the x axis (defaults to **Log Step**), and the y axis (defaults to **Run**).
  * **Compare**: Compare up to four media items side by side, optionally fanning out by step or index. See [Compare mode](#compare-mode) for details.
* **Use original size**: If off (default), W\&B scales images and videos in the panel to appear the same size. No effect in **Compare** mode.
* Configure media fit and scaling behavior:
  * **Images**: Choose one of **Original image**, **Original size**, or **Fit to available space**. With the latter, W\&B scales images in the panel to take up as much room as possible. In **Gallery** mode, each image fills the panel width. In **Grid** mode, each image fills the grid width. No effect in **Compare** mode.
  * **Videos**: Toggle **Use original size** to control whether to scale media to appear the same size. If inactive (default), W\&B scales videos in the panel to take up as much room as possible. In **Gallery** mode, each video fills the panel width. In **Grid** mode, each video fills the grid width. No effect in **Compare** mode.
* **Right-handed system**: If on, point clouds use the right-handed system for plotting points, rather than the default left-handed system.

### All media panels in a section

Configure section-level defaults when you want every media panel in a section to share the same settings, overriding any workspace-level defaults.

To customize the default settings for all media panels in a section, overriding workspace settings for media panels:

1. Click the section's gear icon to open its settings.
2. Click **Media settings**.
3. Within the drawer that appears, click the **Display**, **Layout**, or **Sync** tab to configure the default media settings for the section. You can configure settings for images, videos, audio, and 3D objects. The settings that appear depend on the section's current media panels.

Refer to [Configure a media panel](#configure-a-media-panel) for details about a specific setting for **Display** or **Layout** media setting. The **Sync** tab is available only at the section or workspace level, not for individual media panels.

When **Step slider syncing** is turned on, the section's media panels with the same step slider are kept in sync. To turn on step slider syncing:

1. Click the **Sync** tab.
2. Turn on **Sync slider by key (Step)**.

### All media panels in a workspace

Configure workspace-level defaults when you want a consistent baseline for media panels across every section in the workspace.

To customize the default settings for all media panels in a workspace:

1. Click the workspace's settings, which has a gear with the label **Settings**.
2. Click **Media settings**.
3. Within the drawer that appears, click the **Display** or **Layout** tab to configure the default media settings for the workspace. You can configure settings for images, videos, audio, and 3D objects. The settings that appear depend on the workspace's current media panels.

Except for the **Sync** tab, refer to [Configure a media panel](#configure-a-media-panel) for details about a setting. The **Sync** tab is available only at the section or workspace level, not for individual media panels.

When **Step slider syncing** is turned on, the section's media panels with the same step slider are kept in sync. To turn on step slider syncing:

1. Click the **Sync** tab.
2. Turn on **Sync slider by key (Step)**.

For details about each setting, refer to [Configure a media panel](#configure-a-media-panel).

## Compare mode

Compare mode lets you select two to four images or videos from any combination of runs, steps, or indices and view them in a single grid. You can compare media without downloading one or more files or switching tabs.

Use compare mode to:

* Confirm that a new checkpoint or run produces better outputs than a baseline or previous checkpoint.
* Catch regressions by comparing the current run to a known good checkpoint at the same step or index.
* Validate model outputs against ground truth by viewing reference and generated media side by side.
* Review how generated images or videos change across training steps or across different checkpoints or runs.

Compare mode is useful for image or video generation workflows like diffusion or generative models, multimodal evaluation, vision experiments, and other workflows where you log media and need to compare it across runs, steps, or indices.

To use compare mode:

1. Open a media panel in your workspace (for images or video).
2. Set **Panel mode** to **Compare**.
3. Configure the grid's dimensions and density by specifying the number of columns and the fan-out mode. The best grid geometry depends on how your runs log media.
   * **Number of columns**: How many items to show per row, from `1` to `4`.
   * **Fan out**: Control the granularity of media to display per grid row. The best setting depends on how many media items you log per step per run, and whether you need a quick spot check or deep analysis.
     * **None** (default): No fan-out. Each grid column represents a run, and each grid row represents a logged image or video.
       <Frame>
         <img alt="Media panel in compare mode with two runs and no fan out" />
       </Frame>
     * **Step**: Fan out along the step axis. Each grid column represents a run, and each grid row shows the first image or video logged at a run:step combination.
       <Frame>
         <img alt="Media panel in compare mode with two runs and fan out by step" />
       </Frame>
     * **Index**: Fan out along the index axis, useful when you log several items per step (for example, `run.log({"images": [img0, img1, img2, img3]})`). Each row cell shows the image or video logged at a given run:step:index combination.
       <Frame>
         <img alt="Media panel in compare mode with two runs and fan out by index" />
       </Frame>
4. Configure whether a given variable affects a single tile (**Individual**) or all tiles (**Linked**).
   * **Media key**: Defaults to **Linked**.
   * **Run**: Defaults to **Individual**.
   * **Step**: Defaults to **Linked**.
   * **Index**: Defaults to **Linked**.

W\&B saves your selection so you can return to the same comparison later without reconfiguring the panel.

## Interact with a media panel

After you add and configure a media panel, use the following interactions to step through, compare, and inspect logged media:

* To configure a media panel, hover over it and click the gear icon at the top.
* To move a media panel's step slider, use **Cmd + left or right arrow key** (macOS) or **Ctrl + left or right arrow key** (Windows or Linux). If **Sync slider by key** is turned on for the section or workspace, moving the step slider in one media panel also moves the step slider in other media panels with the same step slider key.
* Use the stepper at the top of a media panel to step through media runs. To move the step slider, use the UI controls.
* Click a media panel to view it in full-screen mode. Click the arrow button at the top of the panel to exit full-screen mode.
* To navigate through a section's panels without exiting full-screen mode, use either the **Previous** and **Next** buttons below the panel or the left and right arrow keys.
* When viewing an image in full-screen mode, click the zoom control at the top right to zoom in or out. By default, images zoom to fit. Click and drag to pan a zoomed-in image. You can also zoom in, zoom out, reset to 100% zoom, or zoom to fit using keyboard shortcuts. See [Keyboard shortcuts](/models/app/keyboard-shortcuts#media-panels).
* For an image that was logged with segmentation masks, you can customize their appearance or turn each one on or off. Hover over the panel, then click the lower gear icon.
* For an image or point cloud that was logged with bounding boxes, you can customize their appearance or turn each one on or off. Hover over the panel, then click the lower gear icon.
* Use the media controls to play, pause, or stop video playback. Use the left and right arrow keys to skip forward or backward by the configured **video seek step**. If **Sync video playback** is turned on, all videos in the section play in sync. If **Loop videos** is turned on, videos in the section restart automatically and play continuously until stopped.


# Parallel coordinates
Source: https://docs.wandb.ai/models/app/features/panels/parallel-coordinates

Compare results across machine learning experiments

Parallel coordinates charts summarize the relationship between large numbers of hyperparameters and model metrics at a glance.

<Frame>
  <img alt="Parallel coordinates plot" />
</Frame>

A parallel coordinates panel has the following components:

* **Axes**: Hyperparameters from [`wandb.Run.config`](/models/tables/evaluate-models) and metrics from [`wandb.Run.log()`](/models/tables/evaluate-models).
* **Lines**: Each line represents a single run. Hover over a line to see a tooltip with details about the run. All lines matching the current filters appear. If you turn off the eye icon for a run, its line is dimmed.

## Create a parallel coordinates panel

To add a parallel coordinates panel to your workspace:

1. Navigate to the landing page for your workspace.
2. Click **Add Panels**.
3. Select **Parallel coordinates**.

## Panel settings

To configure the panel, click the edit button in the upper-right corner of the panel.

You can adjust the following settings:

* **Tooltip**: On hover, a legend appears with information about each run.
* **Titles**: Edit the axis titles to make them more readable.
* **Gradient**: Customize the gradient to any color range.
* **Log scale**: Set each axis to use a log scale independently.
* **Flip axis**: Switch the axis direction. This is useful when you have both accuracy and loss as columns.

For an interactive example, see [a live parallel coordinates panel](https://app.wandb.ai/example-team/sweep-demo/reports/Zoom-in-on-Parallel-Coordinates-Charts--Vmlldzo5MTQ4Nw).


# Parameter importance
Source: https://docs.wandb.ai/models/app/features/panels/parameter-importance

Visualize the relationships between your model's hyperparameters and output metrics

The parameter importance panel helps you discover which of your hyperparameters are the best predictors of, and highly correlated to, desirable values of your metrics. Use this panel to focus future hyperparameter searches on the parameters that matter most for model performance.

<Frame>
  <img alt="Parameter importance panel" />
</Frame>

**Correlation** is the linear correlation between the hyperparameter and the chosen metric (in this case, `val_loss`). A high correlation means that when the hyperparameter has a higher value, the metric also has higher values, and vice versa. Correlation is a useful metric, but it can't capture second-order interactions between inputs, and it can get messy when you compare inputs with different ranges.

W\&B also calculates an **importance** metric. W\&B trains a random forest with the hyperparameters as inputs and the metric as the target output, then reports the feature importance values for the random forest.

A conversation with [Jeremy Howard](https://twitter.com/jeremyphoward) inspired this technique. Jeremy pioneered the use of random forest feature importances to explore hyperparameter spaces at [Fast.ai](https://fast.ai). For more information about the motivation behind this analysis, see the [Fast.ai lesson 4 lecture](https://course18.fast.ai/lessonsml1/lesson4.html) and the [Fast.ai lesson 4 forum notes](https://forums.fast.ai/t/wiki-lesson-thread-lesson-4/7540).

The hyperparameter importance panel untangles the complicated interactions between highly correlated hyperparameters. In doing so, it helps you fine-tune your hyperparameter searches by showing you which of your hyperparameters matter the most for predicting model performance.

## Create a hyperparameter importance panel

To add a hyperparameter importance panel to your workspace:

1. Navigate to your W\&B project.
2. Select the **Add panels** button.
3. Expand the **CHARTS** dropdown, then choose **Parallel coordinates** from the dropdown.

<Note>
  If an empty panel appears, make sure that your runs are ungrouped.
</Note>

<Frame>
  <img alt="Automatic parameter visualization" />
</Frame>

With the parameter manager, you can manually set the visible and hidden parameters.

<Frame>
  <img alt="Manually setting the visible and hidden fields" />
</Frame>

## Interpret a hyperparameter importance panel

The following sections describe how to read the importance and correlation values shown in the panel so you can act on them in your next sweep.

<Frame>
  <img alt="Feature importance analysis" />
</Frame>

This panel shows you all the parameters passed to the [`wandb.Run.config`](/models/track/config/) object in your training script. It also shows the feature importances and correlations of these config parameters with respect to the model metric you select (`val_loss` in this case).

### Importance

The importance column shows you the degree to which each hyperparameter was useful in predicting the chosen metric. Imagine a scenario where you start tuning many hyperparameters and use this plot to focus on which ones merit further exploration. You can then limit subsequent sweeps to the most important hyperparameters, which helps you find a better model faster and at lower cost.

<Note>
  W\&B calculates importances using a tree-based model rather than a linear model, because tree-based models are more tolerant of both categorical data and data that isn't normalized.
</Note>

In the preceding image, you can see that `epochs`, `learning_rate`, `batch_size`, and `weight_decay` were important.

### Correlations

Correlations capture linear relationships between individual hyperparameters and metric values. They answer the question of whether a relationship exists between using a hyperparameter, such as the SGD optimizer, and the `val_loss` (in this case, the answer is yes). Correlation values range from -1 to 1, where positive values represent positive linear correlation, negative values represent negative linear correlation, and a value of 0 represents no correlation. Generally, a value greater than 0.7 in either direction represents strong correlation.

You might use this graph to further explore the values that have a higher correlation to your metric (in this case you might pick stochastic gradient descent or `adam` over `rmsprop` or `nadam`) or train for more epochs.

<Note>
  * Correlations show evidence of association, not necessarily causation.
  * Correlations are sensitive to outliers, which might turn a strong relationship into a moderate one, especially when the sample size of hyperparameters tried is small.
  * Correlations only capture linear relationships between hyperparameters and metrics. Correlations don't capture strong polynomial relationships.
</Note>

The disparities between importance and correlations result because importance accounts for interactions between hyperparameters, whereas correlation only measures the effects of individual hyperparameters on metric values. Correlations also capture only linear relationships, whereas importances can capture more complex ones.

Together, importance and correlations help you understand how your hyperparameters influence model performance.


# Query panels overview
Source: https://docs.wandb.ai/models/app/features/panels/query-panels

Some features on this page are in beta, hidden behind a feature flag. Add `weave-plot` to your bio on your profile page to unlock all related features.

<Note>
  Looking for W\&B Weave, W\&B's suite of tools for generative AI application building? See the [Weave documentation](https://wandb.github.io/weave/?utm_source=wandb_docs\&utm_medium=docs\&utm_campaign=weave-nudge).
</Note>

Use query panels to query and interactively visualize your data. With a query panel, you can pull specific runs, artifacts, tables, and other W\&B objects into a single view and explore them as tables or plots without leaving your workspace or report. This page is for users who want to compose ad hoc queries against logged W\&B data and surface the results inside a workspace or report.

A query panel combines three pieces:

* **[Expression](#expressions)**: The data you select.
* **[Configuration](#configurations)**: Optional settings for the panel, such as the panel type and options from the gear menu.
* **[Result panel](#result-panels)**: How to show the results, such as in a table or plot.

For a set of interactive examples you can try, see a public [Query panel examples report](https://wandb.ai/luis_team_test/weave_example_queries/reports/Query-Panel-Examples---Vmlldzo1NzIxOTY2). For a guided walkthrough of query syntax, see the [Query panel tutorial report](https://wandb.ai/luis_team_test/weave_example_queries/reports/Weave-queries---Vmlldzo1NzIxOTY2). Generated types and ops are listed in the [query expression language overview](/models/ref/query-panel).

<Frame>
  <img alt="Query panel" />
</Frame>

## Create a query panel

Add a query panel so you have a surface to write expressions against and visualize the results. You can add one to a project workspace or within a report.

<Tabs>
  <Tab title="Project workspace">
    1. Navigate to your project's workspace.
    2. In the upper-right corner, click **Add panel**.
    3. From the dropdown menu, select **Query panel**.
  </Tab>

  <Tab title="W&B Report">
    * Type and select **/Query panel**.

    <Frame>
      <img alt="Query panel option" />
    </Frame>

    Alternatively, you can associate a query with a set of runs:

    1. Within your report, type and select **/Panel grid**.
    2. Click the **Add panel** button.
    3. From the dropdown menu, select **Query panel**.
  </Tab>
</Tabs>

## Query components

The following sections describe the three pieces that make up a query panel: the expression that selects data, the configuration that controls how the panel behaves, and the result panel that renders the output.

### Expressions

Use query expressions to query your data stored in W\&B such as runs, artifacts, models, tables, and more.

#### Example: Query a table

Suppose you want to query a W\&B Table. In your training code, you log a table called `"cifar10_sample_table"`:

```python theme={null}
import wandb
with wandb.init() as run:
  run.log({"cifar10_sample_table":[MY-TABLE]})
```

Within the query panel, you can query your table with:

```python theme={null}
runs.summary["cifar10_sample_table"]
```

<Frame>
  <img alt="Table query expression" />
</Frame>

Breaking this down:

* `runs` is a variable automatically injected in query panel expressions when the query panel is in a workspace. Its value is the list of runs visible for that workspace. For details about the different attributes available within a run, see [Understanding the different attributes](/models/track/public-api-guide/#understanding-the-different-attributes).
* `summary` is an op that returns the Summary object for a run. Ops are *mapped*, meaning this op is applied to each run in the list, resulting in a list of Summary objects.
* `["cifar10_sample_table"]` is a Pick op (denoted with brackets) with a key of `cifar10_sample_table`. Because Summary objects act like dictionaries or maps, this operation picks that field from each Summary object.

### Configurations

In the upper-left corner of the panel, click the gear icon to expand the query configuration. The configuration lets you set the type of panel and the parameters for the result panel.

<Frame>
  <img alt="Panel configuration menu" />
</Frame>

#### Panel options

The configuration menu can include options that change how the panel combines or loads table-style results. Exact labels and availability depend on your expression and panel type. For concrete setups, see the [Query panel examples report](https://wandb.ai/luis_team_test/weave_example_queries/reports/Query-Panel-Examples---Vmlldzo1NzIxOTY2).

**Concat**

Use **Concat** in the configuration when you want the panel to merge compatible table-style results and treat them as a single table for viewing and downstream operations. Expression-level row merging (for example, `concat` and `join` in the query) is separate from this setting. For more information, see [Combine tables in expressions](#combine-tables-in-expressions).

**Paginate**

Use **Paginate** when a table result might be too large to render at once. Pagination loads rows in chunks so the panel stays responsive. Pair this option with expressions that return large row lists. For patterns that work well with pagination, see the [Query panel examples report](https://wandb.ai/luis_team_test/weave_example_queries/reports/Query-Panel-Examples---Vmlldzo1NzIxOTY2).

### Result panels

The query result panel renders the result of the query expression, using the selected query panel, configured to display the data in an interactive form. The following images show a Table and a Plot of the same data.

<Frame>
  <img alt="Table result panel" />
</Frame>

<Frame>
  <img alt="Plot result panel" />
</Frame>

### Step through run history

In tables and plots built from `runs` or `runs.history`, the app can show a **step** control (for example, a slider) so you can move through logged steps and inspect metrics, text, or media over the course of your runs. After you change the expression, edit the query panel's configuration and change **Render As** to **Stepper**. The control can follow a different metric instead of `_step` if it better matches how you logged data. For sample expressions, see the [Query panel examples report](https://wandb.ai/luis_team_test/weave_example_queries/reports/Query-Panel-Examples---Vmlldzo1NzIxOTY2).

## Basic operations

After a query panel renders results, you can refine what you see by sorting, filtering, mapping, or grouping the rows. The following are common operations you can perform within your query panels.

### Sort

Sort from the column options:

<Frame>
  <img alt="Column sort options" />
</Frame>

### Filter

You can either filter directly in the query or use the filter button in the upper-left corner of the panel.

<Frame>
  <img alt="Query filter syntax" />
</Frame>

<Frame>
  <img alt="Filter button" />
</Frame>

### Map

Map operations iterate over lists and apply a function to each element in the data. You can do this directly with a panel query or by inserting a new column from the column options.

<Frame>
  <img alt="Map operation query" />
</Frame>

<Frame>
  <img alt="Map column insertion" />
</Frame>

### Group by

You can group by using a query or from the column options.

<Frame>
  <img alt="Group by query" />
</Frame>

<Frame>
  <img alt="Group by column options" />
</Frame>

### Combine tables in expressions

Use `concat`, `join`, and related ops in your expression when you need to stack or merge row lists from tables. See [Join](#join) for a full example. The **Concat** and **Paginate** items in [Panel options](#panel-options) are separate controls for how the UI merges and loads table results.

### Join

You can also join tables directly in the query. Consider the following query expression:

```python theme={null}
project("luis_team_test", "weave_example_queries").runs.summary["short_table_0"].table.rows.concat.join(\
project("luis_team_test", "weave_example_queries").runs.summary["short_table_1"].table.rows.concat,\
(row) => row["Label"],(row) => row["Label"], "Table1", "Table2",\
"false", "false")
```

<Frame>
  <img alt="Table join operation" />
</Frame>

The table on the left is generated from:

```python theme={null}
project("luis_team_test", "weave_example_queries").\
runs.summary["short_table_0"].table.rows.concat.join
```

The table on the right is generated from:

```python theme={null}
project("luis_team_test", "weave_example_queries").\
runs.summary["short_table_1"].table.rows.concat
```

Where:

* `(row) => row["Label"]` are selectors for each table, determining which column to join on.
* `"Table1"` and `"Table2"` are the names of each table when joined.
* `true` and `false` are for left and right inner/outer join settings.

## Runs object

Use query panels to access the `runs` object. Run objects store records of your experiments. For more details, see [Accessing runs object](https://wandb.ai/luis_team_test/weave_example_queries/reports/Weave-queries---Vmlldzo1NzIxOTY2#3.-accessing-runs-object). As a quick overview, the `runs` object has the following available:

* `summary`: A dictionary of information that summarizes the run's results. The summary can contain scalars like accuracy and loss, or large files. By default, `wandb.Run.log()` sets the summary to the final value of a logged time series. You can set the contents of the summary directly. Think of the summary as the run's outputs.
* `history`: A list of dictionaries meant to store values that change while the model is training, such as loss. The command `wandb.Run.log()` appends to this object.
* `config`: A dictionary of the run's configuration information, such as the hyperparameters for a training run or the preprocessing methods for a run that creates a dataset artifact. Think of these as the run's inputs.

<Frame>
  <img alt="Runs object structure" />
</Frame>

## Access artifacts

Artifacts are a core concept in W\&B. They are a versioned, named collection of files and directories. Use artifacts to track model weights, datasets, and any other file or directory. W\&B stores artifacts, and you can download them or use them in other runs. For more details and examples, see [Accessing artifacts](https://wandb.ai/luis_team_test/weave_example_queries/reports/Weave-queries---Vmlldzo1NzIxOTY2#4.-accessing-artifacts). You normally access artifacts from the `project` object:

* `project.artifactVersion()`: Returns the specific artifact version for a given name and version within a project.
* `project.artifact("")`: Returns the artifact for a given name within a project. You can then use `.versions` to get a list of all versions of this artifact.
* `project.artifactType()`: Returns the `artifactType` for a given name within a project. You can then use `.artifacts` to get a list of all artifacts with this type.
* `project.artifactTypes`: Returns a list of all artifact types under the project.

<Frame>
  <img alt="Artifact access methods" />
</Frame>


# Embed objects
Source: https://docs.wandb.ai/models/app/features/panels/query-panels/embedding-projector

W&B's Embedding Projector lets you plot multi-dimensional embeddings on a 2D plane using common dimension reduction algorithms like PCA, UMAP, and t-SNE.

<Frame>
  <img alt="Embedding projector" />
</Frame>

[Embeddings](https://developers.google.com/machine-learning/crash-course/embeddings/video-lecture) represent objects such as people, images, posts, or words with a list of numbers, sometimes referred to as a *vector*. In machine learning and data science use cases, you can generate embeddings using a variety of approaches across a range of applications. This page assumes you're familiar with embeddings and want to visually analyze them inside W\&B.

This guide shows you how to log embeddings to W\&B and use the Embedding Projector to plot them on a 2D plane with dimension reduction algorithms such as PCA, UMAP, and t-SNE. Visualizing embeddings this way helps you explore clusters, inspect relationships between data points, and validate that your embeddings capture the structure you expect.

## Embedding examples

The following resources demonstrate the Embedding Projector in action before you try it yourself:

* [Live interactive demo report](https://wandb.ai/timssweeney/toy_datasets/reports/Feature-Report-W-B-Embeddings-Projector--VmlldzoxMjg2MjY4?accessToken=bo36zrgl0gref1th5nj59nrft9rc4r71s53zr2qvqlz68jwn8d8yyjdz73cqfyhq)
* [Example Colab](https://colab.research.google.com/drive/1DaKL4lZVh3ETyYEM1oJ46ffjpGs8glXA#scrollTo=D--9i6-gXBm_)

### Hello world

This minimal example shows the smallest amount of code needed to log embeddings and view them in the projector. W\&B lets you log embeddings using the `wandb.Table` class. Consider the following example of three embeddings, each consisting of five dimensions:

```python theme={null}
import wandb

with wandb.init(project="embedding_tutorial") as run:
  embeddings = [
      # D1   D2   D3   D4   D5
      [0.2, 0.4, 0.1, 0.7, 0.5],  # embedding 1
      [0.3, 0.1, 0.9, 0.2, 0.7],  # embedding 2
      [0.4, 0.5, 0.2, 0.2, 0.1],  # embedding 3
  ]
  run.log(
      {"embeddings": wandb.Table(columns=["D1", "D2", "D3", "D4", "D5"], data=embeddings)}
  )
  run.finish()
```

After you run the preceding code, the W\&B dashboard contains a new Table with your data. Select **2D Projection** from the upper-right panel selector to plot the embeddings in two dimensions. W\&B automatically selects smart defaults, which you can override in the configuration menu by clicking the gear icon. In this example, W\&B uses all five available numeric dimensions.

<Frame>
  <img alt="2D projection example" />
</Frame>

### Digits MNIST

The next example demonstrates a more realistic workflow with higher-dimensional data and richer overlays. While the preceding example shows the basic mechanics of logging embeddings, you typically work with many more dimensions and samples. Consider the MNIST Digits dataset ([UCI ML hand-written digits dataset](https://archive.ics.uci.edu/ml/datasets/Optical+Recognition+of+Handwritten+Digits)) made available through [SciKit-Learn](https://scikit-learn.org/stable/modules/generated/sklearn.datasets.load_digits.html). This dataset has 1,797 records, each with 64 dimensions. The problem is a 10-class classification use case. You can also convert the input data to an image for visualization.

```python theme={null}
import wandb
from sklearn.datasets import load_digits

with wandb.init(project="embedding_tutorial") as run:

  # Load the dataset
  ds = load_digits(as_frame=True)
  df = ds.data

  # Create a "target" column
  df["target"] = ds.target.astype(str)
  cols = df.columns.tolist()
  df = df[cols[-1:] + cols[:-1]]

  # Create an "image" column
  df["image"] = df.apply(
      lambda row: wandb.Image(row[1:].values.reshape(8, 8) / 16.0), axis=1
  )
  cols = df.columns.tolist()
  df = df[cols[-1:] + cols[:-1]]

  run.log({"digits": df})
```

After you run the preceding code, the UI again presents a Table. Select **2D Projection** to configure the embedding definition, coloring, algorithm (PCA, UMAP, t-SNE), algorithm parameters, and overlay. In this case, W\&B shows the image when you hover over a point. These are all smart defaults, and you should see something similar with a single click of **2D Projection**. [Interact with this embedding tutorial example](https://wandb.ai/timssweeney/embedding_tutorial/runs/k6guxhum?workspace=user-timssweeney).

<Frame>
  <img alt="MNIST digits projection" />
</Frame>

## Logging options

The following sections describe the supported ways to structure embedding data when you log it to W\&B. You can log embeddings in several formats:

* **Single embedding column:** Often your data is already in a matrix-like format. In this case, you can create a single embedding column, where the data type of the cell values can be `list[int]`, `list[float]`, or `np.ndarray`.
* **Multiple numeric columns:** The preceding two examples use this approach and create a column for each dimension. W\&B accepts Python `int` or `float` for the cells.

<Frame>
  <img alt="Single embedding column" />
</Frame>

<Frame>
  <img alt="Multiple numeric columns" />
</Frame>

Just like all tables, you have several options for how to construct the table:

* Directly from a **dataframe** using `wandb.Table(dataframe=df)`.
* Directly from a **list of data** using `wandb.Table(data=[...], columns=[...])`.
* Build the table **incrementally row by row** (great if you have a loop in your code). Add rows to your table using `table.add_data(...)`.
* Add an **embedding column** to your table (great if you have a list of predictions in the form of embeddings): `table.add_col("col_name", ...)`.
* Add a **computed column** (great if you have a function or model you want to map over your table): `table.add_computed_columns(lambda row, ndx: {"embedding": model.predict(row)})`.

## Plotting options

After you log your embeddings, you can adjust how they are projected and rendered. After you select **2D Projection**, click the gear icon to edit the rendering settings. Besides selecting the intended columns (see preceding sections), you can select an algorithm of interest along with the desired parameters. The following images show the parameters for UMAP and t-SNE.

<Frame>
  <img alt="UMAP parameters" />
</Frame>

<Frame>
  <img alt="t-SNE parameters" />
</Frame>

<Note>
  W\&B downsamples to a random subset of 1,000 rows and 50 dimensions for all three algorithms.
</Note>


# Compare run metrics
Source: https://docs.wandb.ai/models/app/features/panels/run-comparer

Use the Run Comparer panel to view and compare configuration and metric differences across W&B experiment runs.

Use the Run Comparer to see differences and similarities across runs in your project. This helps you quickly identify how configuration changes affect metrics so you can decide which experiments to pursue further.

## Add a Run Comparer panel

Add the Run Comparer panel to your workspace before you can use it to compare runs.

1. Select the **Add panels** button in the top right corner of the page.
2. From the **Evaluation** section, select **Run comparer**.

## Use Run Comparer

Run Comparer shows the configuration and logged metrics for the first 10 visible runs in the project, one column per run. After you add the panel, use the following options to explore and refine your comparison:

* To change the runs to compare, search, filter, group, or sort the list of runs on the left. The Run Comparer updates automatically.
* To filter or search for a configuration key or a metadata key such as the Python version or the run's creation time, use the search field at the top of the Run Comparer.
* To see differences and hide identical values, toggle **Diff only** at the top of the panel.
* To adjust the column width or row height, use the formatting buttons at the top of the panel.
* To copy any configuration or metric's value, hover over the value, then click the copy button. The entire value is copied, even if it's too long to display on the screen.

<Note>
  By default, Run Comparer doesn't differentiate runs with different values for [`job_type`](/models/ref/python/functions/init). This means you can compare runs that aren't comparable within a project. For example, you could compare a training run to a model evaluation run. A training run could contain run logs, hyperparameters, training loss metrics, and the model itself. An evaluation run could use the model to check the model's performance on new training data.

  When you search, filter, group, or sort the list of runs in the Runs Table, the Run Comparer automatically updates to compare the first 10 runs. Filter or search within the Runs Table to compare similar runs, such as by filtering or sorting the list by `job_type`. Learn more about [filtering runs](/models/runs/filter-runs/).
</Note>


# Scatter plots
Source: https://docs.wandb.ai/models/app/features/panels/scatter-plot

Create and customize scatter plots in W&B to compare runs and visualize relationships between experiment metrics.

Use scatter plots in W\&B to compare runs and visualize relationships between experiment metrics. Scatter plots help you spot trends, outliers, and trade-offs across many runs, which is useful when tuning hyperparameters or comparing model variants.

## Use case

Use scatter plots to compare multiple runs and visualize the performance of an experiment. With a scatter plot, you can:

* Plot lines for minimum, maximum, and average values.
* Customize metadata tooltips.
* Control point colors.
* Adjust axis ranges.
* Use a log scale for the axes.
* Configure how the **frontier line** and related points are labeled and emphasized. See [Frontier line display options](#frontier-line-display-options).

## Example

The following example shows a scatter plot displaying validation accuracy for different models over several weeks of experimentation. The tooltip includes batch size, dropout, and axis values. A line also shows the running average of validation accuracy.

[See a live example →](https://app.wandb.ai/l2k2/l2k/reports?view=carey%2FScatter%20Plot)

<Frame>
  <img alt="Validation accuracy scatter plot" />
</Frame>

## Create a scatter plot

To create a scatter plot in the W\&B UI:

1. Navigate to the **Workspaces** tab.
2. In the **Charts** panel, click the **action (<Icon icon="ellipsis" />)** menu.
3. From the pop-up menu, select **Add panels**.
4. In the **Add panels** menu, select **Scatter plot**.
5. Set the `x` and `y` axes to plot the data you want to view. Optionally, set maximum and minimum ranges for your axes or add a `z` axis.
6. Click **Apply** to create the scatter plot.

The new scatter plot appears in the **Charts** panel, displaying the data you configured.

## Frontier line display options

The *frontier line* on a scatter plot connects the highest and lowest y-axis values observed so far for the data in the plot. You can tune how the line and the surrounding points appear from the plot settings to emphasize the frontier.

To configure frontier line display options:

1. Hover over the scatter plot panel, then click the gear icon to open the panel settings drawer.
2. Select the **Annotations** tab.
3. Set the options:
   * **Show run labels**: Toggle whether to show a label with the run name or group name.
   * **Dim non-frontier points**: Toggle whether to reduce the visual weight of points that aren't on the frontier line so the frontier stands out.
4. Click **Apply** to save your changes.

The scatter plot updates to reflect your annotation choices.


# Keyboard shortcuts
Source: https://docs.wandb.ai/models/app/keyboard-shortcuts

Learn about the keyboard shortcuts available in W&B.

W\&B supports keyboard shortcuts to help you navigate and interact with experiments, workspaces, and data more efficiently. This guide covers keyboard shortcuts for the W\&B App and the W\&B LEET (Lightweight Experiment Exploration Tool) terminal UI.

<Tabs>
  <Tab title="App">
    The following tables list keyboard shortcuts available in the W\&B App, grouped by where they apply.

    ## Workspace management

    | Shortcut                                             | Description                                                                              |
    | ---------------------------------------------------- | ---------------------------------------------------------------------------------------- |
    | **Cmd+Z** (macOS) / **Ctrl+Z** (Windows/Linux)       | Undo a change you've made in the UI, such as a modification to the workspace or a panel. |
    | **Cmd+Shift+Z** (macOS) / **Ctrl+Y** (Windows/Linux) | Redo a change you undid in the workspace.                                                |

    ## Navigation

    | Shortcut                                       | Description                                                                                                                                                 |
    | ---------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- |
    | **Tab**                                        | Navigate between interactive elements.                                                                                                                      |
    | **Cmd+J** (macOS) / **Ctrl+J** (Windows/Linux) | Switch between the Workspaces and Runs tabs in the project sidebar.                                                                                         |
    | **Cmd+.** (macOS) / **Ctrl+.** (Windows/Linux) | Minimize or restore the Runs selector sidebar to reclaim screen space.                                                                                      |
    | **Cmd+K** (macOS) / **Ctrl+K** (Windows/Linux) | Open the quick search dialog to search across projects, runs, and other resources.                                                                          |
    | **Esc**                                        | Throughout the W\&B App, exit full-screen panel views, close settings drawers, dismiss the quick search dialog, close an editor, or dismiss other overlays. |

    ## Panel navigation

    | Shortcut                                                                                                 | Description                                                           |
    | -------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------- |
    | **Left Arrow / Right Arrow**                                                                             | When in full-screen mode, step through panels in a section.           |
    | **Esc**                                                                                                  | Exit full-screen panel view and return to the workspace.              |
    | **Cmd+Left Arrow / Cmd+Right Arrow** (macOS)<br />**Ctrl+Left Arrow / Ctrl+Right Arrow** (Windows/Linux) | When viewing a media panel in full-screen mode, move the step slider. |

    ## Media panels

    | Shortcut                                                  | Description                                                                                                 |
    | --------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------- |
    | **Cmd + +/-** (macOS)<br />**Ctrl + +/-** (Windows/Linux) | When viewing an image in full-screen, zoom in or out.<br />When zoomed in, click and drag to pan the image. |
    | **Cmd + 0** (macOS)<br />**Ctrl + 0** (Windows/Linux)     | When viewing an image in full-screen, reset to 100% zoom.                                                   |
    | **Shift + L**                                             | When viewing an image in full-screen, zoom to fit.                                                          |
    | **Click**                                                 | In a video panel, select a video, before skipping forward or backward using the keyboard.                   |
    | **Left Arrow / Right Arrow**                              | In a video panel, skip forward or backward by the configured **video seek step**.                           |

    ## Reports

    | Shortcut               | Description                                               |
    | ---------------------- | --------------------------------------------------------- |
    | **Delete / Backspace** | Remove the selected panel grid from the report.           |
    | **Enter**              | Insert a Markdown block after typing `/mark` in a report. |
    | **Esc**                | Exit the report editor.                                   |
    | **Tab**                | Navigate between interactive elements in a report.        |

    ## Notes

    Keep the following points in mind when using keyboard shortcuts in the W\&B App:

    * Most keyboard shortcuts use **Cmd** on macOS and **Ctrl** on Windows/Linux.
    * The W\&B App implements custom handling for some browser default shortcuts.
    * Some shortcuts are context-sensitive and only work in specific areas of the app.
  </Tab>

  <Tab title="LEET">
    The following keyboard shortcuts work in the W\&B LEET (Lightweight Experiment Exploration Tool) terminal UI, which lets you explore experiments without leaving the command line. To launch LEET, run `wandb beta leet` in your terminal. For more information, see [`wandb beta leet`](/models/ref/cli/wandb-beta/wandb-beta-leet/).
  </Tab>
</Tabs>


# Artifacts overview
Source: https://docs.wandb.ai/models/artifacts

Overview of W&B Artifacts, how they work, and how to get started using them.

<CardGroup>
  <ColabLink />

  <TryProductLink />
</CardGroup>

Use W\&B Artifacts to track and version data as the inputs and outputs of your [W\&B Runs](/models/runs). For example, a model training run might take in a dataset as input and produce a trained model as output. You can log hyperparameters, metadata, and metrics to a run, and you can use an artifact to log, track, and version the dataset used to train the model as input and another artifact for the resulting model checkpoints as output.

## Use cases

You can use artifacts throughout your entire ML workflow as inputs and outputs of [runs](/models/runs). You can use datasets, models, or even other artifacts as inputs for processing.

<Frame>
  <img alt="Artifacts workflow diagram with inputs and outputs for model training, data processing, and model evaluation" />
</Frame>

| Use Case               | Input                                  | Output                       |
| ---------------------- | -------------------------------------- | ---------------------------- |
| Model Training         | Dataset (training and validation data) | Trained Model                |
| Dataset Pre-Processing | Dataset (raw data)                     | Dataset (pre-processed data) |
| Model Evaluation       | Model + Dataset (test data)            | [W\&B Table](/models/tables) |
| Model Optimization     | Model                                  | Optimized Model              |

<Note>
  The following code snippets are meant to be run in order.
</Note>

## Create an artifact

Create an artifact with four lines of code:

1. Create a [W\&B run](/models/runs).
2. Create an artifact object with [`wandb.Artifact`](/models/ref/python/experiments/artifact).
3. Add one or more files, such as a model file or dataset, to the artifact object with `wandb.Artifact.add_file()`.
4. Log your artifact to W\&B with `wandb.Run.log_artifact()`.

For example, the following code snippet shows how to log a file called `dataset.h5` to an artifact called `example_artifact`:

```python theme={null}
import wandb

with wandb.init(project="artifacts-example", job_type="add-dataset") as run:
    artifact = wandb.Artifact(name="example_artifact", type="dataset")
    artifact.add_file(local_path="./dataset.h5", name="training_dataset")
    run.log_artifact(artifact)
```

* The `type` of the artifact affects how it appears in the W\&B platform. If you do not specify a `type`, it defaults to `unspecified`.
* Each label of the dropdown represents a different `type` parameter value. In the above code snippet, the artifact's `type` is `dataset`.

<Note>
  See the [track external files](/models/artifacts/track-external-files) page for information on how to add references to files or directories stored in external object storage, like an Amazon S3 bucket.
</Note>

## Download an artifact

Indicate the artifact you want to mark as input to your run with the [`wandb.Run.use_artifact()`](/models/ref/python/experiments/run#use_artifact) method.

Continuing from the previous code snippet, the following code example shows how to use the artifact called `example_artifact` that was created earlier:

```python theme={null}
with wandb.init(project="artifacts-example", job_type="add-dataset") as run:
    artifact = run.use_artifact("training_dataset:latest")  # returns a run object using the "my_data" artifact
```

This returns an artifact object.

Next, use the returned object to download all contents of the artifact:

```python theme={null}
datadir = artifact.download()  # downloads the full `my_data` artifact to the default directory.
```

<Note>
  You can pass a custom path into the `root` [parameter](/models/ref/python/experiments/artifact) to download an artifact to a specific directory. For alternate ways to download artifacts and to see additional parameters, see the guide on [downloading and using artifacts](/models/artifacts/download-and-use-an-artifact).
</Note>

## Next steps

* Learn how to [version](/models/artifacts/create-a-new-artifact-version) and [update](/models/artifacts/update-an-artifact) artifacts.
* Learn how to trigger downstream workflows or notify a Slack channel in response to changes to your artifacts with [automations](/models/automations).
* Learn about the [registry](/models/registry), a space that houses trained models.
* Explore the [Python SDK](/models/ref/python/experiments/artifact) and [CLI](/models/ref/cli/wandb-artifact) reference guides.


# Tutorial: Create, track, and use a dataset artifact
Source: https://docs.wandb.ai/models/artifacts/artifacts-walkthrough

Create, track, and use a dataset artifact with W&B.

This walkthrough demonstrates how to create, track, and use a dataset artifact.

## 1. Log into W\&B

Import the W\&B library and log in to W\&B. You will need to sign up for a free W\&B account if you have not done so already.

```python theme={null}
import wandb

wandb.login()
```

## 2. Initialize a run

Use [`wandb.init()`](/models/ref/python/functions/init) to initialize a run. This generates a background process to sync and log data. Provide a project name and a job type:

```python theme={null}
# Create a W&B Run. Here we specify 'dataset' as the job type since this example
# shows how to create a dataset artifact.
with wandb.init(project="artifacts-example", job_type="upload-dataset") as run:
    # Your code here
```

## 3. Create an artifact object

Create an artifact object with the [`wandb.Artifact()`](/models/ref/python/experiments/artifact). Provide a name for the artifact and a description of the file type for the `name` and `type` parameters, respectively.

For example, the following code snippet demonstrates how to create an artifact called `‘bicycle-dataset’` with a `‘dataset’` label:

```python theme={null}
artifact = wandb.Artifact(name="bicycle-dataset", type="dataset")
```

For more information about how to construct an artifact, see [Construct artifacts](./construct-an-artifact).

## 4. Add the dataset to the artifact

Add a file to the artifact. Common file types include models and datasets. The following example adds a dataset named `dataset.h5` that is saved locally on our machine to the artifact:

```python theme={null}
# Add a file to the artifact's contents
artifact.add_file(local_path="dataset.h5")
```

Replace the filename `dataset.h5` in the previous code snippet with the path to the file you want to add to the artifact.

## 5. Log the dataset

Use the W\&B run objects `wandb.Run.log_artifact()` method to both save your artifact version and declare the artifact as an [output of the run](/models/artifacts/explore-and-traverse-an-artifact-graph).

```python theme={null}
# Save the artifact version to W&B and mark it
# as the output of this run
run.log_artifact(artifact)
```

A `'latest'` [alias](/models/artifacts/create-a-custom-alias) is created by default when you log an artifact. For more information about artifact aliases and versions, see [Create a custom alias](./create-a-custom-alias) and [Create new artifact versions](./create-a-new-artifact-version), respectively.

Putting this together, you script so far should look like this:

```python theme={null}
import wandb

wandb.login()

with wandb.init(project="artifacts-example", job_type="upload-dataset") as run:
    artifact = wandb.Artifact(name="bicycle-dataset", type="dataset")
    artifact.add_file(local_path="dataset.h5")
    run.log_artifact(artifact)
```

## 6. Download and use the artifact

The following code example demonstrates the steps you can take to use an artifact you have logged and saved to the W\&B servers.

1. First, initialize a new run object with **`wandb.init()`.**
2. Second, use the run objects [`wandb.Run.use_artifact()`](/models/ref/python/experiments/run#use_artifact) method to tell W\&B what artifact to use. This returns an artifact object.
3. Third, use the artifacts [`wandb.Artifact.download()`](/models/ref/python/experiments/artifact#download) method to download the contents of the artifact.

```python theme={null}
# Create a W&B Run. Here we specify 'training' for 'type'
# because we will use this run to track training.
with wandb.init(project="artifacts-example", job_type="training") as run:

  # Query W&B for an artifact and mark it as input to this run
  artifact = run.use_artifact("bicycle-dataset:latest")

  # Download the artifact's contents
  artifact_dir = artifact.download()
```

Alternatively, you can use the Public API (`wandb.Api`) to export (or update data) data already saved in a W\&B outside of a Run. See [Track external files](./track-external-files) for more information.


# Create an artifact
Source: https://docs.wandb.ai/models/artifacts/construct-an-artifact

Create and log a W&B Artifact. Learn how to add one or more files or a URI reference to an Artifact.

Use the W\&B Python SDK to construct artifacts from [W\&B Runs](/models/ref/python/experiments/run). You can add [files, directories, URIs, and files from parallel runs to artifacts](#add-files-to-an-artifact). After you add a file to an artifact, save the artifact to the W\&B Server or [your own private server](/platform/hosting/hosting-options/self-managed). Each artifact is associated with a run.

For information on how to track external files, such as files stored in Amazon S3, see the [Track external files](./track-external-files) page.

## Construct an artifact

Construct a [W\&B Artifact](/models/ref/python/experiments/artifact) in three steps:

1. [Create an artifact Python object with `wandb.Artifact()`](/models/artifacts/construct-an-artifact#create-an-artifact-python-object-with-wandb-artifact)
2. [Add one or more files to the artifact](/models/artifacts/construct-an-artifact#add-one-or-more-files-to-the-artifact)
3. [Save your artifact to the W\&B server](/models/artifacts/construct-an-artifact#save-your-artifact-to-the-w\&b-server)

### Create an artifact Python object with `wandb.Artifact()`

Initialize the [`wandb.Artifact()`](/models/ref/python/experiments/artifact) class to create an artifact object. Specify the following parameters:

* **Name**: The name of your artifact. The name should be unique, descriptive, and memorable.
* **Type**: The type of artifact. The type should be simple, descriptive, and correspond to a single step of your machine learning pipeline. Common artifact types include `'dataset'` or `'model'`.

<Note>
  W\&B uses the "name" and "type" you provide to create a directed acyclic graph in the W\&B App. See the [Explore and traverse artifact graphs](./explore-and-traverse-an-artifact-graph) for more information.
</Note>

<Warning>
  Artifacts can not have the same name, regardless of type. In other words, you can not create an artifact named `cats` of type `dataset` and another artifact with the same name of type `model`.
</Warning>

You can optionally provide a description and metadata when you initialize an artifact object. For more information on available attributes and parameters, see the [`wandb.Artifact`](/models/ref/python/experiments/artifact) Class definition in the Python SDK Reference Guide.

Copy and paste the following code snippet to create an artifact object. Replace the `<name>` and `<type>` placeholders with your own values:

```python theme={null}
import wandb

# Create an artifact object
artifact = wandb.Artifact(name="<name>", type="<type>")
```

### Add one or more files to the artifact

[Add files, directories, external URI references (such as Amazon S3) and more](/models/artifacts/construct-an-artifact#add-files-to-an-artifact) to your artifact object.

To add a single file, use the artifact object's [`Artifact.add_file()`](/models/ref/python/experiments/artifact#add_file) method:

```python theme={null}
artifact.add_file(local_path="path/to/file.txt", name="<name>")
```

To add a directory, use the [`Artifact.add_dir()`](/models/ref/python/experiments/artifact#add_dir) method:

```python theme={null}
artifact.add_dir(local_path="path/to/directory", name="<name>")
```

See the next section, [Add files to an artifact](/models/artifacts/construct-an-artifact#add-files-to-an-artifact), for more information on how to add different file types to an artifact.

### Save your artifact to the W\&B server

Save your artifact to the W\&B server. Use the run object's [`wandb.Run.log_artifact()`](/models/ref/python/experiments/run#log_artifact) method to save the artifact.

```python theme={null}
with wandb.init(project="<project>", job_type="<job-type>") as run:
    run.log_artifact(artifact)
```

<Tip>
  **When to use to use `wandb.Run.log_artifact()`  or `Artifact.save()`**

  * Use `wandb.Run.log_artifact()` to create a new artifact and associate it with a specific run.
  * Use `Artifact.save()` to update an existing artifact without creating a new run.
</Tip>

Putting this all together, the following code snippet demonstrates how to create a dataset artifact, add a file to the artifact, and save the artifact to W\&B:

```python theme={null}
import wandb

artifact = wandb.Artifact(name="<name>", type="<type>")
artifact.add_file(local_path="path/to/file.txt", name="<name>")
artifact.add_dir(local_path="path/to/directory", name="<name>")

with wandb.init(project="<project>", job_type="<job-type>") as run:
    run.log_artifact(artifact)
```

Each time you log an artifact with the same name and type, W\&B creates a new version of that artifact. For more information, see [Create a new artifact version](/models/artifacts/create-a-new-artifact-version).

<Warning>
  W\&B performs calls `wandb.Run.log_artifact()` asynchronously for performant uploads. This can cause surprising behavior when logging artifacts in a loop. For example:

  ```python theme={null}
  with wandb.init() as run:
      for i in range(10):
          a = wandb.Artifact(name = "race",
              type="dataset",
              metadata={
                  "index": i,
              },
          )
          # ... add files to artifact a ...
          run.log_artifact(a)
  ```

  The artifact version **v0** might not have an index of 0 in its metadata because artifacts may be logged in an arbitrary order.
</Warning>

## Add files to an artifact

The following sections demonstrate how to add different types of objects to an artifact. Assume you have a directory with the following structure as you read through the examples:

```text theme={null}
root-directory
| - hello.txt
| - images/
| -- | cat.png
| -- | dog.png
| - checkpoints/
| -- | model.h5
| - models/
| -- | model.h5
```

### Add a single file

Use [`wandb.Artifact.add_file()`](/models/ref/python/experiments/artifact#method-artifact-add-file) to add a single local file to an artifact. Provide the local path to the file as the `local_path` parameter:

```python theme={null}
import wandb

# Initialize an artifact object
artifact = wandb.Artifact(name="<name>", type="<type>")

# Add a single file
artifact.add_file(local_path="path/file.format")
```

For example, suppose you had a file called `'hello.txt'` in your working local directory.

```python theme={null}
artifact.add_file("hello.txt")
```

The artifact now has the following content:

```text theme={null}
hello.txt
```

Optionally, pass a different name to the `name` parameter to rename the file within the artifact object itself. Continuing the previous example:

```python theme={null}
artifact.add_file(
    local_path="hello.txt", 
    name="new/path/hello_world.txt"
    )
```

The artifact is stored as:

```text theme={null}
new/path/hello_world.txt
```

The following table shows how different API calls produce different artifact contents:

| API Call                                                  | Resulting artifact  |
| --------------------------------------------------------- | ------------------- |
| `artifact.new_file('hello.txt')`                          | `hello.txt`         |
| `artifact.add_file('model.h5')`                           | `model.h5`          |
| `artifact.add_file('checkpoints/model.h5')`               | `model.h5`          |
| `artifact.add_file('model.h5', name='models/mymodel.h5')` | `models/mymodel.h5` |

### Add multiple files

Use the [`wandb.Artifact.add_dir()`](/models/ref/python/experiments/artifact#method-artifact-add-dir) method to add multiple files from a local directory to an artifact. Provide the local path to the directory as the `local_path` parameter.

```python theme={null}
import wandb

# Initialize an artifact object
artifact = wandb.Artifact(name="<name>", type="<type>")

# Add a local directory to the artifact
artifact.add_dir(local_path="path/file.format", name="optional-prefix")
```

The following table show how different API calls produce different artifact contents:

| API Call                                    | Resulting artifact                                                   |
| ------------------------------------------- | -------------------------------------------------------------------- |
| `artifact.add_dir('images')`                | <p><code>cat.png</code></p><p><code>dog.png</code></p>               |
| `artifact.add_dir('images', name='images')` | <p><code>images/cat.png</code></p><p><code>images/dog.png</code></p> |

### Add a URI reference

Artifacts track checksums and other information for reproducibility if the URI has a scheme that the W\&B library supports.

Add an external URI reference to an artifact with the [`wandb.Artifact.add_reference()`](/models/ref/python/experiments/artifact#method-artifact-add-reference) method. Replace the `'uri'` string with your own URI. Optionally pass the desired path within the artifact for the name parameter.

```python theme={null}
# Add a URI reference
artifact.add_reference(uri="uri", name="optional-name")
```

Artifacts support the following URI schemes:

* `http(s)://`: A path to a file accessible over HTTP. The artifact will track checksums in the form of etags and size metadata if the HTTP server supports the `ETag` and `Content-Length` response headers.
* `s3://`: A path to an object or object prefix in S3. The artifact will track checksums and versioning information (if the bucket has object versioning enabled) for the referenced objects. Object prefixes are expanded to include the objects under the prefix, up to a maximum of 10,000 objects.
* `gs://`: A path to an object or object prefix in GCS. The artifact will track checksums and versioning information (if the bucket has object versioning enabled) for the referenced objects. Object prefixes are expanded to include the objects under the prefix, up to a maximum of 10,000 objects.

The following table shows how different API calls produce different artifact contents:

| API call                                                                      | Resulting artifact contents                                          |
| ----------------------------------------------------------------------------- | -------------------------------------------------------------------- |
| `artifact.add_reference('s3://my-bucket/model.h5')`                           | `model.h5`                                                           |
| `artifact.add_reference('s3://my-bucket/checkpoints/model.h5')`               | `model.h5`                                                           |
| `artifact.add_reference('s3://my-bucket/model.h5', name='models/mymodel.h5')` | `models/mymodel.h5`                                                  |
| `artifact.add_reference('s3://my-bucket/images')`                             | <p><code>cat.png</code></p><p><code>dog.png</code></p>               |
| `artifact.add_reference('s3://my-bucket/images', name='images')`              | <p><code>images/cat.png</code></p><p><code>images/dog.png</code></p> |

### Add files to artifacts from parallel runs

For large datasets or distributed training, multiple parallel runs might need to contribute to a single artifact.

```python theme={null}
import wandb
import time

# This example uses Ray to runs in parallel
# for demonstration purposes.
import ray

ray.init()

artifact_type = "dataset"
artifact_name = "parallel-artifact"
table_name = "distributed_table"
parts_path = "parts"
num_parallel = 5

# Each batch of parallel writers should have its own
# unique group name.
group_name = "writer-group-{}".format(round(time.time()))


@ray.remote
def train(i):
    """
    Our writer job. Each writer will add one image to the artifact.
    """
    with wandb.init(group=group_name) as run:
        artifact = wandb.Artifact(name=artifact_name, type=artifact_type)

        # Add data to a wandb table.
        table = wandb.Table(columns=["a", "b", "c"], data=[[i, i * 2, 2**i]])

        # Add the table to folder in the artifact
        artifact.add(table, "{}/table_{}".format(parts_path, i))

        # Upserting the artifact creates or appends data to the artifact
        run.upsert_artifact(artifact)


# Launch your runs in parallel
result_ids = [train.remote(i) for i in range(num_parallel)]

# Join on all the writers to make sure their files have
# been added before finishing the artifact.
ray.get(result_ids)

# Once all the writers are finished, finish the artifact
# to mark it ready.
with wandb.init(group=group_name) as run:
    artifact = wandb.Artifact(artifact_name, type=artifact_type)

    # Create a "PartitionTable" pointing to the folder of tables
    # and add it to the artifact.
    artifact.add(wandb.data_types.PartitionedTable(parts_path), table_name)

    # Finish artifact finalizes the artifact, disallowing future "upserts"
    # to this version.
    run.finish_artifact(artifact)
```

## Find path for logged artifacts and other metadata

The following code snippet shows how to use the [W\&B Public API](/models/ref/python/public-api/) to list the files in a run, including their names and URLs. Replace the `<entity/project/run-id>` placeholder with your own values:

```python theme={null}
from wandb.apis.public.files import Files
from wandb.apis.public.api import Api

# Example run object
run = Api().run("<entity/project/run-id>")

# Create a Files object to iterate over files in the run
files = Files(api.client, run)

# Iterate over files
for file in files:
    print(f"File Name: {file.name}")
    print(f"File URL: {file.url}")
    print(f"Path to file in the bucket: {file.direct_url}")
```

See the [File](/models/ref/python/public-api/file) Class for more information on available attributes and methods.


# Create an artifact alias
Source: https://docs.wandb.ai/models/artifacts/create-a-custom-alias

Create and manage custom aliases to reference specific W&B artifact versions by meaningful names like best or production.

Use aliases as pointers to specific versions. By default, `wandb.Run.log_artifact()` adds the `latest` alias to the logged version.

W\&B creates an artifact version `v0` and attaches it to your artifact when you log that artifact for the first time. W\&B checksums the contents when you log again to the same artifact. If the artifact changed, W\&B saves a new version `v1`.

For example, if you want your training script to pull the most recent version of a dataset, specify `latest` when you use that artifact. The following code example demonstrates how to download a recent dataset artifact named `bike-dataset` that has an alias, `latest`:

```python theme={null}
import wandb

with wandb.init(project="<project>") as run:
    artifact = run.use_artifact("bike-dataset:latest")
    artifact.download()
```

You can also apply a custom alias to an artifact version. For example, if you want to mark that model checkpoint is the best on the metric AP-50, you could add the string `'best-ap50'` as an alias when you log the model artifact.

```python theme={null}
with wandb.init(project="<project>") as run:
    artifact = wandb.Artifact("run-3nq3ctyy-bike-model", type="model")
    artifact.add_file("model.h5")
    run.log_artifact(artifact, aliases=["latest", "best-ap50"])
```


# Create an artifact version
Source: https://docs.wandb.ai/models/artifacts/create-a-new-artifact-version

Create a new artifact version from a single run or from a distributed process.

Create a new artifact version with a single [run](/models/runs/) or collaboratively with distributed runs. You can optionally create a new artifact version from a previous version known as an [incremental artifact](#create-a-new-artifact-version-from-an-existing-version).

<Note>
  We recommend that you create an incremental artifact when you need to apply changes to a subset of files in an artifact, where the size of the original artifact is significantly larger.
</Note>

## Create new artifact versions from scratch

There are two ways to create a new artifact version: from a single run and from distributed runs. They are defined as follows:

* **Single run**: A single run provides all the data for a new version. This is the most common case and is best suited when the run fully recreates the needed data. For example: outputting saved models or model predictions in a table for analysis.
* **Distributed runs**: A set of runs collectively provides all the data for a new version. This is best suited for distributed jobs which have multiple runs generating data, often in parallel. For example: evaluating a model in a distributed manner, and outputting the predictions.

W\&B will create a new artifact and assign it a `v0` alias if you pass a name to the `wandb.Artifact` API that does not exist in your project. W\&B checksums the contents when you log again to the same artifact. If the artifact changed, W\&B saves a new version `v1`.

W\&B will retrieve an existing artifact if you pass a name and artifact type to the `wandb.Artifact` API that matches an existing artifact in your project. The retrieved artifact will have a version greater than 1.

<Frame>
  <img alt="Artifact workflow comparison" />
</Frame>

### Single run

Log a new version of an Artifact with a single run that produces all the files in the artifact. This case occurs when a single run produces all the files in the artifact.

Based on your use case, select one of the tabs below to create a new artifact version inside or outside of a run:

<Tabs>
  <Tab title="Inside a run">
    Create an artifact version within a W\&B run:

    1. Create a run with `wandb.init()`.
    2. Create a new artifact or retrieve an existing one with `wandb.Artifact`.
    3. Add files to the artifact with `.add_file`.
    4. Log the artifact to the run with `.log_artifact`.

    ```python theme={null}
    with wandb.init() as run:
        artifact = wandb.Artifact("artifact_name", "artifact_type")

        # Add Files and Assets to the artifact using
        # `.add`, `.add_file`, `.add_dir`, and `.add_reference`
        artifact.add_file("image1.png")
        run.log_artifact(artifact)
    ```
  </Tab>

  <Tab title="Outside of a run">
    Create an artifact version outside of a W\&B run:

    1. Create a new artifact or retrieve an existing one with `wanb.Artifact`.
    2. Add files to the artifact with `.add_file`.
    3. Save the artifact with `.save`.

    ```python theme={null}
    artifact = wandb.Artifact("artifact_name", "artifact_type")
    # Add Files and Assets to the artifact using
    # `.add`, `.add_file`, `.add_dir`, and `.add_reference`
    artifact.add_file("image1.png")
    artifact.save()
    ```
  </Tab>
</Tabs>

### Distributed runs

Allow a collection of runs to collaborate on a version before committing it. This is in contrast to single run mode described above where one run provides all the data for a new version.

<Note>
  1. Each run in the collection needs to be aware of the same unique ID (called `distributed_id`) in order to collaborate on the same version. By default, if present, W\&B uses the run's `group` as set by `wandb.init(group=GROUP)` as the `distributed_id`.
  2. There must be a final run that "commits" the version, permanently locking its state.
  3. Use `upsert_artifact` to add to the collaborative artifact and `finish_artifact` to finalize the commit.
</Note>

Consider the following example. Different runs (labelled below as **Run 1**, **Run 2**, and **Run 3**) add a different image file to the same artifact with `upsert_artifact`.

#### Run 1

```python theme={null}
with wandb.init() as run:
    artifact = wandb.Artifact("artifact_name", "artifact_type")
    # Add Files and Assets to the artifact using
    # `.add`, `.add_file`, `.add_dir`, and `.add_reference`
    artifact.add_file("image1.png")
    run.upsert_artifact(artifact, distributed_id="my_dist_artifact")
```

#### Run 2

```python theme={null}
with wandb.init() as run:
    artifact = wandb.Artifact("artifact_name", "artifact_type")
    # Add Files and Assets to the artifact using
    # `.add`, `.add_file`, `.add_dir`, and `.add_reference`
    artifact.add_file("image2.png")
    run.upsert_artifact(artifact, distributed_id="my_dist_artifact")
```

#### Run 3

Must run after Run 1 and Run 2 complete. The Run that calls `wandb.Run.finish_artifact()` can include files in the artifact, but does not need to.

```python theme={null}
with wandb.init() as run:
    artifact = wandb.Artifact("artifact_name", "artifact_type")
    # Add Files and Assets to the artifact
    # `.add`, `.add_file`, `.add_dir`, and `.add_reference`
    artifact.add_file("image3.png")
    run.finish_artifact(artifact, distributed_id="my_dist_artifact")
```

## Create a new artifact version from an existing version

Add, modify, or remove a subset of files from a previous artifact version without the need to re-index the files that didn't change. Adding, modifying, or removing a subset of files from a previous artifact version creates a new artifact version known as an *incremental artifact*.

<Frame>
  <img alt="Incremental artifact versioning" />
</Frame>

Here are some scenarios for each type of incremental change you might encounter:

* add: you periodically add a new subset of files to a dataset after collecting a new batch.
* remove: you discovered several duplicate files and want to remove them from your artifact.
* update: you corrected annotations for a subset of files and want to replace the old files with the correct ones.

You could create an artifact from scratch to perform the same function as an incremental artifact. However, when you create an artifact from scratch, you will need to have all the contents of your artifact on your local disk. When making an incremental change, you can add, remove, or modify a single file without changing the files from a previous artifact version.

<Note>
  You can create an incremental artifact within a single run or with a set of runs (distributed mode).
</Note>

Follow the procedure below to incrementally change an artifact:

1. Obtain the artifact version you want to perform an incremental change on:

<Tabs>
  <Tab title="Inside a run">
    ```python theme={null}
    saved_artifact = run.use_artifact("my_artifact:latest")
    ```
  </Tab>

  <Tab title="Outside of a run">
    ```python theme={null}
    client = wandb.Api()
    saved_artifact = client.artifact("my_artifact:latest")
    ```
  </Tab>
</Tabs>

2. Create a draft with:

```python theme={null}
draft_artifact = saved_artifact.new_draft()
```

3. Perform any incremental changes you want to see in the next version. You can either add, remove, or modify an existing entry.

Select one of the tabs for an example on how to perform each of these changes:

<Tabs>
  <Tab title="Add">
    Add a file to an existing artifact version with the `add_file` method:

    ```python theme={null}
    draft_artifact.add_file("file_to_add.txt")
    ```

    <Note>
      You can also add multiple files by adding a directory with the `add_dir` method.
    </Note>
  </Tab>

  <Tab title="Remove">
    Remove a file from an existing artifact version with the `remove` method:

    ```python theme={null}
    draft_artifact.remove("file_to_remove.txt")
    ```

    <Note>
      You can also remove multiple files with the `remove` method by passing in a directory path.
    </Note>
  </Tab>

  <Tab title="Modify">
    Modify or replace contents by removing the old contents from the draft and adding the new contents back in:

    ```python theme={null}
    draft_artifact.remove("modified_file.txt")
    draft_artifact.add_file("modified_file.txt")
    ```
  </Tab>
</Tabs>

4. Lastly, log or save your changes. The following tabs show you how to save your changes inside and outside of a W\&B run. Select the tab that is appropriate for your use case:

<Tabs>
  <Tab title="Inside a run">
    ```python theme={null}
    run.log_artifact(draft_artifact)
    ```
  </Tab>

  <Tab title="Outside of a run">
    ```python theme={null}
    draft_artifact.save()
    ```
  </Tab>
</Tabs>

Putting it all together, the code examples above look like:

<Tabs>
  <Tab title="Inside a run">
    ```python theme={null}
    with wandb.init(job_type="modify dataset") as run:
        saved_artifact = run.use_artifact(
            "my_artifact:latest"
        )  # fetch artifact and input it into your run
        draft_artifact = saved_artifact.new_draft()  # create a draft version

        # modify a subset of files in the draft version
        draft_artifact.add_file("file_to_add.txt")
        draft_artifact.remove("dir_to_remove/")
        run.log_artifact(
            draft_artifact
        )  # log your changes to create a new version and mark it as output to your run
    ```
  </Tab>

  <Tab title="Outside of a run">
    ```python theme={null}
    client = wandb.Api()
    saved_artifact = client.artifact("my_artifact:latest")  # load your artifact
    draft_artifact = saved_artifact.new_draft()  # create a draft version

    # modify a subset of files in the draft version
    draft_artifact.remove("deleted_file.txt")
    draft_artifact.add_file("modified_file.txt")
    draft_artifact.save()  # commit changes to the draft
    ```
  </Tab>
</Tabs>


# Artifact data privacy and compliance
Source: https://docs.wandb.ai/models/artifacts/data-privacy-and-compliance

Learn where W&B files are stored by default. Explore how to save, store sensitive information.

Files are uploaded to a Google Cloud bucket managed by W\&B when you log artifacts. The contents of the bucket are encrypted both at rest and in transit. Artifact files are only visible to users who have access to the corresponding project.

<Frame>
  <img alt="GCS W&B Client Server diagram" />
</Frame>

When you delete a version of an artifact, it is marked for soft deletion in our database and removed from your storage cost. When you delete an entire artifact, it is queued for permanent deletion and all of its contents are removed from the W\&B bucket. If you have specific needs around file deletion, reach out to [Customer Support](mailto:support@wandb.com).

By default, deleted artifacts are retained for 7 days and can be restored during this period, which is configurable for Dedicated Cloud. Learn more about data retention in [Multi-tenant Cloud](/platform/hosting/hosting-options/multi_tenant_cloud#data-retention-policy) or [Dedicated Cloud](/platform/hosting/hosting-options/dedicated-cloud#data-retention-policy).

For sensitive datasets that cannot reside in a multi-tenant environment, you can use [W\&B Dedicated Cloud](/platform/hosting/hosting-options/dedicated-cloud) or [reference artifacts](/models/artifacts/track-external-files). Reference artifacts track references to private buckets without sending file contents to W\&B. Reference artifacts maintain links to files on your buckets or servers. W\&B only keeps track of the metadata associated with the files, not the files themselves.

<Frame>
  <img alt="W&B Client Server Cloud diagram" />
</Frame>

For more information about deployment options, see [Dedicated Cloud](/platform/hosting/hosting-options/dedicated-cloud) or [Self-Managed](/platform/hosting/hosting-options/self-managed). To discuss your specific requirements, contact [contact@wandb.com](mailto:contact@wandb.com).


# Delete an artifact
Source: https://docs.wandb.ai/models/artifacts/delete-artifacts

Delete artifacts interactively with the App UI or programmatically with the W&B Python SDK.

Delete artifacts interactively with the W\&B App or programmatically with the W\&B Python SDK. When you delete an artifact, W\&B marks that artifact as a *soft-delete*. In other words, the artifact is marked for deletion but files are not immediately deleted from storage.

The contents of the artifact remain as a soft-delete, or pending deletion state, until a regularly run garbage collection process reviews all artifacts marked for deletion. The garbage collection process deletes associated files from storage if the artifact and its associated files are not used by a previous or subsequent artifact versions.

<Note>
  Garbage collection is **best-effort**. W\&B does not guarantee how quickly freed space appears in your object storage after you delete an artifact. Large deployments or backlogs can take longer than expected. For how this fits with run data, retention settings, and optional operator actions, see [Manage bucket storage and costs](/platform/hosting/managing-bucket-storage).
</Note>

## Artifact garbage collection workflow

The following diagram illustrates the complete artifact garbage collection process:

```mermaid theme={null}
graph TB
    Start([Artifact Deletion Initiated]) --> DeleteMethod{Deletion Method}
    
    DeleteMethod -->|UI| UIDelete[Delete via W&B App]
    DeleteMethod -->|SDK| SDKDelete[Delete via W&B Python SDK]
    DeleteMethod -->|TTL| TTLDelete[TTL Policy Expires]
    
    UIDelete --> SoftDelete[Artifact Marked as<br/>'Soft Delete']
    SDKDelete --> SoftDelete
    TTLDelete --> SoftDelete
    
    SoftDelete --> GCWait[(Wait for<br/>best-effort<br/>Garbage Collection)]
    
    GCWait --> GCRun[Garbage Collection<br/>Process Runs<br/><br/>- Reviews all soft-deleted artifacts<br/>- Checks file dependencies]
    
    GCRun --> CheckUsage{Are files used by<br/>other artifact versions?}
    
    CheckUsage -->|Yes| KeepFiles[Files Kept in Storage<br/><br/>- Artifact marked deleted<br/>- Files remain for other versions]
    CheckUsage -->|No| DeleteFiles[Files Deleted from Storage<br/><br/>- Artifact fully removed<br/>- Storage space reclaimed]
    
    KeepFiles --> End([End])
    DeleteFiles --> End
    
    style Start fill:#e1f5fe,stroke:#333,stroke-width:2px,color:#000
    style SoftDelete fill:#fff3e0,stroke:#333,stroke-width:2px,color:#000
    style GCRun fill:#f3e5f5,stroke:#333,stroke-width:2px,color:#000
    style KeepFiles fill:#e8f5e9,stroke:#333,stroke-width:2px,color:#000
    style DeleteFiles fill:#ffebee,stroke:#333,stroke-width:2px,color:#000
    style End fill:#e0e0e0,stroke:#333,stroke-width:2px,color:#000
```

You can schedule when artifacts are deleted from W\&B with TTL policies. For more information, see [Manage data retention with Artifact TTL policy](./ttl).

<Note>
  Artifacts deleted by a TTL policy, the W\&B Python SDK, or the W\&B App are first soft-deleted. Soft-deleted artifacts are then garbage-collected before they are permanently deleted.
</Note>

<Note>
  Deleting an entity, project, or artifact collection triggers the artifact deletion process described on this page. When you delete a run and choose to delete its associated artifacts, those artifacts follow the same soft-delete and garbage collection workflow.
</Note>

## Delete an artifact version

Delete an artifact version interactively with the W\&B App or programmatically with the W\&B Python SDK.

<Tabs>
  <Tab title="W&B App">
    To delete an artifact version:

    1. Navigate to the project that contains the artifact version you want to delete.
    2. Select the **Artifacts** tab.
    3. From the list of artifact types, select the type of artifact that contains the version you want to delete.
    4. Click the **action (<Icon icon="ellipsis" />)** menu next to the artifact version you want to delete.
    5. From the dropdown, choose **Delete Version**.
  </Tab>

  <Tab title="W&B Python SDK">
    Delete an artifact version programmatically with the [wandb.Artifact.delete()](/models/ref/python/experiments/artifact#delete) method. Provide the full name of the artifact. The full name consists of `<entity>/<project>/<artifact_name>:<version>`. Set the `delete_aliases` parameter to `True` to delete the artifact even if it has one or more aliases associated with it.

    ```python theme={null}
    import wandb

    api = wandb.Api()

    # Get the artifact by its path
    artifact = api.artifact("<entity>/<project>/<artifact_name>:<version>")

    # Delete the artifact version along with any aliases
    artifact.delete(delete_aliases=True)
    ```
  </Tab>
</Tabs>

## Delete multiple artifact versions

The following code example shows how to delete multiple artifact versions. Provide the entity, project name, and run ID that created the artifact as arguments to `wandb.Api.run()`. This returns a run object that you can use to access all artifact versions created by that run. Next, iterate through the artifact versions and delete the ones that match your criteria.

<Tip>
  Set the `delete_aliases` parameter to `True` (`wandb.Artifact.delete(delete_aliases=True)`) to delete an artifact version and any aliases associated with it.
</Tip>

Replace the `<entity>`, `<project>`, `<run_id>`, and `<artifact_name>` placeholders with your own values:

```python theme={null}
import wandb

# Initialize W&B API
api = wandb.Api()

# Get the run by its path. Consists of <entity>/<project>/<run_id>
run = api.run("<entity>/<project>/<run_id>")

# Specify the artifact name to delete versions for
artifact_name = "<artifact_name>"

# Search and delete artifact versions with the specified name
for artifact in run.logged_artifacts():
    print(f"Found artifact: {artifact.name}") # Example name run_4dfbufgq_model:v0
    # Grab only the artifact name without the version with split()
    if artifact.name.split(":")[0] == artifact_name:
        print(f"Deleting artifact version: {artifact.name}")
        artifact.delete(delete_aliases=True)
```

## Delete multiple artifact versions with a specific alias

The following code demonstrates how to delete multiple artifact versions that have a specific alias.

Replace the `<entity>`, `<project>`, `<run_id>`, `<artifact_name>`, and `<alias>` placeholders with your own values:

```python theme={null}
import wandb

# Initialize W&B API
api = wandb.Api()

# Get the run by its path. Consists of <entity>/<project>/<run_id>
run = api.run("<entity>/<project>/<run_id>")

# Specify the artifact name to delete versions for
artifact_name = "<artifact_name>"

# Specify the alias to filter artifact versions for deletion
desired_alias = "<alias>"

# Delete artifacts logged to run with alias 'v3' and 'v4
for artifact in run.logged_artifacts():
    print(f"Found artifact: {artifact.name}")
    if (artifact.name.split(":")[0] == artifact_name) and (desired_alias in artifact.aliases):
            artifact.delete(delete_aliases=True)
```

## Delete an artifact collection

<Tabs>
  <Tab title="W&B App">
    To delete an artifact collection:

    1. Navigate to the artifact collection you want to delete.
    2. Select the **action (<Icon icon="ellipsis" />)** menu next to the artifact collection name.
    3. From the dropdown menu, select **Delete**.
  </Tab>

  <Tab title="W&B Python SDK">
    Delete artifact collection programmatically with the [wandb.Artifact.delete()](/models/ref/python/experiments/artifact#delete) method.

    Provide the full path of the artifact collection to `wandb.Api.artifact_collection(name="")`. The full path consists of `<entity>/<project>/<artifact_collection_name>`.

    ```python theme={null}
    import wandb

    # Initialize W&B API
    api = wandb.Api()

    # Get the artifact collection by its path. Consists of
    # <entity>/<project>/<artifact_collection_name>
    collection = api.artifact_collection(
        type_name = "<artifact_type>",
        name = "<entity>/<project>/<artifact_collection_name>"
    )
    collection.delete()
    ```
  </Tab>
</Tabs>

## Protected aliases and deletion permissions

Artifacts with protected aliases have special deletion restrictions. [Protected aliases](/models/registry/aliases#protected-aliases) are aliases in the W\&B Registry that registry admins can set to prevent unauthorized deletion.

<Note>
  **Important considerations for protected aliases:**

  * Artifacts with protected aliases cannot be deleted by non-registry admins.
  * Within a registry, registry admins can unlink protected artifact versions and delete collections/registries that contain protected aliases.
  * For source artifacts: if a source artifact is linked to a registry with a protected alias, it cannot be deleted by any user
  * Registry admins can remove the protected aliases from source artifacts and then delete them.
</Note>

## Enable garbage collection based on how W\&B is hosted

<Note>Garbage collection timing is not guaranteed. See [Manage bucket storage and costs](/platform/hosting/managing-bucket-storage) for details.</Note>

Garbage collection is active by default if you use W\&B Multi-tenant Cloud. In W\&B Dedicated and Self-Managed, you might need to take these additional steps to activate garbage collection.

1. **W\&B Self-Managed**: Set `GORILLA_ARTIFACT_GC_ENABLED=true`.
2. **Dedicated Cloud**: Contact support to verify that garbage collection is active.
3. Enable bucket versioning if you use [AWS](https://docs.aws.amazon.com/AmazonS3/latest/userguide/manage-versioning-examples.html), [Google Cloud](https://cloud.google.com/storage/docs/object-versioning) or any other storage provider such as [Minio](https://min.io/docs/minio/linux/administration/object-management/object-versioning.html#enable-bucket-versioning). If you use Azure, [enable soft deletion](https://learn.microsoft.com/azure/storage/blobs/soft-delete-blob-overview), which is equivalent to bucket versioning.

The following table describes how to satisfy requirements to enable garbage collection based on your deployment type.

The `X` indicates you must satisfy the requirement:

|                                                                                                  | Environment variable | Enable versioning |
| ------------------------------------------------------------------------------------------------ | -------------------- | ----------------- |
| Multi-tenant Cloud                                                                               |                      |                   |
| Multi-tenant Cloud with [BYOB storage](/platform/hosting/data-security/secure-storage-connector) |                      | X                 |
| Dedicated Cloud                                                                                  |                      |                   |
| Dedicated Cloud with [BYOB storage](/platform/hosting/data-security/secure-storage-connector)    |                      | X                 |
| Self-Managed                                                                                     | X                    | X                 |

<Note>
  note
  Secure storage connector is currently only available for Google Cloud Platform and Amazon Web Services.
</Note>


# Download and use artifacts
Source: https://docs.wandb.ai/models/artifacts/download-and-use-an-artifact

Download and use Artifacts from multiple projects.

Download and use an artifact that is already stored on the W\&B server or construct an artifact object and pass it in to for de-duplication as necessary.

<Note>
  Team members with view-only seats cannot download artifacts.
</Note>

### Download and use an artifact stored on W\&B

Download and use an artifact stored in W\&B either inside or outside of a W\&B Run. Use the Public API ([`wandb.Api`](/models/ref/python/public-api/api)) to export (or update data) already saved in W\&B.

<Tabs>
  <Tab title="During a run">
    First, import the W\&B Python SDK. Next, create a W\&B [Run](/models/ref/python/experiments/run):

    ```python theme={null}
    import wandb

    with wandb.init(project="<example>", job_type="<job-type>") as run:
        # See next step
    ```

    Indicate the artifact you want to use with the [`wandb.Run.use_artifact()`](/models/ref/python/experiments/run#use_artifact) method. This returns a run object. In the following code snippet specifies an artifact called `'bike-dataset'` with the alias `'latest'`:

    ```python theme={null}
    # Indicate the artifact to use. Format is "name:alias"
    artifact = run.use_artifact("bike-dataset:latest")
    ```

    Use the object returned to download all the contents of the artifact:

    ```python theme={null}
    # Download the entire artifact
    datadir = artifact.download()
    ```

    You can optionally pass a path to the root parameter to download the contents of the artifact to a specific directory.

    Use the [`wandb.Artifact.get_entry()`](/models/ref/python/experiments/artifact#get_entry) method to download only a subset of files:

    ```python theme={null}
    # Download a specific file
    entry = artifact.get_entry(name)
    ```

    Putting this together, the complete code example looks like this:

    ```python theme={null}
    import wandb    

    with wandb.init(project="<example>", job_type="<job-type>") as run:
        # Indicate the artifact to use. Format is "name:alias"
        artifact = run.use_artifact("bike-dataset:latest")

        # Download the entire artifact
        datadir = artifact.download()

        # Download a specific file
        entry = artifact.get_entry("bike.png")
    ```

    This fetches only the file at the path `name`. It returns an `Entry` object with the following methods:

    * `Entry.download`: Downloads file from the artifact at path `name`
    * `Entry.ref`: If `add_reference` stored the entry as a reference, returns the URI
  </Tab>

  <Tab title="Outside of a run">
    First, import the W\&B SDK. Next, create an artifact object from the Public API Class. Provide the entity, project, artifact, and alias associated with that artifact:

    ```python theme={null}
    import wandb

    api = wandb.Api()
    artifact = api.artifact("entity/project/artifact:alias")
    ```

    Use the object returned to download the contents of the artifact:

    ```python theme={null}
    artifact.download()
    ```

    You can optionally pass a path the `root` parameter to download the contents of the artifact to a specific directory. For more information, see the [Python SDK Reference Guide](/models/ref/python/experiments/artifact#download).
  </Tab>

  <Tab title="W&B CLI">
    Use the `wandb artifact get` command to download an artifact from the W\&B server.

    ```
    $ wandb artifact get project/artifact:alias --root mnist/
    ```
  </Tab>
</Tabs>

### Partially download an artifact

You can optionally download part of an artifact based on a prefix. Use the `path_prefix=` (`wandb.Artifact.download(path_prefix=)`) parameter to download a single file or the content of a sub-folder.

```python theme={null}
with wandb.init(project="<example>", job_type="<job-type>") as run:
    # Indicate the artifact to use. Format is "name:alias"
    artifact = run.use_artifact("bike-dataset:latest")

    # Download a specific file or sub-folder
    artifact.download(path_prefix="bike.png") # downloads only bike.png
```

Alternatively, you can download files from a certain directory. To do so, specify the directory within the `path_prefix=` parameter. Continuing from the previous code snippet:

```python theme={null}
# downloads files in the images/bikes directory
artifact.download(path_prefix="images/bikes/") 
```

### Use an artifact from a different project

Specify the name of artifact along with its project name to reference an artifact. You can also reference artifacts across entities by specifying the name of the artifact with its entity name.

The following code example demonstrates how to query an artifact from another project as input to the current W\&B run.

```python theme={null}
with wandb.init(project="<example>", job_type="<job-type>") as run:
    # Query W&B for an artifact from another project and mark it
    # as an input to this run.
    artifact = run.use_artifact("my-project/artifact:alias")

    # Use an artifact from another entity and mark it as an input
    # to this run.
    artifact = run.use_artifact("my-entity/my-project/artifact:alias")

```

### Construct and use an artifact simultaneously

Simultaneously construct and use an artifact. Create an artifact object and pass it to use\_artifact. This creates an artifact in W\&B if it does not exist yet. The [`wandb.Run.use_artifact()`](/models/ref/python/experiments/run#use_artifact) API is idempotent, so you can call it as many times as you like.

```python theme={null}
import wandb

with wandb.init(project="<example>", job_type="<job-type>") as run:
    artifact = wandb.Artifact("reference model")
    artifact.add_file("model.h5")
    run.use_artifact(artifact)
```

For more information about constructing an artifact, see [Construct an artifact](/models/artifacts/construct-an-artifact/).


# Explore artifact lineage graphs
Source: https://docs.wandb.ai/models/artifacts/explore-and-traverse-an-artifact-graph

View and traverse artifact lineage graphs to track the inputs and outputs of W&B runs as a directed acyclic graph.

W\&B tracks the inputs and outputs of runs using directed acyclic graphs (DAGs) called *lineage graphs*. Lineage graphs are visual representations of the relationships between artifacts and runs in an ML experiment. They show how data and models
flow through different stages of the ML lifecycle, from raw data ingestion to model training and evaluation.

Tracking artifact lineage provides several key advantages:

* **Reproducibility**: Enables teams to reproduce experiments, models, and results for debugging, experimentation, and validation.
* **Version control**: Tracks changes to artifacts over time, allowing teams to revert to previous data or model versions when needed.
* **Auditing**: Maintains a detailed record of artifacts and transformations to support compliance and governance.
* **Collaboration**: Helps to improve teamwork by making experiment history transparent, reducing duplicated effort, and accelerating development.

## View an artifact's lineage graph

To view an artifact's lineage graph:

1. Navigate to your project's workspace in the W\&B App.
2. Click on the **Artifacts** tab in the project sidebar.
3. Select an artifact, then click the **Lineage** tab.

## Navigate lineage graphs

The lineage graph is a visual representation of the relationships between artifacts and runs in an ML experiment. Use the W\&B App UI or the Python SDK to explore and traverse an artifact's lineage graph.

<Tabs>
  <Tab title="W&B App UI">
    Nodes with green icons represent runs. Nodes with blue icons represent artifacts. Arrows between nodes indicate the input and output of a run or artifact.

    Artifact nodes display the artifact's name along with the version of the artifact in the form `<artifact_name>:<version>`. An artifact's type is displayed above the name of the artifact.

    <Note>
      You can view the type and the name of artifact in both the left sidebar and in the lineage graph node.
    </Note>

    Run nodes display the run's name.

    <Frame>
      <img alt="Run and artifact nodes" />
    </Frame>

    Click any individual run to get more information about that runs such as the run's: start time, time duration, author, job type, and more. Click any individual artifact to get more information about the artifact's: aliases, creation time, type, version, description, the run that logged the artifact, file size, and more.

    <Frame>
      <img alt="Previewing a run" />
    </Frame>

    Runs that create multiple versions of the same artifact are grouped together in a cluster. Click on a specific artifact version listed within the cluster to view specific information about that artifact version.

    <Frame>
      <img alt="Cluster of artifact versions in a lineage graph" />
    </Frame>

    Click and drag a node to rearrange the graph to customize the layout. You can also zoom in and out of the graph to get a better view of the nodes and their relationships.

    <Frame>
      <img alt="Rearranging nodes in a lineage graph" />
    </Frame>

    Hover your mouse over a node and click on the eye icon to hide or show a node in the graph. This is useful for decluttering the graph to focus on specific nodes and their relationships.
  </Tab>

  <Tab title="W&B Python SDK">
    Programmatically navigate a graph using the W\&B Python SDK. Use an artifact object's
    [`logged_by()`](/models/ref/python/experiments/artifact#method-artifact-logged-by) and [`used_by()`](/models/ref/python/experiments/artifact#method-artifact-used-by) methods to walk the graph:

    ```python theme={null}
    with wandb.init() as run:
        artifact = run.use_artifact("artifact_name:latest")

        # Walk up and down the graph from an artifact:
        producer_run = artifact.logged_by()
        consumer_runs = artifact.used_by()
    ```
  </Tab>
</Tabs>

## Enable lineage graph tracking

To enable lineage graph tracking, you need to mark artifacts as [inputs](/models/artifacts/explore-and-traverse-an-artifact-graph) or
[outputs](/models/artifacts/explore-and-traverse-an-artifact-graph#track-the-output-of-a-run) of a run using the W\&B Python SDK.

### Track the input of a run

Mark an artifact as the input (or dependency) of a run with the [`wandb.Run.use_artifact()`](/models/ref/python/experiments/run#method-runuse_artifact)
method. Specify the name of the artifact and an optional alias to reference a specific version of that artifact. The name of the
artifact is in the format `<artifact_name>:<version>` or `<artifact_name>:<alias>`.

Replace values enclosed in angle brackets (`< >`) with your values:

```python theme={null}
import wandb

# Initialize a run
with wandb.init(entity="<entity>", project="<project>") as run:
  # Get artifact, mark it as a dependency
  artifact = run.use_artifact(artifact_or_name="<name>", aliases="<alias>")
```

### Track the output of a run

Use [`wandb.Run.log_artifact()`](/models/ref/python/experiments/run#log_artifact) to declare an artifact as an output of a run. First,
create an artifact with the [`wandb.Artifact()`](/models/ref/python/experiments/artifact#wandb.Artifact) constructor. Then, log the
artifact as an output of the run with `wandb.Run.log_artifact()`.

Replace values enclosed in angle brackets (`< >`) with your values:

```python theme={null}
import wandb

# Initialize a run
with wandb.init(entity="<entity>", project="<project>") as run:
  
  # Create an artifact
  artifact = wandb.Artifact(name = "<artifact_name>", type = "<artifact_type>")
  artifact.add_file(local_path = "<local_filepath>", name="<optional-name>")

  # Log the artifact as an output of the run
  run.log_artifact(artifact_or_path = artifact)
```

## Artifact clusters

When a level of the graph has five or more runs or artifacts, it creates a cluster. A cluster has a search bar to find specific versions of runs or artifacts and pulls an individual node from a cluster to continue investigating the lineage of a node inside a cluster.

Clicking on a node opens a preview with an overview of the node. Clicking on the arrow extracts the individual run or artifact so you can examine the lineage of the extracted node.

<Frame>
  <img alt="Searching a run cluster" />
</Frame>


# Manage artifact storage and memory allocation
Source: https://docs.wandb.ai/models/artifacts/storage

Manage storage, memory allocation of W&B Artifacts.

W\&B stores artifact files in a private Google Cloud Storage bucket located in the United States by default. All files are encrypted at rest and in transit.

For sensitive files, we recommend you set up [Private Hosting](/platform/hosting/) or use [reference artifacts](/models/artifacts/track-external-files/).

During training, W\&B locally saves logs, artifacts, and configuration files in the following local directories:

| File                         | Default location       | To change default location set:                                     |
| ---------------------------- | ---------------------- | ------------------------------------------------------------------- |
| logs                         | `./wandb`              | `dir` in `wandb.init()` or set the `WANDB_DIR` environment variable |
| artifacts                    | `~/.cache/wandb`       | the `WANDB_CACHE_DIR` environment variable                          |
| configs                      | `~/.config/wandb`      | the `WANDB_CONFIG_DIR` environment variable                         |
| staging artifacts for upload | `~/.cache/wandb-data/` | the `WANDB_DATA_DIR` environment variable                           |
| downloaded artifacts         | `./artifacts`          | the `WANDB_ARTIFACT_DIR` environment variable                       |

For a complete guide to using environment variables to configure W\&B, see the [environment variables reference](/models/track/environment-variables/).

<Warning>
  Depending on the machine on `wandb` is initialized on, these default folders may not be located in a writeable part of the file system. This might trigger an error.
</Warning>

### Clean up local artifact cache

W\&B caches artifact files to speed up downloads across versions that share files in common. Over time this cache directory can become large. Run the [`wandb artifact cache cleanup`](/models/ref/cli/wandb-artifact/wandb-artifact-cache/) command to prune the cache and to remove any files that have not been used recently.

The following code snippet demonstrates how to limit the size of the cache to 1GB. Copy and paste the code snippet into your terminal:

```bash theme={null}
$ wandb artifact cache cleanup 1GB
```


# Track external files
Source: https://docs.wandb.ai/models/artifacts/track-external-files

Track files saved in an external bucket, HTTP file server, or an NFS share.

Use *reference artifacts* to track and use files saved outside of W\&B servers. Common external storage solutions include: CoreWeave AI Object Storage, an Amazon Simple Storage Service (Amazon S3) bucket, GCS bucket, Azure blob, HTTP file server, or NFS share.

Reference artifacts behave similar to non-reference artifacts. The key difference is that the reference artifacts only consists of metadata about the files, such as their sizes and MD5 checksums. The files themselves never leave your system.

You can interact with reference artifact similarly to non-reference artifacts. In the W\&B App, you can browse the contents of the reference artifact using the file browser, explore the full dependency graph, and scan through the versioned history of your artifact. However, the UI cannot render rich media such as images, audio, because the data itself is not contained within the artifact.

<Note>
  If you log an artifact that does not track external files, W\&B saves the artifact's files to W\&B servers. This is the default behavior when you log artifacts with the W\&B Python SDK.

  If you log an artifact that tracks external files, W\&B logs metadata about the object, such as the object's ETag and size. If object versioning is enabled on the bucket, the version ID is also logged.
</Note>

The following sections describe how to track external reference artifacts.

## Track an artifact in an external bucket

Use the W\&B Python SDK to track references to files stored outside of W\&B.

1. Initialize a run with `wandb.init()`.
2. Create an artifact object with `wandb.Artifact()`.
3. Specify the reference to the bucket path with the artifact object's `wandb.Artifact.add_reference()` method.
4. Log the artifact's metadata with `run.log_artifact()`.

```python theme={null}
import wandb

# Initialize a W&B run
with wandb.init(project="my-project") as run:

  # Create an artifact object
  artifact = wandb.Artifact(name="name", type="type")

  # Add a reference to the bucket path
  artifact.add_reference(uri = "uri/to/your/bucket/path")

  # Log the artifact's metadata
  run.log_artifact(artifact)
```

As an example, suppose your bucket has the following directory structure:

```text theme={null}
s3://my-bucket

|datasets/
  |-- mnist/
|models/
  |-- cnn/
```

The `datasets/mnist/` directory contains a collection of images. To track the image `datasets/mnist/` directory as a dataset artifact, specify:

1. Provide a name for the artifact, such as `"mnist"`.
2. Set the `type` parameter to `"dataset"` when you construct the artifact object (`wandb.Artifact(type="dataset")`).
3. Provide the path to the `datasets/mnist/` directory as an Amazon S3 URI (`s3://my-bucket/datasets/mnist/`) when you call `wandb.Artifact.add_reference()`.
4. Log the artifact with `run.log_artifact()`.

The following code sample creates a reference artifact `mnist:latest`:

```python theme={null}
import wandb

with wandb.init(project="my-project") as run:
  artifact = wandb.Artifact(name="mnist", type="dataset")
  artifact.add_reference(uri="s3://my-bucket/datasets/mnist")
  run.log_artifact(artifact)
```

Within the W\&B App, you can look through the contents of the reference artifact using the file browser, [explore the full dependency graph](/models/artifacts/explore-and-traverse-an-artifact-graph/), and scan through the versioned history of your artifact. The W\&B App does not render rich media such as images, audio, and so forth because the data itself is not contained within the artifact.

<Note>
  W\&B Artifacts support any Amazon S3 compatible interface, including CoreWeave Storage and MinIO. The scripts described below work as-is with both providers, when you set the `AWS_S3_ENDPOINT_URL` environment variable to point at your CoreWeave Storage or MinIO server.
</Note>

<Warning>
  By default, W\&B imposes a 10,000 object limit when adding an object prefix. You can adjust this limit by specifying `max_objects=` when you call `wandb.Artifact.add_reference()`.
</Warning>

## Download an artifact from an external bucket

W\&B retrieves the files from the underlying bucket when it downloads a reference artifact using the metadata recorded when the artifact is logged. If your bucket has object versioning enabled, W\&B retrieves the object version that corresponds to the state of the file at the time an artifact was logged. As you evolve the contents of your bucket, you can always point to the exact version of your data a given model was trained on, because the artifact serves as a snapshot of your bucket during the training run.

The following code sample shows how to download a reference artifact. The APIs for downloading artifacts are the same for both reference and non-reference artifacts:

```python theme={null}
import wandb

with wandb.init(project="my-project") as run:
  artifact = run.use_artifact("mnist:latest", type="dataset")
  artifact_dir = artifact.download()
```

<Note>
  W\&B recommends that you enable 'Object Versioning' on your storage buckets if you overwrite files as part of your workflow.

  If versioning is enabled, W\&B can always retrieve the correct version of the file when you download an artifact, even if the file has been overwritten since the artifact was logged.

  Based on your use case, read the instructions to enable object versioning: [AWS](https://docs.aws.amazon.com/AmazonS3/latest/userguide/manage-versioning-examples.html), [Google Cloud](https://cloud.google.com/storage/docs/using-object-versioning#set), [Azure](https://learn.microsoft.com/azure/storage/blobs/versioning-enable).
</Note>

## Add and download an external from a bucket

The following code sample uploads a dataset to an Amazon S3 bucket, tracks it with a reference artifact, then downloads it:

```python theme={null}
import boto3
import wandb

with wandb.init() as run:
  # Training here...

  s3_client = boto3.client("s3")
  s3_client.upload_file(file_name="my_model.h5", bucket="my-bucket", object_name="models/cnn/my_model.h5")

  # Log the model artifact
  model_artifact = wandb.Artifact("cnn", type="model")
  model_artifact.add_reference("s3://my-bucket/models/cnn/")
  run.log_artifact(model_artifact)
```

At a later point, you can download the model artifact. Specify the name of the artifact and its type:

```python theme={null}
import wandb

with wandb.init() as run:
  artifact = run.use_artifact(artifact_or_name = "cnn", type="model")
  datadir = artifact.download()
```

<Note>
  See the following reports for an end-to-end walkthrough on how to track artifacts by reference for Google Cloud or Azure:

  * [Guide to Tracking Artifacts by Reference with Google Cloud](https://wandb.ai/stacey/artifacts/reports/Tracking-Artifacts-by-Reference--Vmlldzo1NDMwOTE)
  * [Working with Reference Artifacts in Microsoft Azure](https://wandb.ai/andrea0/azure-2023/reports/Efficiently-Harnessing-Microsoft-Azure-Blob-Storage-with-Weights-Biases--Vmlldzo0NDA2NDgw)
</Note>

## Cloud storage credentials

W\&B uses the default mechanism to look for credentials based on the cloud provider you use. Read the documentation from your cloud provider to learn more about the credentials used:

| Cloud provider              | Credentials Documentation                                                                                                                                   |
| --------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- |
| CoreWeave AI Object Storage | [CoreWeave AI Object Storage documentation](https://docs.coreweave.com/docs/products/storage/object-storage/how-to/manage-access-keys/cloud-console-tokens) |
| AWS                         | [Boto3 documentation](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html#configuring-credentials)                               |
| Google Cloud                | [Google Cloud documentation](https://cloud.google.com/docs/authentication/provide-credentials-adc)                                                          |
| Azure                       | [Azure documentation](https://learn.microsoft.com/python/api/azure-identity/azure.identity.defaultazurecredential?view=azure-python)                        |

For AWS, if the bucket is not located in the configured user's default region, you must set the `AWS_REGION` environment variable to match the bucket region.

<Warning>
  Rich media such as images, audio, video, and point clouds may fail to render in the App UI depending on the CORS configuration of your bucket. Allow listing **app.wandb.ai** in your bucket's CORS settings will allow the W\&B App to properly render such rich media.

  If rich media such as images, audio, video, and point clouds does not render in the App UI, ensure that `app.wandb.ai` is allowlisted in your bucket's CORS policy.
</Warning>

## Track an artifact in a filesystem

A common pattern for accessing datasets is to expose an NFS mount point to a remote filesystem on all machines running training jobs. This can be an alternative solution to a cloud storage bucket because from the perspective of the training script, the files appear local to your filesystem.

To track an artifact in a filesystem:

1. Initialize a run with `wandb.init()`.
2. Create an artifact object with `wandb.Artifact()`.
3. Specify the reference to the filesystem path with the artifact object's `wandb.Artifact.add_reference()` method.
4. Log the artifact's metadata with `run.log_artifact()`.

Copy and paste the following code snippet to track files in a mounted filesystem. Replace the values enclosed in angle brackets (`< >`) with your own values.

```python theme={null}
import wandb

# Initialize a run
with wandb.init(entity="<entity>", project="<project>") as run:

  # Create an artifact object
  artifact = wandb.Artifact(name="<name>", type="<type>")

  # Add a reference to the filesystem path
  artifact.add_reference("file:///path/to/dataset/")

  # Log the artifact (metadata only)
  run.log_artifact(artifact)
```

Note the triple slash in the URL. The first component is the `file://` prefix that denotes the use of filesystem references. The second component is the root `/` of the filesystem. The remaining components are the path to the directory or file you want to track.

As an example, suppose you have a filesystem mounted at `/mount` with the following structure:

```text theme={null}
mount
|datasets/
  |-- mnist/
|models/
  |-- cnn/
```

You want to track the `datasets/mnist/` directory as a dataset artifact. To track it, you could use the following code snippet.

```python theme={null}
import wandb

with wandb.init() as run:
  artifact = wandb.Artifact("mnist", type="dataset")
  artifact.add_reference("file:///mount/datasets/mnist/")
  run.log_artifact(artifact)
```

This creates a reference artifact `mnist:latest` that points to the files stored under `/mount/datasets/mnist/`.

<Warning>
  By default, W\&B imposes a 10,000 file limit when adding a reference to a directory. You can adjust this limit by specifying `max_objects=` when you call `wandb.Artifact.add_reference()`.
</Warning>

Similarly, to track a model stored at `models/cnn/my_model.h5`, you could use the following code snippet:

```python theme={null}
import wandb

with wandb.init() as run:

  # Training here...

  # Write model to disk

  # Create an artifact object
  model_artifact = wandb.Artifact("cnn", type="model")

  # Add a reference to the model file path
  model_artifact.add_reference("file:///mount/cnn/my_model.h5")

  # Log the artifact to W&B
  run.log_artifact(model_artifact)
```

## Download an artifact from an external filesystem

Download files from a referenced filesystem using the same APIs as non-reference artifacts:

1. Initialize a run with `wandb.init()`.
2. Use the `wandb.Run.use_artifact()` method to indicate the artifact you want to download.
3. Call the artifact's `wandb.Artifact.download()` method to download the files from the referenced filesystem

```python theme={null}
with wandb.init() as run:
  artifact = run.use_artifact("entity/project/mnist:latest", type="dataset")
  artifact_dir = artifact.download()
```

W\&B copies the contents of `/mount/datasets/mnist` to the `artifacts/mnist:v0/` directory.

<Info>
  `Artifact.download()` throws an error if it cannot reconstruct the artifact. For example, if an artifact contains a reference to a file that was overwritten, `Artifact.download()` will throw an error because the artifact can no longer be reconstructed.
</Info>


# Manage artifact data retention
Source: https://docs.wandb.ai/models/artifacts/ttl

Set time-to-live (TTL) policies on W&B artifacts to schedule automatic deletion and manage storage consumption.

<ColabLink />

Schedule when artifacts are deleted from W\&B with a W\&B Artifact time-to-live (TTL) policy. When you delete an artifact, W\&B marks that artifact as a *soft-delete*. In other words, the artifact is marked for deletion but files are not immediately deleted from storage. For more information on how W\&B deletes artifacts, see the [Delete artifacts](./delete-artifacts) page.

Watch a [Managing data retention with Artifacts TTL](https://www.youtube.com/watch?v=hQ9J6BoVmnc) video tutorial to learn how to manage data retention with Artifacts TTL in the W\&B App.

<Note>
  W\&B deactivates the option to set a TTL policy for artifacts linked to the Registry. This is to help ensure that linked artifacts do not accidentally expire if used in production workflows.
</Note>

<Note>
  * Only team admins can view a [team's settings](/platform/app/settings-page/teams) and access team level TTL settings such as (1) permitting who can set or edit a TTL policy or (2) setting a team default TTL.
  * If you do not see the option to set or edit a TTL policy in an artifact's details in the W\&B App UI or if setting a TTL programmatically does not successfully change an artifact's TTL property, your team admin has not given you permissions to do so.
</Note>

## Auto-generated Artifacts

Only user-generated artifacts can use TTL policies. Artifacts auto-generated by W\&B cannot have TTL policies set for them.

The following Artifact types indicate an auto-generated Artifact:

* `run_table`
* `code`
* `job`
* Any Artifact type starting with: `wandb-*`

You can check an Artifact's type on the [W\&B platform](/models/artifacts/explore-and-traverse-an-artifact-graph/) or programmatically:

```python theme={null}
import wandb

with wandb.init(project="<my-project-name>") as run:
    artifact = run.use_artifact(artifact_or_name="<my-artifact-name>")
    print(artifact.type)
```

Replace the values enclosed with `<>` with your own.

## Define who can edit and set TTL policies

Define who can set and edit TTL policies within a team. You can either grant TTL permissions only to team admins, or you can grant both team admins and team members TTL permissions.

<Note>
  Only team admins can define who can set or edit a TTL policy.
</Note>

1. Navigate to your team’s profile page.
2. Select the **Settings** tab.
3. Navigate to the **Artifacts time-to-live (TTL) section**.
4. From the **TTL permissions dropdown**, select who can set and edit TTL policies.
5. Click on **Review and save settings**.
6. Confirm the changes and select **Save settings**.

<Frame>
  <img alt="Setting TTL permissions" />
</Frame>

## Create a TTL policy

Set a TTL policy for an artifact either when you create the artifact or retroactively after the artifact is created.

For all the code snippets below, replace the content wrapped in `<>` with your information to use the code snippet.

### Set a TTL policy when you create an artifact

Use the W\&B Python SDK to define a TTL policy when you create an artifact. TTL policies are typically defined in days.

<Note>
  Defining a TTL policy when you create an artifact is similar to how you normally [create an artifact](/models/artifacts/construct-an-artifact/). With the exception that you pass in a time delta to the artifact's `ttl` attribute.
</Note>

The steps are as follows:

1. [Create an artifact](/models/artifacts/construct-an-artifact/).
2. [Add content to the artifact](/models/artifacts/construct-an-artifact/#add-files-to-an-artifact) such as files, a directory, or a reference.
3. Define a TTL time limit with the [`datetime.timedelta`](https://docs.python.org/3/library/datetime.html) data type that is part of Python's standard library.
4. [Log the artifact](/models/artifacts/construct-an-artifact/#3-save-your-artifact-to-the-wb-server).

The following code snippet demonstrates how to create an artifact and set a TTL policy.

```python theme={null}
import wandb
from datetime import timedelta

with wandb.init(project="<my-project-name>", entity="<my-entity>") as run:
    artifact = wandb.Artifact(name="<artifact-name>", type="<type>")
    artifact.add_file("<my_file>")

    artifact.ttl = timedelta(days=30)  # Set TTL policy
    run.log_artifact(artifact)
```

The preceding code snippet sets the TTL policy for the artifact to 30 days. In other words, W\&B deletes the artifact after 30 days.

### Set or edit a TTL policy after you create an artifact

Use the W\&B App UI or the W\&B Python SDK to define a TTL policy for an artifact that already exists.

<Note>
  When you modify an artifact's TTL, the time the artifact takes to expire is still calculated using the artifact's `createdAt` timestamp.
</Note>

<Tabs>
  <Tab title="Python SDK">
    1. [Fetch your artifact](/models/artifacts/download-and-use-an-artifact/).
    2. Pass in a time delta to the artifact's `ttl` attribute.
    3. Update the artifact with the [`save`](/models/ref/python/experiments/run#save) method.

    The following code snippet shows how to set a TTL policy for an artifact:

    ```python theme={null}
    import wandb
    from datetime import timedelta

    artifact = run.use_artifact("<my-entity/my-project/my-artifact:alias>")
    artifact.ttl = timedelta(days=365 * 2)  # Delete in two years
    artifact.save()
    ```

    The preceding code example sets the TTL policy to two years.
  </Tab>

  <Tab title="W&B App">
    1. Navigate to your W\&B project in the W\&B App UI.
    2. Select the artifact icon in the project sidebar.
    3. From the list of artifacts, expand the artifact type you
    4. Select on the artifact version you want to edit the TTL policy for.
    5. Click on the **Version** tab.
    6. From the dropdown, select **Edit TTL policy**.
    7. Within the modal that appears, select **Custom** from the TTL policy dropdown.
    8. Within the **TTL duration** field, set the TTL policy in units of days.
    9. Select the **Update TTL** button to save your changes.

    <Frame>
      <img alt="Editing TTL policy" />
    </Frame>
  </Tab>
</Tabs>

### Set default TTL policies for a team

<Note>
  Only team admins can set a default TTL policy for a team.
</Note>

Set a default TTL policy for your team. Default TTL policies apply to all existing and future artifacts based on their respective creation dates. Artifacts with existing version-level TTL policies are not affected by the team's default TTL.

1. Navigate to your team’s profile page.
2. Select the **Settings** tab.
3. Navigate to the **Artifacts time-to-live (TTL) section**.
4. Click on the **Set team's default TTL policy**.
5. Within the **Duration** field, set the TTL policy in units of days.
6. Click on **Review and save settings**.
   7/ Confirm the changes and then select **Save settings**.

<Frame>
  <img alt="Setting default TTL policy" />
</Frame>

### Set a TTL policy outside of a run

Use the public API to retrieve an artifact without fetching a run, and set the TTL policy. TTL policies are typically defined in days.

The following code sample shows how to fetch an artifact using the public API and set the TTL policy.

```python theme={null}
api = wandb.Api()

artifact = api.artifact("entity/project/artifact:alias")

artifact.ttl = timedelta(days=365)  # Delete in one year

artifact.save()
```

## Deactivate a TTL policy

Use the W\&B Python SDK or W\&B App UI to deactivate a TTL policy for a specific artifact version.

<Tabs>
  <Tab title="Python SDK">
    1. [Fetch your artifact](/models/artifacts/download-and-use-an-artifact/).
    2. Set the artifact's `ttl` attribute to `None`.
    3. Update the artifact with the [`save`](/models/ref/python/experiments/run#save) method.

    The following code snippet shows how to turn off a TTL policy for an artifact:

    ```python theme={null}
    artifact = run.use_artifact("<my-entity/my-project/my-artifact:alias>")
    artifact.ttl = None
    artifact.save()
    ```
  </Tab>

  <Tab title="W&B App">
    1. Navigate to your W\&B project in the W\&B App UI.
    2. Select the artifact icon in the project sidebar.
    3. From the list of artifacts, expand the artifact type you
    4. Select on the artifact version you want to edit the TTL policy for.
    5. Click on the Version tab.
    6. Click the **action (<Icon icon="ellipsis" />)** menu next to the **Link to registry** button.
    7. From the dropdown, select **Edit TTL policy**.
    8. Within the modal that appears, select **Deactivate** from the TTL policy dropdown.
    9. Select the **Update TTL** button to save your changes.

    <Frame>
      <img alt="Removing TTL policy" />
    </Frame>
  </Tab>
</Tabs>

## View TTL policies

View TTL policies for artifacts with the Python SDK or with the W\&B App UI.

<Tabs>
  <Tab title="Python SDK">
    Use a print statement to view an artifact's TTL policy. The following example shows how to retrieve an artifact and view its TTL policy:

    ```python theme={null}
    artifact = run.use_artifact("<my-entity/my-project/my-artifact:alias>")
    print(artifact.ttl)
    ```
  </Tab>

  <Tab title="W&B App">
    View a TTL policy for an artifact with the W\&B App UI.

    1. Navigate to the [W\&B App](https://wandb.ai).
    2. Navigate to your W\&B Project.
    3. Within your project, select the Artifacts tab in the project sidebar.
    4. Click on a collection.

    Within the collection view you can see all of the artifacts in the selected collection. Within the `Time to Live` column you will see the TTL policy assigned to that artifact.

    <Frame>
      <img alt="TTL collection view" />
    </Frame>
  </Tab>
</Tabs>


# Update an artifact
Source: https://docs.wandb.ai/models/artifacts/update-an-artifact

Update an existing artifact while a run is active or using only the Public API.

Pass desired values to update the `description`, `metadata`, and `alias` of an artifact. Update a run previously logged to W\&B with the W\&B Public API with ([`wandb.Api`](/models/ref/python/public-api/api)). Use `wandb.Run.save()` to update an artifact when is first initialized and still active.

<Note>
  **When to use wandb.Artifact.save() or wandb.Run.log\_artifact()**

  * Use `Artifact.save()` to update an existing artifact without starting a new run.
  * Use `wandb.Run.log_artifact()` to create a new artifact and associate it with a specific run.
</Note>

Use the W\&B Public API ([`wandb.Api`](/models/ref/python/public-api/api)) to update an artifact. Use the wandb.Artifact ([`wandb.Artifact`](/models/ref/python/experiments/artifact)) Class while a run is active.

<Warning>
  You can not update the alias of artifact linked to a model in Model Registry.
</Warning>

<Tabs>
  <Tab title="During a run">
    The following code example demonstrates how to update the description of an artifact using the [`wandb.Artifact`](/models/ref/python/experiments/artifact) API:

    ```python theme={null}
    import wandb

    with wandb.init(project="<example>") as run:
        artifact = run.use_artifact("<artifact-name>:<alias>")
        artifact.description = "<description>"
        artifact.save()
    ```
  </Tab>

  <Tab title="W&B Public API">
    The following example updates an artifact with [`wandb.Api`](/models/ref/python/public-api/api):

    ```python theme={null}
    import wandb

    api = wandb.Api()

    artifact = api.artifact("entity/project/artifact:alias")

    # Update the description
    artifact.description = "My new description"

    # Selectively update metadata keys
    artifact.metadata["oldKey"] = "new value"

    # Replace the metadata entirely
    artifact.metadata = {"newKey": "new value"}

    # Add an alias
    artifact.aliases.append("best")

    # Remove an alias
    artifact.aliases.remove("latest")

    # Completely replace the aliases
    artifact.aliases = ["replaced"]

    # Persist all artifact modifications
    artifact.save()
    ```

    For more information, see the Weights and Biases [Artifact API](/models/ref/python/experiments/artifact).
  </Tab>

  <Tab title="With collections">
    You can also update an Artifact collection in the same way as a singular artifact:

    ```python theme={null}
    import wandb
    with wandb.init(project="<example>") as run:
        api = wandb.Api()
        artifact = api.artifact_collection(type="<type-name>", collection="<collection-name>")
        artifact.name = "<new-collection-name>"
        artifact.description = "<This is where you'd describe the purpose of your collection.>"
        artifact.save()
    ```

    For more information, see the [Artifacts Collection](/models/ref/python/public-api/api) reference.
  </Tab>
</Tabs>


# Automations overview
Source: https://docs.wandb.ai/models/automations

Use W&B Automations for triggering workflows based on events in W&B

Automations exist for both **projects** and **registries**. Where you create an automation, which events you can use, and how scope works all differ. For event types by scope, see [Automation events and scopes](/models/automations/automation-events).

W\&B Automations follow this pattern: when an **event** occurs and optional **conditions** are met, an **action** runs automatically. For example:

* When a run fails (event), notify a Slack channel (action).
  <TutorialDiagramProject />
* When the `production` alias is added to an artifact (event), call a webhook to trigger deployment (action).
  <TutorialDiagramRegistry />

Events and available conditions differ for automations scoped to a [project](/models/automations/automation-events#project) or a [registry](/models/automations/automation-events#registry). See [Automation events and scopes](/models/automations/automation-events).

```mermaid theme={null}
flowchart LR
  Event[Event] --> Condition[Condition]
  Condition --> Action[Action]
```

**Example:** Run fails (event) and optional run name filter (condition) then Slack notification (action). Or: alias `production` added (event) then webhook (action).

## Where to create automations

* **In a project**: Open the project, then select the **Automations** tab in the project sidebar.
* **In a registry**: Open the registry, then select the **Automations** tab.

## Use cases

* **Run monitoring and alerting**: Notify the team when a run fails or when a metric crosses a threshold (for example, loss goes to NaN or accuracy drops).
* **Registry CI/CD**: When a new model version is linked or an alias (such as `staging` or `production`) is added, trigger a webhook to run tests or deploy.
* **Project artifact workflows**: When a new artifact version is created or an alias is added in a project, run a downstream job or post to Slack.

For full event and scope details, see [Automation events and scopes](/models/automations/automation-events).

## Automation actions

When an event triggers an automation, it can perform one of these actions:

* **Slack notification**: Send a message to a Slack channel with details about the triggering event. The message summarizes the event, with a link to view more details in W\&B.
* **Webhook**: Call a webhook URL with a JSON payload containing information about the triggering event. This enables integration with external systems such as CI/CD pipelines, model deployment services, or custom workflows.

For implementation details, see [Create a Slack automation](/models/automations/create-automations/slack) and [Create a webhook automation](/models/automations/create-automations/webhook).

## How automations work

To [create an automation](/models/automations/create-automations), you:

1. If required, configure [secrets](/platform/secrets) for sensitive strings the automation requires, such as access tokens, passwords, or sensitive configuration details. Secrets are defined in your **Team Settings**. Secrets are most commonly used in webhook automations to securely pass credentials or tokens to the webhook's external service without exposing it in plain text or hard-coding it in the webhook's payload.
2. Configure team-level webhook or Slack integrations to authorize W\&B to post to Slack or run the webhook on your behalf. A single automation action (webhook or Slack notification) can be used by multiple automations. These actions are defined in your **Team Settings**.
3. In the project or registry, create the automation:
   1. Define the [event](/models/automations/automation-events) to watch for, such as when a new artifact version is added.
   2. Define the action to take when the event occurs (posting to a Slack channel or running a webhook). For a webhook, specify a secret to use for the access token and/or a secret to send with the payload, if required.

## Recommendations

* **Start small**: Begin with one or two automations for high-value events (for example, run failures or production alias changes). Validate that they work as expected before adding more.
* **Test before production**: Create automations in a test project or with a test webhook or Slack channel first. Manually trigger the event and confirm the action runs and the payload or message looks correct.
* **Avoid alert fatigue**: Use run filters, metric thresholds, or alias patterns to limit how often an automation fires. If you have multiple severities, route them to different channels.

## Limitations

[Run metric automations](/models/automations/automation-events/#run-metrics-events) and [run metrics z-score change automations](/models/automations/automation-events/#run-metrics-z-score-change-automations) are currently supported only in [W\&B Multi-tenant Cloud](/platform/hosting/#wb-multi-tenant-cloud).

## Next steps

* [Automations tutorial](/models/automations/tutorial): Guides you to create a project automation to alert on run failures and a Registry automation to run a webhoook when an alias is added. The tutorial uses the W\&B App.
* [Create an automation](/models/automations/create-automations).
* [Automation events and scopes](/models/automations/automation-events).
* [Create a secret](/platform/secrets).

<Info>
  Looking for companion tutorials for automations?

  * [Learn to automatically trigger a Github Action for model evaluation and deployment](https://wandb.ai/wandb/wandb-model-cicd/reports/Model-CI-CD-with-W-B--Vmlldzo0OTcwNDQw).
  * [Watch a video demonstrating automatically deploying a model to a Sagemaker endpoint](https://www.youtube.com/watch?v=s5CMj_w3DaQ).
  * [Watch a video series introducing automations](https://youtube.com/playlist?list=PLD80i8An1OEGECFPgY-HPCNjXgGu-qGO6\&feature=shared).
</Info>


# Manage automations with the API
Source: https://docs.wandb.ai/models/automations/api

Programmatic automation management with the Python API. Create and update may be affected on some client versions. Prefer the W&B App until the SDK fix ships.

Manage W\&B automations programmatically with the `wandb` Python API when you script automation workflows instead of using the W\&B App.

<Note>
  Programmatic **create** and **update** for automations through `wandb.Api().create_automation()` and related helpers can fail on some `wandb` Python client versions because of a client-side feature check (for example, run-state automations). Until a fixed SDK ships, use the [W\&B App](/models/automations/create-automations) to create and edit automations. The `list`, `get`, and `delete` methods aren't affected. Internal tracking: WB-34263.
</Note>

## Next steps

Refer to the following resources to manage automations and learn more about how they work:

* [Automations overview](/models/automations)
* [Create an automation](/models/automations/create-automations) (W\&B App)
* [Automation events and scopes](/models/automations/automation-events)
* [Automations API reference](/models/ref/python/public-api/automations) (Python types and `Api` methods)


# Automation events and scopes
Source: https://docs.wandb.ai/models/automations/automation-events

Learn about events and scopes that trigger W&B Automations, including artifact changes, run status, and metric conditions.

An automation can start when a specific event occurs within a project or registry. This page describes the events that can trigger an automation within each scope, so you can choose the right trigger when you configure an automation. Learn more about automations in the [Automations overview](/models/automations) or [Create an automation](/models/automations/create-automations).

## Registry

The following sections describe the scopes and events for an automation in a [Registry](/models/registry).

### Scopes

A [Registry](/models/registry) automation watches for the event taking place on any collection within a specific registry, including collections added in the future.

### Events <a aria-label="Registry events" />

A Registry automation can watch for these events:

* **A new version is linked to a collection**: Test and validate new models or datasets when they're added to a registry.
* **An artifact alias is added**: Trigger a specific step of your workflow when a new artifact version has a specific alias applied. For example, deploy a model when it has the `production` alias applied.

When the automation calls a webhook, it can access the same team-level webhook configurations and [team secrets](/platform/secrets) as project-scoped automations.

## Project

The following sections describe the scopes and events for an automation in a [project](/models/track/project-page).

### Scopes

A project-level automation watches for the event taking place on any collection in the project. Depending on the event you specify, you can further limit the scope of the automation.

### Artifact events

This section describes the events related to an artifact that can trigger an automation.

* **A new version is added to an artifact**: Apply recurring actions to each version of an artifact. For example, start a training job when a new dataset artifact version is created.
* **An artifact alias is added**: Trigger a specific step of your workflow when a new artifact version in a project has an alias applied that matches the **Alias regex** you specify. For example, run a series of downstream processing steps when an artifact has the `test-set-quality-check` alias applied, or run a workflow each time a new artifact version has the `latest` alias. Only one artifact version can have a given alias at a time.
* **An artifact tag is added**: Trigger a specific step of your workflow when an artifact version in a project has a tag applied that matches the **Tag regex** you specify. For example, specify `^europe.*` to trigger a geo-specific workflow when a tag beginning with the string `europe` is added to an artifact version. Use artifact tags for grouping and filtering. You can assign the same tag to multiple artifact versions.

### Run events

The following sections describe how to configure an automation that starts based on a change in a [run's status](/models/runs/run-states) or a change in a [run's metric value](/models/track/log#what-data-is-logged-with-specific-wb-api-calls).

#### Run status change

<Note>
  * Currently available only in [W\&B Multi-tenant Cloud](/platform/hosting#wb-multi-tenant-cloud) and [Dedicated Cloud](/platform/hosting#wb-dedicated-cloud).
  * A run with **Killed** status can't trigger an automation. This status indicates that an admin forcibly stopped the run.
</Note>

Trigger a workflow when a run changes its [status](/models/runs/run-states) to **Running**, **Finished**, or **Failed**. Optionally, you can further limit the runs that can trigger an automation by specifying a user or run name filter.

<Frame>
  <img alt="Screenshot showing a run status change automation" />
</Frame>

Because run status is a property of the entire run, you can create a run status automation only from the **Automations** page, not from a workspace.

#### Run metrics change

<Note>
  Currently available only in [W\&B Multi-tenant Cloud](/platform/hosting#wb-multi-tenant-cloud) and [Dedicated Cloud](/platform/hosting#wb-dedicated-cloud).
</Note>

Trigger a workflow based on a logged value for a metric, either a metric in a run's history or a [system metric](/models/ref/python/experiments/system-metrics) such as `cpu`, which tracks the percentage of CPU utilization. W\&B logs system metrics automatically every 15 seconds.

You can create a run metrics automation from the project's **Automations** tab or directly from a line plot panel in a workspace.

To set up a run metric automation, configure how to compare the metric's value with the threshold you specify. Your choices depend on the event type and on any filters you specify.

Optionally, you can further limit the runs that can trigger an automation by specifying a user or run name filter.

##### Threshold

Use a threshold event to start an automation when a metric crosses a fixed value. For **Run metrics threshold met** events, configure:

1. The window of most recently logged values to consider (defaults to 5).
2. Whether to evaluate the **Average**, **Min**, or **Max** value within the window.
3. The comparison to make:
   * Above
   * Above or equal to
   * Below
   * Below or equal to
   * Not equal to
   * Equal to

For example, trigger an automation when average `accuracy` is above `0.6`.

<Frame>
  <img alt="Screenshot showing a run metrics threshold automation" />
</Frame>

##### Change threshold

Use a change threshold event to start an automation when a metric shifts between two recent windows of values. For **Run metrics change threshold met** events, the automation uses two "windows" of values to check whether to start:

* The *current window* of recently logged values to consider (defaults to 10).
* The *prior window* of recently logged values to consider (defaults to 50).

The current and prior windows are consecutive and don't overlap.

To create the automation, configure:

1. The current window of logged values (defaults to 10).
2. The prior window of logged values (defaults to 50).
3. Whether to evaluate the values as relative or absolute (defaults to **Relative**).
4. The comparison to make:
   * Increases by at least
   * Decreases by at least
   * Increases or decreases by at least

For example, trigger an automation when average `loss` decreases by at least `0.25`.

<Frame>
  <img alt="Screenshot showing a run metrics change threshold automation" />
</Frame>

#### Run metrics z-score change

<Note>
  Currently available only in [W\&B Multi-tenant Cloud](/platform/hosting#wb-multi-tenant-cloud) and [Dedicated Cloud](/platform/hosting#wb-dedicated-cloud).
</Note>

W\&B can trigger an automation when a metric's z-score (standard score) exceeds a specified threshold. A z-score measures how many standard deviations the value is from the mean for that metric across a configurable window of runs in the project (defaults to 30 runs).

To use a z-score as an event trigger, select the **Run metrics z-score threshold met** event.

Automations based on z-score keep your team informed about unusual performance without checking absolute thresholds, which may vary as your model or training process evolves.

You can create a run metrics z-score automation from the project's **Automations** tab or directly from a line plot panel in a workspace.

To create a z-score automation, configure:

1. The target z-score threshold, expressed as a positive float value (for example, 2.0).
2. The window of logged values that determine the mean value (defaults to 30).
3. The comparison to make:
   * Above (triggers when performance is unusually high).
   * Below (triggers when performance is unusually low).
   * Either above or below.

For example, trigger an automation when `accuracy` has a z-score above 2, which means the run is performing well above other runs in the project.

Z-score values have the following meanings:

* A z-score of 0 means the metric is at the average.
* A z-score of +2.0 means the metric is 2 standard deviations above average.
* A z-score of -2.0 means the metric is 2 standard deviations below average.
* Values beyond ±2 are often considered statistically significant outliers.

#### Run filters

This section describes how the automation selects runs to evaluate.

By default, any run in the project triggers the automation when the event occurs. You can limit which runs trigger an automation by configuring one of the following filters:

| Filter                        | Description                                                       |
| ----------------------------- | ----------------------------------------------------------------- |
| **Filter to one user's runs** | Include only runs created by the specified user.                  |
| **Filter on run name**        | Include only runs whose names match the given regular expression. |

The automation evaluates each run as follows:

* Each run is considered individually and can potentially trigger the automation.
* Each run's values are put into a separate window and compared to the threshold separately.
* In a 24-hour period, a particular automation can fire at most once per run.

For details, see [Create automations](/models/automations/create-automations).

## Next steps

* [Create a Slack automation](/models/automations/create-automations/slack)
* [Create a webhook automation](/models/automations/create-automations/webhook)


# Overview
Source: https://docs.wandb.ai/models/automations/create-automations

Create and manage W&B automations to streamline your ML workflows

This page shows how to create and manage W\&B [automations](/models/automations), which let you trigger Slack notifications or webhooks in response to project and registry events. Use this overview to understand the prerequisites and high-level workflow, then follow the detailed instructions in [Create a Slack automation](/models/automations/create-automations/slack) or [Create a webhook automation](/models/automations/create-automations/webhook).

<Note>
  Looking for companion tutorials for automations?

  * [Learn to automatically trigger a GitHub Action for model evaluation and deployment](https://wandb.ai/wandb/wandb-model-cicd/reports/Model-CI-CD-with-W-B--Vmlldzo0OTcwNDQw).
  * [Watch a video demonstrating automatically deploying a model to a SageMaker endpoint](https://www.youtube.com/watch?v=s5CMj_w3DaQ).
  * [Watch a video series introducing automations](https://youtube.com/playlist?list=PLD80i8An1OEGECFPgY-HPCNjXgGu-qGO6\&feature=shared).
</Note>

## Requirements

When creating automations:

* A team admin can create and manage automations for the team's projects, as well as components of their automations, such as webhooks, secrets, and Slack integrations. Refer to [Team settings](/platform/app/settings-page/teams).
* To create a registry automation, you must have access to the registry. Refer to [Configure Registry access](/models/registry/configure_registry/#registry-roles).
* To create a Slack automation, you must have permission to post to the Slack instance and channel you select.

## Create an automation

You can create an automation from the project or registry's **Automations** tab, or directly from a line plot panel in the workspace.

### Create from the Automations tab

At a high level, to create an automation from the **Automations** tab, follow these steps:

1. If necessary, [create a W\&B secret](/platform/secrets) for each sensitive string required by the automation, such as an access token, password, or SSH key. Secrets let you reuse sensitive strings safely and keep them out of automation configurations. Define secrets in your **Team Settings**. Webhook automations most commonly use secrets.
2. Configure team-level webhook or Slack integrations to authorize W\&B to post to Slack or run the webhook on your behalf. Multiple automations can use a single webhook or Slack integration. Define these actions in your **Team Settings**.
3. In the project or registry, create the automation, which specifies the event to watch for and the action to take (such as posting to Slack or running a webhook). When you create a webhook automation, you configure the payload it sends.

After you complete these steps, the automation runs whenever the specified event occurs in the project or registry.

### Create a run metric automation from a workspace panel

From a line plot in the workspace, you can also create a [run metric automation](/models/automations/automation-events/#run-events) for the metric it shows:

1. Hover over the panel, then click the bell icon at the top of the panel.

   <Frame>
     <img alt="Automation bell icon location" />
   </Frame>
2. Configure the automation using the basic or advanced configuration controls. For example, apply a run filter to limit the scope of the automation, or configure an absolute threshold.

For details, refer to:

* [Create a Slack automation](/models/automations/create-automations/slack)
* [Create a webhook automation](/models/automations/create-automations/webhook)

## View and manage automations

View and manage automations from a project or registry's **Automations** tab:

* To view an automation's details, click its name.
* To view an automation's execution history (which actions ran and whether they succeeded), click its name and select the **History** tab. For more information, see [View an automation's history](/models/automations/view-automation-history).
* To edit an automation, click its **action (<Icon icon="ellipsis" />)** menu, then click **Edit automation**.
* To delete an automation, click its **action (<Icon icon="ellipsis" />)** menu, then click **Delete automation**.

## Go further

For more information, see the following resources:

* [Automations tutorials](/models/automations/tutorial): Build a [project run failure alert](/models/automations/project-automation-tutorial) or a [registry alias-to-webhook automation](/models/automations/registry-automation-tutorial) in the W\&B App.
* [Automation events and scopes](/models/automations/automation-events).
* [Create a Slack automation](/models/automations/create-automations/slack).
* [Create a webhook automation](/models/automations/create-automations/webhook).
* [Create a secret](/platform/secrets).


# Create a Slack automation
Source: https://docs.wandb.ai/models/automations/create-automations/slack

Set up a Slack integration and create a W&B Automation that sends notifications to a Slack channel on specific events.

This page shows how to create a Slack [automation](/models/automations) so that your team receives notifications in a Slack channel when specific events occur in W\&B. Example events include a new model version or a run reaching a metric threshold. To create a webhook automation instead, refer to [Create a webhook automation](/models/automations/create-automations/webhook).

At a high level, to create a Slack automation, you take these steps:

1. [Add a Slack integration](#add-a-slack-integration), which authorizes W\&B to post to the Slack instance and channel.
2. [Create the automation](#create-an-automation), which defines the [event](/models/automations/automation-events) to watch for and the channel to notify.

## Add a Slack integration

A team admin can add a Slack integration to the team.

1. Log in to W\&B and go to **Team Settings**.
2. In the **Slack channel integrations** section, click **Connect Slack** to add a new Slack instance. To add a channel for an existing Slack instance, click **New integration**.

   <img alt="Two Slack integrations in a Team" />
3. If necessary, sign in to Slack in your browser. When prompted, grant W\&B permission to post to the Slack channel you select. Read the page, then click **Search for a channel** and begin typing the channel name. Select the channel from the list, then click **Allow**.
4. In Slack, go to the channel you selected. If you see a post like `[YOUR-SLACK-HANDLE] added an integration to this channel: Weights & Biases`, where `[YOUR-SLACK-HANDLE]` is your Slack username, the integration is configured correctly.

Now you can [create an automation](#create-an-automation) that notifies the Slack channel you configured.

## View and manage Slack integrations

A team admin can view and manage the team's Slack instances and channels.

1. Log in to W\&B and go to **Team Settings**.
2. View each Slack destination in the **Slack channel integrations** section.
3. Delete a destination by clicking its trash icon.

## Create an automation

After you [add a Slack integration](#add-a-slack-integration), you can create an automation that uses it to send notifications. Select **Registry** or **Project** based on the scope you want the automation to apply to. Then follow these steps to create an automation that notifies the Slack channel.

<Tabs>
  <Tab title="Registry">
    A Registry admin can create automations in that registry.

    1. Log in to W\&B.
    2. Click the name of a registry to view its details.
    3. To create an automation scoped to the registry, click the **Automations** tab, then click **Create automation**. An automation that is scoped to a registry is automatically applied to all of its collections (including those created in the future).
    4. Choose the [event](/models/automations/automation-events/#registry-events) to watch for.

       Fill in any additional fields that appear, which depend upon the event. For example, if you select **An artifact alias is added**, you must specify the **Alias regex**.

       Click **Next step**.
    5. Select the team that owns the [Slack integration](#add-a-slack-integration).
    6. Set **Action type** to **Slack notification**. Select the Slack channel, then click **Next step**.
    7. Provide a name for the automation. Optionally, provide a description.
    8. Click **Create automation**.

    The automation is now active and notifies the selected Slack channel whenever the chosen event occurs in the registry.
  </Tab>

  <Tab title="Project">
    A W\&B admin can create automations in a project.

    1. Log in to W\&B.
    2. Go to the project page and click the **Automations** tab, then click **Create automation**.

       Or, directly from a line plot in the workspace, you can create a [run metric automation](/models/automations/automation-events/#run-events) for the metric it shows. Hover over the panel, then click the bell icon at the top of the panel.

       <Frame>
         <img alt="Automation bell icon location" />
       </Frame>
    3. Choose the [event](/models/automations/automation-events/#project) to watch for.

       1. Fill in any additional fields that appear. For example, if you select **An artifact alias is added**, you must specify the **Alias regex**.

          1. For automations triggered by a run, optionally specify one or more run filters:

             * **Filter to one user's runs**: Include only runs created by the specified user. Click the toggle to turn on the filter, then specify a username.
             * **Filter on run name**: Include only runs whose names match the given regular expression. Click the toggle to turn on the filter, then specify a regular expression.
       2. Click **Next step**.
    4. Select the team that owns the [Slack integration](#add-a-slack-integration).
    5. Set **Action type** to **Slack notification**. Select the Slack channel, then click **Next step**.
    6. Provide a name for the automation. Optionally, provide a description.
    7. Click **Create automation**.

    The automation is now active and notifies the selected Slack channel whenever the chosen event occurs in the project.
  </Tab>
</Tabs>

## View and manage automations

<Tabs>
  <Tab title="Registry">
    Manage the registry's automations from the registry's **Automations** tab:

    * To view an automation's details, click its name.
    * To edit an automation, click its **action (<Icon icon="ellipsis" />)** menu, then click **Edit automation**.
    * To delete an automation, click its **action (<Icon icon="ellipsis" />)** menu, then click **Delete automation**. Confirmation is required.
  </Tab>

  <Tab title="Project">
    A W\&B admin can view and manage a project's automations from the project's **Automations** tab.

    * To view an automation's details, click its name.
    * To edit an automation, click its **action (<Icon icon="ellipsis" />)** menu, then click **Edit automation**.
    * To delete an automation, click its **action (<Icon icon="ellipsis" />)** menu, then click **Delete automation**. Confirmation is required.
  </Tab>
</Tabs>


# Create a webhook automation
Source: https://docs.wandb.ai/models/automations/create-automations/webhook

Create a webhook automation in W&B to send HTTP requests to external services when specific events occur.

This page shows how to create a webhook [automation](/models/automations), which sends an HTTP request to an external service when a specific event occurs in W\&B. Use a webhook automation to integrate W\&B with external systems such as CI/CD pipelines, notification services, or custom tooling. To create a Slack automation, refer to [Create a Slack automation](/models/automations/create-automations/slack) instead.

At a high level, to create a webhook automation, you take these steps:

1. If necessary, [create a W\&B secret](/platform/secrets) for each sensitive string required by the automation, such as an access token, password, or SSH key. Secrets are defined in your **Team Settings**.
2. [Create a webhook](#create-a-webhook) to define the endpoint and authorization details and grant the integration access to any secrets it needs.
3. [Create the automation](#create-an-automation) to define the [event](/models/automations/automation-events) to watch for and the payload W\&B sends. Grant the automation access to any secrets it needs for the payload.

## Create a webhook

A team admin can add a webhook for the team. The webhook defines the endpoint W\&B sends requests to and any credentials required to authenticate with it.

<Note>
  If the webhook requires a Bearer token or its payload requires a sensitive string, [create a secret that contains it](/platform/secrets/#add-a-secret) before creating the webhook. You can configure at most one access token and one other secret for a webhook. The webhook's service determines your webhook's authentication and authorization requirements.
</Note>

1. Log in to W\&B and go to the **Team Settings** page.
2. In the **Webhooks** section, click **New webhook**.
3. Provide a name for the webhook.
4. Provide the endpoint URL of the webhook.
5. If the webhook requires a Bearer token, set **Access token** to the [secret](/platform/secrets) that contains it. When using the webhook automation, W\&B sets the `Authorization: Bearer` HTTP header to the access token, and you can access the token in the `${ACCESS_TOKEN}` [payload variable](#payload-variables). For more information about the structure of the `POST` request W\&B sends to the webhook service, see [Troubleshoot your webhook](#troubleshoot-your-webhook).
6. If the webhook requires a password or other sensitive string in its payload, set **Secret** to the secret that contains it. When you configure the automation that uses the webhook, you can access the secret as a [payload variable](#payload-variables) by prefixing its name with `$`.

   If the webhook's access token is stored in a secret, you must *also* complete the next step to specify the secret as the access token.
7. To verify that W\&B can connect and authenticate to the endpoint:

   1. Optionally, provide a payload to test. To refer to a secret the webhook has access to in the payload, prefix its name with `$`. W\&B uses this payload only for testing and doesn't save it. You configure an automation's payload when you [create the automation](#create-an-automation). For more information about where the secret and access token appear in the `POST` request, see [Troubleshoot your webhook](#troubleshoot-your-webhook).
   2. Click **Test**. W\&B attempts to connect to the webhook's endpoint using the credentials you configured. If you provided a payload, W\&B sends it.

   If the test doesn't succeed, verify the webhook's configuration and try again. If necessary, refer to [Troubleshoot your webhook](#troubleshoot-your-webhook).

<Frame>
  <img alt="Screenshot showing two webhooks in a Team" />
</Frame>

Now you can [create an automation](#create-an-automation) that uses the webhook.

## Create an automation

After you [configure a webhook](#create-a-webhook), create an automation that defines which W\&B event triggers the webhook and what payload to send. Select **Registry** or **Project** depending on the scope you want, then follow these steps to create an automation that triggers the webhook.

<Tabs>
  <Tab title="Registry">
    A Registry admin can create automations in that registry. Registry automations apply to all collections in the registry, including those added in the future.

    1. Log in to W\&B.

    2. Click the name of a registry to view its details.

    3. To create an automation scoped to the registry, click the **Automations** tab, then click **Create automation**.

    4. Choose the [event](/models/automations/automation-events/#registry-events) to watch for.

       Fill in any additional fields that appear. For example, if you select **An artifact alias is added**, you must specify the **Alias regex**.

       Click **Next step**.

    5. Select the team that owns the [webhook](#create-a-webhook).

    6. Set **Action type** to **Webhooks**. Then select the [webhook](#create-a-webhook) to use.

    7. If you configured an access token for the webhook, you can access the token in the `${ACCESS_TOKEN}` [payload variable](#payload-variables). If you configured a secret for the webhook, you can access it in the payload by prefixing its name with `$`. The webhook's service determines your webhook's requirements.

    8. Click **Next step**.

    9. Provide a name for the automation. Optionally, provide a description. Click **Create automation**.
  </Tab>

  <Tab title="Project">
    A W\&B admin can create automations in a project.

    1. Log in to W\&B and go to the project page.
    2. In the project sidebar, click **Automations**, then click **Create automation**.

       Or, from a line plot in the workspace, you can quickly create a [run metric automation](/models/automations/automation-events/#run-events) for the metric it shows. Hover over the panel, then click the bell icon at the top of the panel.

       <Frame>
         <img alt="Automation bell icon location" />
       </Frame>
    3. Choose the [event](/models/automations/automation-events/#project) to watch for, such as when an artifact alias is added or when a run metric meets a given threshold.

       1. Fill in any additional fields that appear, which depend upon the event. For example, if you select **An artifact alias is added**, you must specify the **Alias regex**.

          1. For automations triggered by a run, optionally specify one or more run filters.

             * **Filter to one user's runs**: Include only runs created by the specified user. Click the toggle to turn on the filter, then specify a username.
             * **Filter on run name**: Include only runs whose names match the given regular expression. Click the toggle to turn on the filter, then specify a regular expression.

          The automation applies to all collections in the project, including those added in the future.
       2. Click **Next step**.
    4. Select the team that owns the [webhook](#create-a-webhook).
    5. Set **Action type** to **Webhooks**. Then select the [webhook](#create-a-webhook) to use.
    6. If your webhook requires a payload, construct it and paste it into the **Payload** field. If you configured an access token for the webhook, you can access the token in the `${ACCESS_TOKEN}` [payload variable](#payload-variables). If you configured a secret for the webhook, you can access it in the payload by prefixing its name with `$`. The webhook's service determines your webhook's requirements.
    7. Click **Next step**.
    8. Provide a name for the automation. Optionally, provide a description. Click **Create automation**.
  </Tab>
</Tabs>

After you complete these steps, the automation is active and runs the webhook whenever the specified event occurs in the registry or project.

## View and manage automations

Review, edit, or delete an automation from the **Automations** tab.

<Tabs>
  <Tab title="Registry">
    Manage a registry's automations from the registry's **Automations** tab.

    * To view an automation's details, click its name.
    * To edit an automation, click its **action (<Icon icon="ellipsis" />)** menu, then click **Edit automation**.
    * To delete an automation, click its **action (<Icon icon="ellipsis" />)** menu, then click **Delete automation**. W\&B prompts you to confirm.
  </Tab>

  <Tab title="Project">
    A W\&B admin can view and manage a project's automations from the project's **Automations** tab.

    * To view an automation's details, click its name.
    * To edit an automation, click its **action (<Icon icon="ellipsis" />)** menu, then click **Edit automation**.
    * To delete an automation, click its **action (<Icon icon="ellipsis" />)** menu, then click **Delete automation**. W\&B prompts you to confirm.
  </Tab>
</Tabs>

## Payload reference

Use these sections to construct your webhook's payload. The following sections describe the variables available to your payload and show example payloads for common services. For details about testing your webhook and its payload, refer to [Troubleshoot your webhook](#troubleshoot-your-webhook).

### Payload variables

The following table describes the variables you can use to construct your webhook's payload.

| Variable                      | Details                                                                                                                                                                                                                               |
| ----------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `${project_name}`             | The name of the project that owns the mutation that triggered the action.                                                                                                                                                             |
| `${entity_name}`              | The name of the entity or team that owns the mutation that triggered the action.                                                                                                                                                      |
| `${event_type}`               | The type of event that triggered the action.                                                                                                                                                                                          |
| `${event_author}`             | The user that triggered the action.                                                                                                                                                                                                   |
| `${alias}`                    | Contains an artifact's alias if the automation is triggered by the **An artifact alias is added** event. For other automations, this variable is blank.                                                                               |
| `${tag}`                      | Contains an artifact's tags if the automation is triggered by the **An artifact tag is added** event. For other automations, this variable is blank.                                                                                  |
| `${artifact_collection_name}` | The name of the artifact collection that the artifact version is linked to.                                                                                                                                                           |
| `${artifact_metadata.<KEY>}`  | The value of an arbitrary top-level metadata key from the artifact version that triggered the action. Replace `<KEY>` with the name of a top-level metadata key. Only top-level metadata keys are available in the webhook's payload. |
| `${artifact_version}`         | The [`Wandb.Artifact`](/models/ref/python/experiments/artifact) representation of the artifact version that triggered the action.                                                                                                     |
| `${artifact_version_string}`  | The `string` representation of the artifact version that triggered the action.                                                                                                                                                        |
| `${ACCESS_TOKEN}`             | The value of the access token configured in the [webhook](#create-a-webhook), if you configured an access token. W\&B automatically passes it in the `Authorization: Bearer` HTTP header.                                             |
| `${SECRET_NAME}`              | If configured, the value of a secret configured in the [webhook](#create-a-webhook). Replace `SECRET_NAME` with the name of the secret.                                                                                               |

### Payload examples

The following examples show webhook payloads for common use cases. The examples demonstrate how to use [payload variables](#payload-variables).

<Tabs>
  <Tab title="GitHub repository dispatch">
    <Note>
      Verify that your access tokens have the required set of permissions to trigger your GitHub Actions workflow. For more information, see the [GitHub repository dispatch event documentation](https://docs.github.com/en/rest/repos/repos?#create-a-repository-dispatch-event).
    </Note>

    Send a repository dispatch from W\&B to trigger a GitHub action. For example, suppose you have a GitHub workflow file that accepts a repository dispatch as a trigger for the `on` key:

    ```yaml theme={null}
    on:
    repository_dispatch:
      types: BUILD_AND_DEPLOY
    ```

    The payload for the repository might look something like:

    ```json theme={null}
    {
      "event_type": "BUILD_AND_DEPLOY",
      "client_payload":
      {
        "event_author": "${event_author}",
        "artifact_version": "${artifact_version}",
        "artifact_version_string": "${artifact_version_string}",
        "artifact_collection_name": "${artifact_collection_name}",
        "project_name": "${project_name}",
        "entity_name": "${entity_name}"
        }
    }
    ```

    <Note>
      The `event_type` key in the webhook payload must match the `types` field in the GitHub workflow YAML file.
    </Note>

    The contents and positioning of rendered template strings depend on the event or model version you configured the automation for. `${event_type}` renders as either `LINK_ARTIFACT` or `ADD_ARTIFACT_ALIAS`. The following shows an example mapping:

    ```text theme={null}
    ${event_type} --> "LINK_ARTIFACT" or "ADD_ARTIFACT_ALIAS"
    ${event_author} --> "<wandb-user>"
    ${artifact_version} --> "wandb-artifact://_id/QXJ0aWZhY3Q6NTE3ODg5ODg3"
    ${artifact_version_string} --> "<entity>/model-registry/<registered_model_name>:<alias>"
    ${artifact_collection_name} --> "<registered_model_name>"
    ${project_name} --> "model-registry"
    ${entity_name} --> "<entity>"
    ```

    Use template strings to dynamically pass context from W\&B to GitHub Actions and other tools. If those tools can call Python scripts, they can consume the registered model artifacts through the [W\&B API](/models/artifacts/download-and-use-an-artifact).

    For more information, see the following resources:

    * For more information about repository dispatch, see the [official documentation on the GitHub Marketplace](https://github.com/marketplace/actions/repository-dispatch).
    * Watch the videos [Webhook Automations for Model Evaluation](https://www.youtube.com/watch?v=7j-Mtbo-E74\&ab_channel=Weights%26Biases) and [Webhook Automations for Model Deployment](https://www.youtube.com/watch?v=g5UiAFjM2nA\&ab_channel=Weights%26Biases), which guide you to create automations for model evaluation and deployment.
    * Review the W\&B report [Model CI/CD with W\&B](https://wandb.ai/wandb/wandb-model-cicd/reports/Model-CI-CD-with-W-B--Vmlldzo0OTcwNDQw), which illustrates how to use a GitHub Actions webhook automation for model CI.
    * For an example of model CI with a Modal Labs webhook, see the [wandb-modal-webhook GitHub repository](https://github.com/hamelsmu/wandb-modal-webhook).
  </Tab>

  <Tab title="Microsoft Teams notification">
    This example payload shows how to notify your Teams channel using a webhook:

    ```json theme={null}
    {
    "@type": "MessageCard",
    "@context": "http://schema.org/extensions",
    "summary": "New Notification",
    "sections": [
      {
        "activityTitle": "Notification from WANDB",
        "text": "This is an example message sent via Teams webhook.",
        "facts": [
          {
            "name": "Author",
            "value": "${event_author}"
          },
          {
            "name": "Event Type",
            "value": "${event_type}"
          }
        ],
        "markdown": true
      }
    ]
    }
    ```

    You can use template strings to inject W\&B data into your payload at the time of execution. See the [Teams example](#microsoft-teams-notification).
  </Tab>

  <Tab title="Slack notifications">
    <Note>
      This section is provided for historical purposes. If you use a webhook to integrate with Slack, W\&B recommends that you update your configuration to use the [Slack integration](/models/automations/create-automations/slack) instead.
    </Note>

    Set up your Slack app and add an incoming webhook integration by following the instructions in the [Slack API documentation](https://api.slack.com/messaging/webhooks). Ensure that you have the secret specified under `Bot User OAuth Token` as your W\&B webhook's access token.

    The following is an example payload:

    ```json theme={null}
    {
        "text": "New alert from WANDB!",
    "blocks": [
        {
                "type": "section",
            "text": {
                "type": "mrkdwn",
                "text": "Registry event: ${event_type}"
            }
        },
            {
                "type":"section",
                "text": {
                "type": "mrkdwn",
                "text": "New version: ${artifact_version_string}"
            }
            },
            {
            "type": "divider"
        },
            {
                "type": "section",
            "text": {
                "type": "mrkdwn",
                "text": "Author: ${event_author}"
            }
            }
        ]
    }
    ```
  </Tab>
</Tabs>

## Troubleshoot your webhook

If a webhook isn't working as expected, you can troubleshoot it interactively with the W\&B App UI or programmatically with a shell script. You can troubleshoot a webhook during creation or afterward.

For details about the format W\&B uses for the `POST` request, refer to the **Shell script** tab.

<Tabs>
  <Tab title="W&B App UI">
    A team admin can test a webhook interactively with the W\&B App UI.

    1. Go to your team page, then click **Settings**.
    2. Scroll to the **Webhooks** section.
    3. Click the **action (<Icon icon="ellipsis" />)** menu next to the name of your webhook.
    4. Select **Test**.
    5. From the UI panel that appears, paste your `POST` request into the field that appears.
       <Frame>
         <img alt="Demo of testing a webhook payload" />
       </Frame>
    6. Click **Test webhook**. Within the W\&B App UI, W\&B posts the response from your endpoint.
       <Frame>
         <img alt="Demo of testing a webhook" />
       </Frame>

    Watch the video [Testing Webhooks in W\&B](https://www.youtube.com/watch?v=bl44fDpMGJw\&ab_channel=Weights%26Biases) for a demonstration.
  </Tab>

  <Tab title="Shell script">
    This shell script shows one method to generate a `POST` request similar to the request W\&B sends to your webhook automation when it's triggered.

    Copy and paste the following code into a shell script to troubleshoot your webhook. Specify your own values for:

    * `ACCESS_TOKEN`
    * `SECRET`
    * `PAYLOAD`
    * `API_ENDPOINT`

    ```bash webhook_test.sh theme={null}
    #!/bin/bash

    # Your access token and secret
    ACCESS_TOKEN="your_api_key"
    SECRET="your_api_secret"

    # URL of your webhook endpoint
    API_ENDPOINT="https://your.webhook.endpoint/path"

    # The data you want to send (for example, in JSON format)
    PAYLOAD='{"key1": "value1", "key2": "value2"}'

    # Generate the HMAC signature. For security, W&B includes the
    # X-Wandb-Signature header computed from the payload and the shared secret
    # associated with the webhook using HMAC with SHA-256.
    SIGNATURE=$(echo -n "$PAYLOAD" | openssl dgst -sha256 -hmac "$SECRET" | awk '{print $2}')

    # Make the cURL request
    curl -X POST \
      -H "Content-Type: application/json" \
      -H "Authorization: Bearer $ACCESS_TOKEN" \
      -H "X-Wandb-Signature: $SIGNATURE" \
      -d "$PAYLOAD" "$API_ENDPOINT"
    ```
  </Tab>
</Tabs>


# Tutorial: Project run-failure alert automation
Source: https://docs.wandb.ai/models/automations/project-automation-tutorial

Build a run-failure alert that sends a Slack notification when a run in your project fails.

This tutorial walks you through building a **project** automation triggered by run status: when a run in your project transitions to **Failed**, W\&B sends a Slack notification. This automation lets your team learn about failed runs in real time, so you can investigate and remediate them instead of discovering failures later.

<br />

<br />

```mermaid theme={null}
%%{init: {'flowchart': {'rankSpacing': 200;}}}%%
flowchart LR
  Event[Run state change to Failed]
  Action[Slack notification]
  Event --> Action
```

<br />

<Tip>For guidance creating a Registry automation, see [Tutorial: Registry artifact alias automation](/models/automations/registry-automation-tutorial).</Tip>

## Prerequisites

* A W\&B project.
* A [Slack integration](/models/automations/create-automations/slack#add-a-slack-integration) configured in **Team Settings**.

## Create a project automation

Set up a project-scoped automation so that when a run in the project transitions to **Failed**, W\&B sends a Slack notification.

1. Open the project and click the **Automations** tab in the sidebar, then click **Create automation**.
2. Choose the event **Run state change**. Set the state to **Failed**. Optionally add a run name or user filter to limit which runs trigger the automation.
3. Click **Next step**. Set **Action type** to **Slack notification** and select the Slack channel.
4. Click **Next step**. Give the automation a name (for example, "Run failure alert") and an optional description, then click **Create automation**.

Your project now has an active automation that posts to the Slack channel you selected when a run fails. For more detail, see [Create a Slack automation](/models/automations/create-automations/slack#create-an-automation) (Project tab).

## Test the automation

To confirm the automation is configured correctly, trigger it with a deliberately failed run. Create a run and log it to the project, explicitly marking it as failed:

```python theme={null}
import wandb

with wandb.init(project="my-project") as run:
    run.log({"loss": 1.23})
    run.finish(exit_code=1)
```

Shortly after, you see a Slack message with the run link and status.

## Go further

For more information, see the following resources:

* [Automation events and scopes](/models/automations/automation-events) for all project and registry event types.
* [Create a Slack automation](/models/automations/create-automations/slack) and [Create a webhook automation](/models/automations/create-automations/webhook) for full UI and payload details.


# Tutorial: Registry artifact alias automation
Source: https://docs.wandb.ai/models/automations/registry-automation-tutorial

Build an automation that runs a webhook when a Registry artifact gets a specific alias like "production".

This tutorial walks you through building a **registry** automation triggered by artifact metadata: when an artifact in your registry gets a specific alias (for example, **production**), W\&B sends a `POST` request to your webhook. Use this pattern to notify downstream systems like deployment pipelines, paging services, or notification channels, whenever you promote a model to a known stage. This tutorial is intended for ML engineers and MLOps practitioners who manage model lifecycles in W\&B Registry.

<br />

<br />

```mermaid theme={null}
flowchart LR
  Event[Artifact alias added]
  Action[Webhook]
  Event --> Action
```

<br />

<Tip>For guidance creating a project automation, see [Tutorial: Project run-failure alert automation](/models/automations/project-automation-tutorial).</Tip>

## Prerequisites

* A [webhook](/models/automations/create-automations/webhook#create-a-webhook) configured in **Team Settings**.
* A W\&B [registry](/models/registry/create_registry) with at least one collection, or reuse an existing registry.

## Create a registry automation

Set up a registry-scoped automation so that when an artifact in any collection in the registry gets a specific alias (for example, **production**), W\&B sends a `POST` request to your webhook.

1. Open the registry and click the **Automations** tab, then click **Create automation**.
2. Choose the event **An artifact alias is added**. Enter an **Alias regex** that matches the alias you care about (for example, **production** or **staging**).
3. Click **Next step**. Set **Action type** to **Webhooks** and select your webhook. If the webhook expects a payload, paste a JSON body and use [payload variables](/models/automations/create-automations/webhook#payload-variables) such as `${artifact_collection_name}` and `${artifact_version_string}`.
4. Click **Next step**. Give the automation a name and optional description, then click **Create automation**.

For more information, see [Create a webhook automation](/models/automations/create-automations/webhook#create-an-automation) (Registry tab).

## Test the automation

To confirm that the automation fires end-to-end, trigger the configured event by adding the alias to an artifact version. Add the alias (for example, **production**) to an artifact version in the registry, using the W\&B App or the public API. For example:

```python theme={null}
import wandb

with wandb.init(project="my-project") as run:
    artifact = wandb.Artifact("my-model", type="model")
    # ... log files or metadata to artifact as needed ...
    run.log_artifact(artifact)
    run.wait()  # Ensure the artifact is logged before proceeding

# Add an alias to the latest version in the collection
api = wandb.Api()
collection = api.artifact_collection(name="my-model", type_name="model")
version = next(collection.versions())  # Get the latest version

version.aliases.append("production")
version.save()
print("Added alias 'production' to", version.name)

```

Within a short time, your webhook endpoint receives a `POST` with the payload you configured. You now have a working registry automation that fires whenever a matching alias is applied to an artifact in this registry.

## Go further

For more information, see the following resources:

* [Automation events and scopes](/models/automations/automation-events) for all project and registry event types.
* [Create a Slack automation](/models/automations/create-automations/slack) and [Create a webhook automation](/models/automations/create-automations/webhook) for full UI and payload details.


# Automation tutorial overview
Source: https://docs.wandb.ai/models/automations/tutorial

Learn to build a project run-failure alert or a registry alias automation.

W\&B Automations follow this pattern: when an **event** occurs and optional **conditions** are met, an **action** runs automatically. For example:

* When a run fails (event), notify a Slack channel (action).
  <br />
  <br />
  ```mermaid theme={null}
  %%{init: {'flowchart': {'rankSpacing': 200;}}}%%
  flowchart LR
    Event[Run state change to Failed]
    Action[Slack notification]
    Event --> Action
  ```
  <br />
* When the `production` alias is added to an artifact (event), call a webhook to trigger deployment (action).
  <br />
  <br />
  ```mermaid theme={null}
  flowchart LR
    Event[Artifact alias added]
    Action[Webhook]
    Event --> Action
  ```
  <br />

Events and available conditions differ for automations scoped to a [project](/models/automations/automation-events#project) or a [registry](/models/automations/automation-events#registry). See [Automation events and scopes](/models/automations/automation-events).

This page links to two end-to-end tutorials that show you how to create an automation in W\&B. Each tutorial walks through a different scenario, so you can follow the one that matches what you want to build. For detailed guidance, select a tutorial.

* **[Project automation tutorial](/models/automations/project-automation-tutorial)**: Alert when a run fails (Slack notification).
* **[Registry automation tutorial](/models/automations/registry-automation-tutorial)**: Trigger a webhook when an alias (for example, `production`) is added to an artifact.


# View an automation's history
Source: https://docs.wandb.ai/models/automations/view-automation-history

View the execution history of your W&B Automations to check status, triggering events, and action results.

<Note>
  Automation execution history is available on W\&B Multi-tenant Cloud, W\&B Dedicated Cloud, and W\&B Self-Managed v0.75.0 and later.
</Note>

This page describes how to view and understand the execution history of your W\&B [automations](/models/automations). Execution history shows what triggered an automation, what actions ran, and whether they succeeded or failed. Reviewing automation history helps you confirm that automations run as expected, troubleshoot failures, and audit which events triggered downstream actions.

Each executed automation generates a record that includes:

* **Execution timestamp**: When the automation triggered.
* **Triggering event**: The specific event that triggered the automation.
* **Status**: The execution's status. See [Execution status](#execution-status).
* **Action details**: Information about which action ran, such as notifying a Slack channel or running a webhook.
* **Result details**: Additional information, if any, about the final outcome of the automation, including the error for a failed execution.

Automation history is available for both registry-scoped and project-scoped automations. Select the **Registry** or **Project** tab for detailed instructions.

<Tabs>
  <Tab title="Registry">
    1. Click **Registry** in the project sidebar.
    2. Select your registry from the list.
    3. Navigate to the **Automations** tab, which lists the registry's automations. Click the **Last execution** timestamp to view the execution history details. Use the search bar to filter by automation name, and sort by the last triggered date to find recently executed automations.
    4. View the registry's automation executions in reverse chronological order in the **Automations history** tab, including the event, action, and status. Click an execution timestamp to view details about a particular execution.

    <Tip>If a collection has associated automation executions, the icon <img alt="Collection automation execution symbol, a circle with a right-pointing arrow" /> displays, along with the number of associated executions.</Tip>
  </Tab>

  <Tab title="Project">
    1. Navigate to your project.
    2. Click the **Automations** tab in the project sidebar, which lists the project's automations. Click the **Last execution** timestamp to view the execution history details. Use the search bar to filter by automation name, and sort by the last triggered date to find recently executed automations.
    3. In the **History** tab, view all executions of the project's automations in reverse chronological order. Each execution's metadata displays, including the event, action, and status. Click an execution timestamp to view details about a particular execution.
  </Tab>
</Tabs>

## Execution details

After locating an execution in the history, use the following information to interpret its status and outcome.

### Execution status

Each automation execution has one of the following statuses:

* **Finished**: The automation completed all actions successfully.
* **Failed**: The automation encountered an error and didn't complete.
* **Pending**: The automation is queued for execution.

### Execution metadata

Click any execution in the history to view the following details:

* **Event details**: The specific event that triggered the automation, including:
  * Event type (for example, "New artifact version", "Run completed").
  * Entity information, such as run ID or artifact name.
  * User who triggered the event (if applicable).
* **Action details**: Information about what the automation attempted:
  * Action type (Slack notification or webhook).
  * Target (Slack channel or webhook URL).
  * Payload sent (for webhooks).
* **Result information**:
  * Response status (for webhooks).
  * Any error messages or stack traces (for failed executions).

## Next steps

* [Create an automation](/models/automations/create-automations)
* [Automation events and scopes](/models/automations/automation-events)
* [Create a secret](/platform/secrets)


# Integrations overview
Source: https://docs.wandb.ai/models/integrations

Explore W&B integrations with ML frameworks, cloud platforms, and workflow orchestration tools

W\&B integrates with popular machine learning frameworks, cloud platforms, and workflow orchestration tools to help you track experiments, log metrics, and manage models seamlessly.

## Popular ML integrations:

<CardGroup>
  <Card title="PyTorch Lightning" href="/models/integrations/lightning">
    Integrate W\&B with your PyTorch Lightning code to add experiment tracking to your pipeline.
  </Card>

  <Card title="HuggingFace Transformers" href="/models/integrations/huggingface">
    Optimize HuggingFace Transformer models with W\&B for experiment tracking and model management.
  </Card>
</CardGroup>

<CardGroup>
  <Card title="Keras" href="/models/integrations/keras">
    Use W\&B and Keras for machine learning experiment tracking, dataset versioning, and project collaboration.
  </Card>

  <Card title="YOLOv5" href="/models/integrations/yolov5">
    Use the "You Only Look Once" (aka YOLOv5) real-time object detection framework and W\&B to track model metrics, inspect model outputs, and restart interrupted runs.
  </Card>
</CardGroup>

If the library you use is not supported natively, you can still integrate W\&B using the W\&B [Python SDK](/models/ref/python). See [Add W\&B to any library](/models/integrations/add-wandb-to-any-library) for best practices and implementation guidance.


# Hugging Face Accelerate
Source: https://docs.wandb.ai/models/integrations/accelerate

Training and inference at scale made simple, efficient and adaptable

Hugging Face Accelerate is a library that enables the same PyTorch code to run across any distributed configuration, to simplify model training and inference at scale.

Accelerate includes a W\&B Tracker which we show how to use below. You can also read more about [Accelerate Trackers in Hugging Face](https://huggingface.co/docs/accelerate/main/en/usage_guides/tracking).

## Start logging with Accelerate

To get started with Accelerate and W\&B you can follow the pseudocode below:

```python theme={null}
from accelerate import Accelerator

# Tell the Accelerator object to log with wandb
accelerator = Accelerator(log_with="wandb")

# Initialise your wandb run, passing wandb parameters and any config information
accelerator.init_trackers(
    project_name="my_project", 
    config={"dropout": 0.1, "learning_rate": 1e-2}
    init_kwargs={"wandb": {"entity": "my-wandb-team"}}
    )

...

# Log to wandb by calling accelerator.log(); step is optional
accelerator.log({"train_loss": 1.12, "valid_loss": 0.8}, step=global_step)


# Make sure that the wandb tracker finishes correctly
accelerator.end_training()
```

Explaining more, you need to:

1. Pass `log_with="wandb"` when initialising the Accelerator class
2. Call the [`init_trackers`](https://huggingface.co/docs/accelerate/main/en/package_reference/accelerator#accelerate.Accelerator.init_trackers) method and pass it:

* a project name via `project_name`
* any parameters you want to pass to [`wandb.init()`](/models/ref/python/functions/init) via a nested dict to `init_kwargs`
* any other experiment config information you want to log to your wandb run, via `config`

3. Use the `wandb.Run.log()` method to log to Weigths & Biases; the `step` argument is optional
4. Call `.end_training()` when finished training

## Access the W\&B tracker

To access the W\&B tracker, use the `Accelerator.get_tracker()` method. Pass in the string corresponding to a tracker’s `.name` attribute, which returns the tracker on the `main` process.

```python theme={null}
wandb_tracker = accelerator.get_tracker("wandb")

```

From there you can interact with wandb’s run object like normal:

```python theme={null}
wandb_tracker.log_artifact(some_artifact_to_log)
```

<Warning>
  Trackers built in Accelerate will automatically execute on the correct process, so if a tracker is only meant to be ran on the main process it will do so automatically.

  If you want to truly remove Accelerate’s wrapping entirely, you can achieve the same outcome with:

  ```python theme={null}
  wandb_tracker = accelerator.get_tracker("wandb", unwrap=True)
  with accelerator.on_main_process:
      wandb_tracker.log_artifact(some_artifact_to_log)
  ```
</Warning>

## Accelerate articles

Below is an Accelerate article you may enjoy

<details>
  <summary>HuggingFace Accelerate Super Charged With W\&B</summary>

  * In this article, we'll look at what HuggingFace Accelerate has to offer and how simple it is to perform distributed training and evaluation, while logging results to W\&B.

  Read the [Hugging Face Accelerate Super Charged with W\&B report](https://wandb.ai/gladiator/HF%20Accelerate%20+%20W\&B/reports/Hugging-Face-Accelerate-Super-Charged-with-Weights-Biases--VmlldzoyNzk3MDUx?utm_source=docs\&utm_medium=docs\&utm_campaign=accelerate-docs).
</details>

<br />

<br />


# Add W&B to a Python library
Source: https://docs.wandb.ai/models/integrations/add-wandb-to-any-library

Best practices for integrating Weights & Biases into your Python library for experiment tracking, system monitoring, and model management.

This guide explains how to integrate Weights & Biases (W\&B) into a Python library.

Follow these recommendations if you are integrating W\&B into a complex codebase—such as a training framework, SDK, or reusable library.

<Tip>
  If you are new to W\&B, review the core guides (for example, [Experiment Tracking](/models/track/)) before continuing.
</Tip>

Below we cover best tips and best practices when the codebase you are working on is more complicated than a single Python training script or Jupyter notebook.

## Decide how users install W\&B

Before you start, decide whether W\&B should be a required dependency or an optional feature of your library.

### Require W\&B as a dependency

If W\&B is central to your library’s functionality, add the W\&B Python SDK (`wandb`) to your dependencies:

```txt theme={null}
torch==1.8.0 
...
wandb==0.13.*
```

### Make W\&B optional on installation

If W\&B is an optional feature, allow your library to run without it installed.

You can either import `wandb` conditionally in Python or declare it as an optional dependency in `pyproject.toml`.

<Tabs>
  <Tab title="Python">
    Detect whether `wandb` is available and raise a clear error if a user enables W\&B features without installing it:

    ```python theme={null}
    try:
        import wandb
        _WANDB_AVAILABLE = True
    except ImportError:
        _WANDB_AVAILABLE = False
    ```
  </Tab>

  <Tab title="pyproject.toml">
    Declare `wandb` as an optional dependency to your `pyproject.toml` file:

    ```toml theme={null}
    [project]
    name = "my_awesome_lib"
    version = "0.1.0"
    dependencies = [
        "torch",
        "sklearn"
    ]

    [project.optional-dependencies]
    dev = [
        "wandb"
    ]
    ```
  </Tab>
</Tabs>

## Authenticate users

W\&B uses API keys to authenticate users and machines.

### Create an API key

An API key authenticates a client or machine to W\&B. You can generate an API key from your user profile.

<Note>
  For a more streamlined approach, create an API key by going directly to [User Settings](https://wandb.ai/settings). Copy the newly created API key immediately and save it in a secure location such as a password manager.
</Note>

1. Click your user profile icon in the upper right corner.
2. Select **User Settings**, then scroll to the **API Keys** section.

### Install and log in to W\&B

To install the `wandb` library locally and log in:

<Tabs>
  <Tab title="Command Line">
    1. Set the `WANDB_API_KEY` [environment variable](/models/track/environment-variables/) to your API key:

       ```bash theme={null}
       export WANDB_API_KEY=<your_api_key>
       ```

    2. Install the `wandb` library and log in:

       ```bash theme={null}
       pip install wandb

       wandb login
       ```
  </Tab>

  <Tab title="Python">
    1. Navigate to your terminal and install the Python SDK.
       ```bash theme={null}
       pip install wandb
       ```

    2. Log in to W\&B from your Python script or notebook. This will prompt you to enter
       your API key.
       ```python theme={null}
       import wandb
       wandb.login()
       ```
  </Tab>

  <Tab title="Python notebook">
    Copy and paste the following code snippet into a cell in your Jupyter notebook and run it. This will prompt you to enter your API key.

    ```notebook theme={null}
    !pip install wandb

    import wandb
    wandb.login()
    ```
  </Tab>
</Tabs>

## Start a run

A *run* represents a single unit of computation, such as a training experiment. Most libraries create one run per training job. For more information about runs, see [W\&B Runs](/models/runs/).

Initialize a run with [`wandb.init()`](/models/ref/python/functions/init) and specify a name for your project and your team entity (team name). If you do not specify a project, W\&B stores your run in a default project called "uncategorized".:

```python theme={null}
with wandb.init(project="<project_name>", entity="<entity>") as run:
    ...
```

W\&B recommends that you use a context manager to ensure that your run is properly closed, even if an error occurs. If you do not use a context manager, you must call `run.finish()` to close the run and log all the data to W\&B.

<Tip>
  **When to call `wandb.init()`**

  Call `wandb.init()` as early as possible. W\&B captures stdout, stderr, and error messages, which makes debugging easier.

  Wrap your entire training loop in a `wandb.init()` context manager to ensure that all relevant information is captured in the run. This includes any error messages, which can be crucial for debugging.
</Tip>

### Set `wandb` as an optional dependency

If you want to make `wandb` optional when your users use your library, you can either:

* Define a `wandb` flag such as:

<Tabs>
  <Tab title="Python">
    ```python theme={null}
    trainer = my_trainer(..., use_wandb=True)
    ```
  </Tab>

  <Tab title="Bash">
    ```bash theme={null}
    python train.py ... --use-wandb
    ```
  </Tab>
</Tabs>

* Or, set `wandb` to be `disabled` in `wandb.init()`:

<Tabs>
  <Tab title="Python">
    ```python theme={null}
    wandb.init(mode="disabled")
    ```
  </Tab>

  <Tab title="Bash">
    ```bash theme={null}
    export WANDB_MODE=disabled
    ```

    or

    ```bash theme={null}
    wandb disabled
    ```
  </Tab>
</Tabs>

* Or, set `wandb` to be offline - note this will still run `wandb`, it just won't try and communicate back to W\&B over the internet:

<Tabs>
  <Tab title="Environment Variable">
    ```bash theme={null}
    export WANDB_MODE=offline
    ```

    or

    ```python theme={null}
    os.environ['WANDB_MODE'] = 'offline'
    ```
  </Tab>

  <Tab title="Bash">
    ```bash theme={null}
    wandb offline
    ```
  </Tab>
</Tabs>

## Define a run config

Provide a configuration dictionary when you initialize your run to log hyperparameters and other metadata to W\&B.

Use the W\&B App to compare runs based on their config parameters and filter them in the Runs table. You can also use these parameters to group runs together in the W\&B App.

For example, in the following image, the batch size (bathch\_size) was defined as a config parameter and is visible(see first column) in the Runs table. This allows users to filter and compare runs based on their batch size:

<Frame>
  <img alt="W&B Runs table" />
</Frame>

Typical config parameters values include:

* Model name, version, architecture parameters, and hyperparameters.
* Dataset name, version, number of training or validation examples.
* Training parameters such as learning rate, batch size, and optimizer.

The following code snippet shows how to log a config:

```python theme={null}
config = {"batch_size": 32, ...}
with wandb.init(..., config=config) as run:
    ...
```

### Update the run config

If values are not available at initialization time, update the config later with `wandb.Run.config.update`. For example, you might want to add a model’s parameters after the model is instantiated:

```python theme={null}
with wandb.init(...) as run:
    model = MyModel(...)
    run.config.update({"model_parameters": 3500})
```

For details, see [Configure experiments](/models/track/config/).

## Log metrics and data

### Log metrics

Create a dictionary where the key value is the name of the metric. Pass this dictionary object to [`wandb.Run.log()`](/models/ref/python/experiments/run#method-run-log) to log it to W\&B:

```python theme={null}
NUM_EPOCHS = 10

for epoch in range(NUM_EPOCHS):
    for input, ground_truth in data: 
        prediction = model(input) 
        loss = loss_fn(prediction, ground_truth) 
        metrics = { "loss": loss } 
        run.log(metrics)
```

Use metric name prefixes to group related metrics in the W\&B App. Common prefixes include `train/` and `val/` for training and validation metrics, respectively, but you can use any prefix that makes sense for your use case.

This will create separate sections in your project's workspace for your training and validation metrics, or other metric types you'd like to separate:

```python theme={null}
with wandb.init(...) as run:
    metrics = {
        "train/loss": 0.4,
        "train/learning_rate": 0.4,
        "val/loss": 0.5, 
        "val/accuracy": 0.7
    }
    run.log(metrics)
```

<Frame>
  <img alt="W&B Workspace" />
</Frame>

See [`wandb.Run.log()`](/models/ref/python/experiments/run#method-run-log) for more details.

### Control the x-axis

If you perform multiple calls to `wandb.Run.log()` for the same training step, the wandb SDK increments an internal step counter for each call to `wandb.Run.log()`. This counter may not align with the training step in your training loop.

To avoid this situation, define your x-axis step explicitly with `wandb.Run.define_metric()`, one time, immediately after you call `wandb.init()`:

```python theme={null}
with wandb.init(...) as run:
    run.define_metric("*", step_metric="global_step")
```

The glob pattern, `*`, means that every metric will use `global_step` as the x-axis in your charts. If you only want certain metrics to be logged against `global_step`, you can specify them instead:

```python theme={null}
run.define_metric("train/loss", step_metric="global_step")
```

Now, log your metrics, your `step` metric, and your `global_step` each time you call `wandb.Run.log()`:

```python theme={null}
for step, (input, ground_truth) in enumerate(data):
    ...
    run.log({"global_step": step, "train/loss": 0.1})
    run.log({"global_step": step, "eval/loss": 0.2})
```

If you do not have access to the independent step variable, for example "global\_step" is not available during your validation loop, the previously logged value for "global\_step" is automatically used by wandb. In this case, ensure you log an initial value for the metric so it has been defined when it’s needed.

### Log media and structured data

In addition to scalars, you can log images, tables, text, audio, video, and more.

Some considerations when logging data include:

* How often should the metric be logged? Should it be optional?
* What type of data could be helpful in visualizing?
  * For images, you can log sample predictions, segmentation masks, etc., to see the evolution over time.
  * For text, you can log tables of sample predictions for later exploration.

See the [Log objects and media](/models/track/log) for examples.

## Support distributed training

For frameworks supporting distributed environments, you can adapt any of the following workflows:

* Log only from the main process (recommended).
* Log from every process and group runs using a shared `group` name.

See [Log Distributed Training Experiments](/models/track/log/distributed-training/) for more details.

## Track models and datasets with artifacts

Use [W\&B Artifacts](/models/artifacts/) to track and version models and datasets. Artifacts provide storage and versioning for machine learning assets, and they automatically track lineage to show how data and models are related.

<Frame>
  <img alt="Stored Datasets and Model Checkpoints in W&B" />
</Frame>

Consider the following when integrating artifacts into your library:

* Whether to log model checkpoints or datasets as artifacts (in case you want to make it optional).
* Artifact input references (for example, `entity/project/artifact`).
* Logging frequency of model checkpoints or datasets. For example, every epoch, every 500 steps, and so on.

### Log model checkpoints

Log model checkpoints to W\&B. A common approach is to log checkpoints as artifacts using the unique run ID generated by W\&B as part of the artifact name.

```python theme={null}
metadata = {"eval/accuracy": 0.8, "train/steps": 800} 

artifact = wandb.Artifact(
                name=f"model-{run.id}", 
                metadata=metadata, 
                type="model"
                ) 
artifact.add_dir("output_model") # local directory where the model weights are stored

aliases = ["best", "epoch_10"] 
run.log_artifact(artifact, aliases=aliases)
```

The previous code snippet demonstrates how to log a model checkpoint as an artifact and add metadata such as evaluation accuracy and training steps. The artifact is given a name that includes the unique run ID, and it is tagged with [custom aliases](/models/artifacts/create-a-custom-alias/) for easy reference.

### Log input artifacts

Log datasets or pretrained models used as inputs:

```python theme={null}
dataset = wandb.Artifact(name="flowers", type="dataset")
dataset.add_file("flowers.npy")
run.use_artifact(dataset)
```

The previous code snippet creates an artifact for a dataset called "flowers" and adds a file to it. The artifact is then associated with the current run using `run.use_artifact()`, which allows W\&B to track the lineage of the dataset used in the run.

### Download artifacts

Download previously logged artifacts from W\&B to use in your training or inference code.

If you have a run context, use [`wandb.Run.use_artifact()`](/models/ref/python/experiments/run) to reference an artifact in W\&B and then call [`wandb.Artifact.download()`](/models/ref/python/experiments/artifact) to download it to a local directory.

```python theme={null}
with wandb.init(...) as run:
    artifact = run.use_artifact("user/project/artifact:latest")
    local_path = artifact.download()
```

Use the [W\&B Public API](/models/ref/python/public-api/) to reference and download an artifact without initializing a run. This is useful in scenarios such as distributed environments or when performing inference, where you may not want to create a new run.

```python theme={null}
import wandb
artifact = wandb.Api().artifact("user/project/artifact:latest")
local_path = artifact.download()
```

See [Download and use artifacts](/models/artifacts/download-and-use-an-artifact/) for more information.

## Tune hyper-parameters

If your library supports hyperparameter tuning, you can integrate [W\&B Sweeps](/models/sweeps/) to manage and visualize experiments.


# Azure OpenAI Fine-Tuning
Source: https://docs.wandb.ai/models/integrations/azure-openai-fine-tuning

Fine-tune Azure OpenAI models with W&B experiment tracking to log metrics, hyperparameters, and training progress.

## Introduction

Fine-tuning GPT-3.5 or GPT-4 models on Microsoft Azure using W\&B tracks, analyzes, and improves model performance by automatically capturing metrics and facilitating systematic evaluation through W\&B's experiment tracking and evaluation tools.

<Frame>
  <img alt="Azure OpenAI fine-tuning metrics" />
</Frame>

## Prerequisites

* Set up Azure OpenAI service according to [official Azure documentation](https://wandb.me/aoai-wb-int).
* Configure a W\&B account with an API key.

## Workflow overview

### 1. Fine-tuning setup

* Prepare training data according to Azure OpenAI requirements.
* Configure the fine-tuning job in Azure OpenAI.
* W\&B automatically tracks the fine-tuning process, logging metrics and hyperparameters.

### 2. Experiment tracking

During fine-tuning, W\&B captures:

* Training and validation metrics
* Model hyperparameters
* Resource utilization
* Training artifacts

### 3. Model evaluation

After fine-tuning, use [W\&B Weave](https://weave-docs.wandb.ai) to:

* Evaluate model outputs against reference datasets
* Compare performance across different fine-tuning runs
* Analyze model behavior on specific test cases
* Make data-driven decisions for model selection

## Real-world example

* Explore the [medical note generation demo](https://wandb.me/aoai-ft-colab) to see how this integration facilitates:
  * Systematic tracking of fine-tuning experiments
  * Model evaluation using domain-specific metrics
* Go through an [interactive demo of fine-tuning a notebook](https://colab.research.google.com/github/wandb/examples/blob/master/colabs/azure/azure_gpt_medical_notes.ipynb)

## Additional resources

* [Azure OpenAI W\&B Integration Guide](https://wandb.me/aoai-wb-int)
* [Azure OpenAI Fine-tuning Documentation](https://learn.microsoft.com/azure/ai-services/openai/how-to/fine-tuning?tabs=turbo%2Cpython\&pivots=programming-language-python)


# Hugging Face Diffusers
Source: https://docs.wandb.ai/models/integrations/diffusers

Use W&B autolog with Hugging Face Diffusers to track prompts, generated media, configs, and pipeline architecture.

<ColabLink />

[Hugging Face Diffusers](https://huggingface.co/docs/diffusers/index) is the go-to library for state-of-the-art pre-trained diffusion models for generating images, audio, and even 3D structures of molecules. The W\&B integration adds rich, flexible experiment tracking, media visualization, pipeline architecture, and configuration management to interactive centralized dashboards without compromising that ease of use.

## Next-level logging in just two lines

Log all the prompts, negative prompts, generated media, and configs associated with your experiment by simply including 2 lines of code. Here are the 2 lines of code to begin logging:

```python theme={null}
# import the autolog function
from wandb.integration.diffusers import autolog

# call the autolog before calling the pipeline
autolog(init=dict(project="diffusers_logging"))
```

<Frame>
  <img alt="Experiment results logging" />
</Frame>

## Get started

1. Install `diffusers`, `transformers`, `accelerate`, and `wandb`.

   * Command line:

     ```shell theme={null}
     pip install --upgrade diffusers transformers accelerate wandb
     ```

   * Notebook:

     ```bash theme={null}
     !pip install --upgrade diffusers transformers accelerate wandb
     ```

2. Use `autolog` to initialize a W\&B Run and automatically track the inputs and the outputs from [all supported pipeline calls](https://github.com/wandb/wandb/blob/main/wandb/integration/diffusers/autologger.py#L12-L72).

   You can call the `autolog()` function with the `init` parameter, which accepts a dictionary of parameters required by [`wandb.init()`](/models/ref/python/functions/init).

   When you call `autolog()`, it initializes a W\&B Run and automatically tracks the inputs and the outputs from [all supported pipeline calls](https://github.com/wandb/wandb/blob/main/wandb/integration/diffusers/autologger.py#L12-L72).

   * Each pipeline call is tracked into its own [table](/models/tables/) in the workspace, and the configs associated with the pipeline call is appended to the list of workflows in the configs for that run.
   * The prompts, negative prompts, and the generated media are logged in a [`wandb.Table`](/models/tables/).
   * All other configs associated with the experiment including seed and the pipeline architecture are stored in the config section for the run.
   * The generated media for each pipeline call are also logged in [media panels](/models/track/log/media/) in the run.

   <Note>
     You can find a [list of supported pipeline calls](https://github.com/wandb/wandb/blob/main/wandb/integration/diffusers/autologger.py#L12-L72). In case, you want to request a new feature of this integration or report a bug associated with it, open an issue on the [W\&B GitHub issues page](https://github.com/wandb/wandb/issues).
   </Note>

## Examples

### Autologging

Here is a brief end-to-end example of the autolog in action:

<Tabs>
  <Tab title="Script">
    ```python theme={null}
    import torch
    from diffusers import DiffusionPipeline

    # import the autolog function
    from wandb.integration.diffusers import autolog

    # call the autolog before calling the pipeline
    autolog(init=dict(project="diffusers_logging"))

    # Initialize the diffusion pipeline
    pipeline = DiffusionPipeline.from_pretrained(
        "stabilityai/stable-diffusion-2-1", torch_dtype=torch.float16
    ).to("cuda")

    # Define the prompts, negative prompts, and seed.
    prompt = ["a photograph of an astronaut riding a horse", "a photograph of a dragon"]
    negative_prompt = ["ugly, deformed", "ugly, deformed"]
    generator = torch.Generator(device="cpu").manual_seed(10)

    # call the pipeline to generate the images
    images = pipeline(
        prompt,
        negative_prompt=negative_prompt,
        num_images_per_prompt=2,
        generator=generator,
    )
    ```
  </Tab>

  <Tab title="Notebook">
    ```python theme={null}
    import torch
    from diffusers import DiffusionPipeline

    import wandb

    # import the autolog function
    from wandb.integration.diffusers import autolog

    run = wandb.init()

    # call the autolog before calling the pipeline
    autolog(init=dict(project="diffusers_logging"))

    # Initialize the diffusion pipeline
    pipeline = DiffusionPipeline.from_pretrained(
        "stabilityai/stable-diffusion-2-1", torch_dtype=torch.float16
    ).to("cuda")

    # Define the prompts, negative prompts, and seed.
    prompt = ["a photograph of an astronaut riding a horse", "a photograph of a dragon"]
    negative_prompt = ["ugly, deformed", "ugly, deformed"]
    generator = torch.Generator(device="cpu").manual_seed(10)

    # call the pipeline to generate the images
    images = pipeline(
        prompt,
        negative_prompt=negative_prompt,
        num_images_per_prompt=2,
        generator=generator,
    )

    # Finish the experiment
    run.finish()
    ```
  </Tab>
</Tabs>

* The results of a single experiment:

  <Frame>
    <img alt="Experiment results logging" />
  </Frame>

* The results of multiple experiments:

  <Frame>
    <img alt="Experiment results logging" />
  </Frame>

* The config of an experiment:

  <Frame>
    <img alt="Experiment config logging" />
  </Frame>

<Note>
  You need to explicitly call [`wandb.Run.finish()`](/models/ref/python/functions/finish) when executing the code in IPython notebook environments after calling the pipeline. This is not necessary when executing python scripts.
</Note>

### Tracking multi-pipeline workflows

This section demonstrates the autolog with a typical [Stable Diffusion XL + Refiner](https://huggingface.co/docs/diffusers/using-diffusers/sdxl#base-to-refiner-model) workflow, in which the latents generated by the [`StableDiffusionXLPipeline`](https://huggingface.co/docs/diffusers/api/pipelines/stable_diffusion/stable_diffusion_xl) is refined by the corresponding refiner.

<Tabs>
  <Tab title="Python Script">
    ```python theme={null}
    import torch
    from diffusers import StableDiffusionXLImg2ImgPipeline, StableDiffusionXLPipeline
    from wandb.integration.diffusers import autolog

    # initialize the SDXL base pipeline
    base_pipeline = StableDiffusionXLPipeline.from_pretrained(
        "stabilityai/stable-diffusion-xl-base-1.0",
        torch_dtype=torch.float16,
        variant="fp16",
        use_safetensors=True,
    )
    base_pipeline.enable_model_cpu_offload()

    # initialize the SDXL refiner pipeline
    refiner_pipeline = StableDiffusionXLImg2ImgPipeline.from_pretrained(
        "stabilityai/stable-diffusion-xl-refiner-1.0",
        text_encoder_2=base_pipeline.text_encoder_2,
        vae=base_pipeline.vae,
        torch_dtype=torch.float16,
        use_safetensors=True,
        variant="fp16",
    )
    refiner_pipeline.enable_model_cpu_offload()

    prompt = "a photo of an astronaut riding a horse on mars"
    negative_prompt = "static, frame, painting, illustration, sd character, low quality, low resolution, greyscale, monochrome, nose, cropped, lowres, jpeg artifacts, deformed iris, deformed pupils, bad eyes, semi-realistic worst quality, bad lips, deformed mouth, deformed face, deformed fingers, deformed toes standing still, posing"

    # Make the experiment reproducible by controlling randomness.
    # The seed would be automatically logged to WandB.
    seed = 42
    generator_base = torch.Generator(device="cuda").manual_seed(seed)
    generator_refiner = torch.Generator(device="cuda").manual_seed(seed)

    # Call WandB Autolog for Diffusers. This would automatically log
    # the prompts, generated images, pipeline architecture and all
    # associated experiment configs to W&B, thus making your
    # image generation experiments easy to reproduce, share and analyze.
    autolog(init=dict(project="sdxl"))

    # Call the base pipeline to generate the latents
    image = base_pipeline(
        prompt=prompt,
        negative_prompt=negative_prompt,
        output_type="latent",
        generator=generator_base,
    ).images[0]

    # Call the refiner pipeline to generate the refined image
    image = refiner_pipeline(
        prompt=prompt,
        negative_prompt=negative_prompt,
        image=image[None, :],
        generator=generator_refiner,
    ).images[0]
    ```
  </Tab>

  <Tab title="Notebook">
    ```python theme={null}
    import torch
    from diffusers import StableDiffusionXLImg2ImgPipeline, StableDiffusionXLPipeline

    import wandb
    from wandb.integration.diffusers import autolog

    run = wandb.init()

    # initialize the SDXL base pipeline
    base_pipeline = StableDiffusionXLPipeline.from_pretrained(
        "stabilityai/stable-diffusion-xl-base-1.0",
        torch_dtype=torch.float16,
        variant="fp16",
        use_safetensors=True,
    )
    base_pipeline.enable_model_cpu_offload()

    # initialize the SDXL refiner pipeline
    refiner_pipeline = StableDiffusionXLImg2ImgPipeline.from_pretrained(
        "stabilityai/stable-diffusion-xl-refiner-1.0",
        text_encoder_2=base_pipeline.text_encoder_2,
        vae=base_pipeline.vae,
        torch_dtype=torch.float16,
        use_safetensors=True,
        variant="fp16",
    )
    refiner_pipeline.enable_model_cpu_offload()

    prompt = "a photo of an astronaut riding a horse on mars"
    negative_prompt = "static, frame, painting, illustration, sd character, low quality, low resolution, greyscale, monochrome, nose, cropped, lowres, jpeg artifacts, deformed iris, deformed pupils, bad eyes, semi-realistic worst quality, bad lips, deformed mouth, deformed face, deformed fingers, deformed toes standing still, posing"

    # Make the experiment reproducible by controlling randomness.
    # The seed would be automatically logged to WandB.
    seed = 42
    generator_base = torch.Generator(device="cuda").manual_seed(seed)
    generator_refiner = torch.Generator(device="cuda").manual_seed(seed)

    # Call WandB Autolog for Diffusers. This would automatically log
    # the prompts, generated images, pipeline architecture and all
    # associated experiment configs to W&B, thus making your
    # image generation experiments easy to reproduce, share and analyze.
    autolog(init=dict(project="sdxl"))

    # Call the base pipeline to generate the latents
    image = base_pipeline(
        prompt=prompt,
        negative_prompt=negative_prompt,
        output_type="latent",
        generator=generator_base,
    ).images[0]

    # Call the refiner pipeline to generate the refined image
    image = refiner_pipeline(
        prompt=prompt,
        negative_prompt=negative_prompt,
        image=image[None, :],
        generator=generator_refiner,
    ).images[0]

    # Finish the experiment
    run.finish()
    ```
  </Tab>
</Tabs>

* Example of a Stable Diffisuion XL + Refiner experiment:
  <Frame>
    <img alt="Stable Diffusion XL experiment tracking" />
  </Frame>

## More resources

* [A Guide to Prompt Engineering for Stable Diffusion](https://wandb.ai/geekyrakshit/diffusers-prompt-engineering/reports/A-Guide-to-Prompt-Engineering-for-Stable-Diffusion--Vmlldzo1NzY4NzQ3)
* [PIXART-α: A Diffusion Transformer Model for Text-to-Image Generation](https://wandb.ai/geekyrakshit/pixart-alpha/reports/PIXART-A-Diffusion-Transformer-Model-for-Text-to-Image-Generation--Vmlldzo2MTE1NzM3)


# Hugging Face
Source: https://docs.wandb.ai/models/integrations/huggingface

Visualize and track Hugging Face model performance with W&B, logging hyperparameters, metrics, and GPU utilization.

<ColabLink />

Visualize your [Hugging Face](https://github.com/huggingface/transformers) model's performance quickly with a seamless [W\&B](https://wandb.ai/site) integration.

Compare hyperparameters, output metrics, and system stats like GPU utilization across your models.

## Why should I use W\&B?

<Frame>
  <img alt="Benefits of using W&B" />
</Frame>

* **Unified dashboard**: Central repository for all your model metrics and predictions
* **Lightweight**: No code changes required to integrate with Hugging Face
* **Accessible**: Free for individuals and academic teams
* **Secure**: All projects are private by default
* **Trusted**: Used by machine learning teams at OpenAI, Toyota, Lyft and more

Think of W\&B like GitHub for machine learning models— save machine learning experiments to your private, hosted dashboard. Experiment quickly with the confidence that all the versions of your models are saved for you, no matter where you're running your scripts.

W\&B lightweight integrations works with any Python script, and all you need to do is sign up for a free W\&B account to start tracking and visualizing your models.

In the Hugging Face Transformers repo, we've instrumented the Trainer to automatically log training and evaluation metrics to W\&B at each logging step.

Here's an in depth look at how the integration works: [Hugging Face + W\&B Report](https://app.wandb.ai/jxmorris12/huggingface-demo/reports/Train-a-model-with-Hugging-Face-and-Weights-%26-Biases--VmlldzoxMDE2MTU).

## Install, import, and log in

Install the Hugging Face and W\&B libraries, and the GLUE dataset and training script for this tutorial.

* [Hugging Face Transformers](https://github.com/huggingface/transformers): Natural language models and datasets
* [W\&B](/): Experiment tracking and visualization
* [GLUE dataset](https://gluebenchmark.com/): A language understanding benchmark dataset
* [GLUE script](https://raw.githubusercontent.com/huggingface/transformers/refs/heads/main/examples/pytorch/text-classification/run_glue.py): Model training script for sequence classification

```notebook theme={null}
!pip install datasets wandb evaluate accelerate -qU
!wget https://raw.githubusercontent.com/huggingface/transformers/refs/heads/main/examples/pytorch/text-classification/run_glue.py
```

```notebook theme={null}
# the run_glue.py script requires transformers dev
!pip install -q git+https://github.com/huggingface/transformers
```

Before continuing, [sign up for a free account](https://app.wandb.ai/login?signup=true).

## Put in your API key

Once you've signed up, run the next cell and click on the link to get your API key and authenticate this notebook.

```python theme={null}
import wandb
wandb.login()
```

Optionally, we can set environment variables to customize W\&B logging. See the [Hugging Face integration guide](/models/integrations/huggingface/).

```python theme={null}
# Optional: log both gradients and parameters
%env WANDB_WATCH=all
```

## Train the model

Next, call the downloaded training script [run\_glue.py](https://huggingface.co/transformers/examples.html#glue) and see training automatically get tracked to the W\&B dashboard. This script fine-tunes BERT on the Microsoft Research Paraphrase Corpus— pairs of sentences with human annotations indicating whether they are semantically equivalent.

```python theme={null}
%env WANDB_PROJECT=huggingface-demo
%env TASK_NAME=MRPC

!python run_glue.py \
  --model_name_or_path bert-base-uncased \
  --task_name $TASK_NAME \
  --do_train \
  --do_eval \
  --max_seq_length 256 \
  --per_device_train_batch_size 32 \
  --learning_rate 2e-4 \
  --num_train_epochs 3 \
  --output_dir /tmp/$TASK_NAME/ \
  --overwrite_output_dir \
  --logging_steps 50
```

## Visualize results in dashboard

Click the link printed out above, or go to [wandb.ai](https://app.wandb.ai) to see your results stream in live. The link to see your run in the browser will appear after all the dependencies are loaded. Look for the following output: "**wandb**: View run at \[URL to your unique run]"

**Visualize Model Performance**
It's easy to look across dozens of experiments, zoom in on interesting findings, and visualize highly dimensional data.

<Frame>
  <img alt="Model metrics dashboard" />
</Frame>

**Compare Architectures**
Here's an example comparing [BERT vs DistilBERT](https://app.wandb.ai/jack-morris/david-vs-goliath/reports/Does-model-size-matter%3F-Comparing-BERT-and-DistilBERT-using-Sweeps--VmlldzoxMDUxNzU). It's easy to see how different architectures effect the evaluation accuracy throughout training with automatic line plot visualizations.

<Frame>
  <img alt="BERT vs DistilBERT comparison" />
</Frame>

## Track key information effortlessly by default

W\&B saves a new run for each experiment. Here's the information that gets saved by default:

* **Hyperparameters**: Settings for your model are saved in Config
* **Model Metrics**: Time series data of metrics streaming in are saved in Log
* **Terminal Logs**: Command line outputs are saved and available in a tab
* **System Metrics**: GPU and CPU utilization, memory, temperature etc.

## Learn more

* [Video walkthroughs on YouTube](http://wandb.me/youtube)


# Hugging Face Transformers
Source: https://docs.wandb.ai/models/integrations/huggingface_transformers

Use W&B with Hugging Face Transformers Trainer for experiment tracking, model checkpointing, and dataset versioning.

<ColabLink />

The [Hugging Face Transformers](https://huggingface.co/docs/transformers/index) library makes state-of-the-art NLP models like BERT and training techniques like mixed precision and gradient checkpointing easy to use. The [W\&B integration](https://huggingface.co/transformers/main_classes/callback.html#transformers.integrations.WandbCallback) adds rich, flexible experiment tracking and model versioning to interactive centralized dashboards without compromising that ease of use.

## Next-level logging in few lines

```python theme={null}
os.environ["WANDB_PROJECT"] = "<my-amazing-project>"  # name your W&B project
os.environ["WANDB_LOG_MODEL"] = "checkpoint"  # log all model checkpoints

from transformers import TrainingArguments, Trainer

args = TrainingArguments(..., report_to="wandb")  # turn on W&B logging
trainer = Trainer(..., args=args)
```

<Frame>
  <img alt="HuggingFace dashboard" />
</Frame>

<Note>
  If you'd rather dive straight into working code, check out this [Google Colab](https://wandb.me/hf).
</Note>

## Get started: track experiments

### Sign up and create an API key

An API key authenticates your machine to W\&B. You can generate an API key from your user profile.

<Note>
  For a more streamlined approach, create an API key by going directly to [User Settings](https://wandb.ai/settings). Copy the newly created API key immediately and save it in a secure location such as a password manager.
</Note>

1. Click your user profile icon in the upper right corner.
2. Select **User Settings**, then scroll to the **API Keys** section.

### Install the `wandb` library and log in

To install the `wandb` library locally and log in:

<Tabs>
  <Tab title="Command Line">
    1. Set the `WANDB_API_KEY` [environment variable](/models/track/environment-variables/) to your API key.

       ```bash theme={null}
       export WANDB_API_KEY=<your_api_key>
       ```

    2. Install the `wandb` library and log in.

       ```shell theme={null}
       pip install wandb

       wandb login
       ```
  </Tab>

  <Tab title="Python">
    ```bash theme={null}
    pip install wandb
    ```

    ```python theme={null}
    import wandb
    wandb.login()
    ```
  </Tab>

  <Tab title="Python notebook">
    ```notebook theme={null}
    !pip install wandb

    import wandb
    wandb.login()
    ```
  </Tab>
</Tabs>

If you are using W\&B for the first time you might want to check out our [quickstart](/models/quickstart/)

### Name the project

A W\&B Project is where all of the charts, data, and models logged from related runs are stored. Naming your project helps you organize your work and keep all the information about a single project in one place.

To add a run to a project simply set the `WANDB_PROJECT` environment variable to the name of your project. The `WandbCallback` will pick up this project name environment variable and use it when setting up your run.

<Tabs>
  <Tab title="Command Line">
    ```bash theme={null}
    WANDB_PROJECT=amazon_sentiment_analysis
    ```
  </Tab>

  <Tab title="Python">
    ```python theme={null}
    import os
    os.environ["WANDB_PROJECT"]="amazon_sentiment_analysis"
    ```
  </Tab>

  <Tab title="Python notebook">
    ```notebook theme={null}
    %env WANDB_PROJECT=amazon_sentiment_analysis
    ```
  </Tab>
</Tabs>

<Note>
  Make sure you set the project name *before* you initialize the `Trainer`.
</Note>

If a project name is not specified the project name defaults to `huggingface`.

### Log your training runs to W\&B

This is **the most important step** when defining your `Trainer` training arguments, either inside your code or from the command line, is to set `report_to` to `"wandb"` in order enable logging with W\&B.

The `logging_steps` argument in `TrainingArguments` will control how often training metrics are pushed to W\&B during training. You can also give a name to the training run in W\&B using the `run_name` argument.

That's it. Now your models will log losses, evaluation metrics, model topology, and gradients to W\&B while they train.

<Tabs>
  <Tab title="Command Line">
    ```bash theme={null}
    python run_glue.py \     # run your Python script
      --report_to wandb \    # enable logging to W&B
      --run_name bert-base-high-lr \   # name of the W&B run (optional)
      # other command line arguments here
    ```
  </Tab>

  <Tab title="Python">
    ```python theme={null}
    from transformers import TrainingArguments, Trainer

    args = TrainingArguments(
        # other args and kwargs here
        report_to="wandb",  # enable logging to W&B
        run_name="bert-base-high-lr",  # name of the W&B run (optional)
        logging_steps=1,  # how often to log to W&B
    )

    trainer = Trainer(
        # other args and kwargs here
        args=args,  # your training args
    )

    trainer.train()  # start training and logging to W&B
    ```
  </Tab>
</Tabs>

<Note>
  Using TensorFlow? Just swap the PyTorch `Trainer` for the TensorFlow `TFTrainer`.
</Note>

### Turn on model checkpointing

Using [Artifacts](/models/artifacts/), you can store up to 100GB of models and datasets for free and then use the W\&B [Registry](/models/registry/). Using Registry, you can register models to explore and evaluate them, prepare them for staging, or deploy them in your production environment.

To log your Hugging Face model checkpoints to Artifacts, set the `WANDB_LOG_MODEL` environment variable to *one* of:

* **`checkpoint`**: Upload a checkpoint every `args.save_steps` from the [`TrainingArguments`](https://huggingface.co/docs/transformers/main/en/main_classes/trainer#transformers.TrainingArguments).
* **`end`**: Upload the model at the end of training, if `load_best_model_at_end` is also set.
* **`false`**: Do not upload the model.

<Tabs>
  <Tab title="Command Line">
    ```bash theme={null}
    WANDB_LOG_MODEL="checkpoint"
    ```
  </Tab>

  <Tab title="Python">
    ```python theme={null}
    import os

    os.environ["WANDB_LOG_MODEL"] = "checkpoint"
    ```
  </Tab>

  <Tab title="Python notebook">
    ```notebook theme={null}
    %env WANDB_LOG_MODEL="checkpoint"
    ```
  </Tab>
</Tabs>

Any Transformers `Trainer` you initialize from now on will upload models to your W\&B project. The model checkpoints you log will be viewable through the [Artifacts](/models/artifacts/) UI, and include the full model lineage (see an example model checkpoint in the UI [here](https://wandb.ai/wandb/arttest/artifacts/model/iv3_trained/5334ab69740f9dda4fed/lineage?_gl=1*yyql5q*_ga*MTQxOTYyNzExOS4xNjg0NDYyNzk1*_ga_JH1SJHJQXJ*MTY5MjMwNzI2Mi4yNjkuMS4xNjkyMzA5NjM2LjM3LjAuMA..)).

<Note>
  By default, your model will be saved to W\&B Artifacts as `model-{run_id}` when `WANDB_LOG_MODEL` is set to `end` or `checkpoint-{run_id}` when `WANDB_LOG_MODEL` is set to `checkpoint`.
  However, If you pass a [`run_name`](https://huggingface.co/docs/transformers/main/en/main_classes/trainer#transformers.TrainingArguments.run_name) in your `TrainingArguments`, the model will be saved as `model-{run_name}` or `checkpoint-{run_name}`.
</Note>

#### W\&B Registry

Once you have logged your checkpoints to Artifacts, you can then register your best model checkpoints and centralize them across your team with [Registry](/models/registry/). Using Registry, you can organize your best models by task, manage the lifecycles of models, track and audit the entire ML lifecycle, and [automate](/models/automations/) downstream actions.

To link a model Artifact, refer to [Registry](/models/registry/).

### Visualise evaluation outputs during training

Visualing your model outputs during training or evaluation is often essential to really understand how your model is training.

By using the callbacks system in the Transformers Trainer, you can log additional helpful data to W\&B such as your models' text generation outputs or other predictions to W\&B Tables.

See the [Custom logging section](#custom-logging-log-and-view-evaluation-samples-during-training) below for a full guide on how to log evaluation outputs while training to log to a W\&B Table like this:

<Frame>
  <img alt="Shows a W&B Table with evaluation outputs" />
</Frame>

### Finish your W\&B Run (Notebook only)

If your training is encapsulated in a Python script, the W\&B run will end when your script finishes.

If you are using a Jupyter or Google Colab notebook, you'll need to tell us when you're done with training by calling `run.finish()`.

```python theme={null}
run = wandb.init()
trainer.train()  # start training and logging to W&B

# post-training analysis, testing, other logged code

run.finish()
```

### Visualize your results

Once you have logged your training results you can explore your results dynamically in the [W\&B Dashboard](/models/track/workspaces/). It's easy to compare across dozens of runs at once, zoom in on interesting findings, and coax insights out of complex data with flexible, interactive visualizations.

## Advanced features and FAQs

### How do I save the best model?

If you pass `TrainingArguments` with `load_best_model_at_end=True` to your `Trainer`, W\&B saves the best performing model checkpoint to Artifacts.

If you save your model checkpoints as Artifacts, you can promote them to the [Registry](/models/registry/). In Registry, you can:

* Organize your best model versions by ML task.
* Centralize models and share them with your team.
* Stage models for production or bookmark them for further evaluation.
* Trigger downstream CI/CD processes.

### How do I load a saved model?

If you saved your model to W\&B Artifacts with `WANDB_LOG_MODEL`, you can download your model weights for additional training or to run inference. You just load them back into the same Hugging Face architecture that you used before.

```python theme={null}
# Create a new run
with wandb.init(project="amazon_sentiment_analysis") as run:
    # Pass the name and version of Artifact
    my_model_name = "model-bert-base-high-lr:latest"
    my_model_artifact = run.use_artifact(my_model_name)

    # Download model weights to a folder and return the path
    model_dir = my_model_artifact.download()

    # Load your Hugging Face model from that folder
    #  using the same model class
    model = AutoModelForSequenceClassification.from_pretrained(
        model_dir, num_labels=num_labels
    )

    # Do additional training, or run inference
```

### How do I resume training from a checkpoint?

If you had set `WANDB_LOG_MODEL='checkpoint'` you can also resume training by you can using the `model_dir` as the `model_name_or_path` argument in your `TrainingArguments` and pass `resume_from_checkpoint=True` to `Trainer`.

```python theme={null}
last_run_id = "xxxxxxxx"  # fetch the run_id from your wandb workspace

# resume the wandb run from the run_id
with wandb.init(
    project=os.environ["WANDB_PROJECT"],
    id=last_run_id,
    resume="must",
) as run:
    # Connect an Artifact to the run
    my_checkpoint_name = f"checkpoint-{last_run_id}:latest"
    my_checkpoint_artifact = run.use_artifact(my_model_name)

    # Download checkpoint to a folder and return the path
    checkpoint_dir = my_checkpoint_artifact.download()

    # reinitialize your model and trainer
    model = AutoModelForSequenceClassification.from_pretrained(
        "<model_name>", num_labels=num_labels
    )
    # your awesome training arguments here.
    training_args = TrainingArguments()

    trainer = Trainer(model=model, args=training_args)

    # make sure use the checkpoint dir to resume training from the checkpoint
    trainer.train(resume_from_checkpoint=checkpoint_dir)
```

### How do I log and view evaluation samples during training

Logging to W\&B via the Transformers `Trainer` is taken care of by the [`WandbCallback`](https://huggingface.co/transformers/main_classes/callback.html#transformers.integrations.WandbCallback) in the Transformers library. If you need to customize your Hugging Face logging you can modify this callback by subclassing `WandbCallback` and adding additional functionality that leverages additional methods from the Trainer class.

Below is the general pattern to add this new callback to the HF Trainer, and further down is a code-complete example to log evaluation outputs to a W\&B Table:

```python theme={null}
# Instantiate the Trainer as normal
trainer = Trainer()

# Instantiate the new logging callback, passing it the Trainer object
evals_callback = WandbEvalsCallback(trainer, tokenizer, ...)

# Add the callback to the Trainer
trainer.add_callback(evals_callback)

# Begin Trainer training as normal
trainer.train()
```

#### View evaluation samples during training

The following section shows how to customize the `WandbCallback` to run model predictions and log evaluation samples to a W\&B Table during training. We will every `eval_steps` using the `on_evaluate` method of the Trainer callback.

Here, we wrote a `decode_predictions` function to decode the predictions and labels from the model output using the tokenizer.

Then, we create a pandas DataFrame from the predictions and labels and add an `epoch` column to the DataFrame.

Finally, we create a `wandb.Table` from the DataFrame and log it to wandb.
Additionally, we can control the frequency of logging by logging the predictions every `freq` epochs.

**Note**: Unlike the regular `WandbCallback` this custom callback needs to be added to the trainer **after** the `Trainer` is instantiated and not during initialization of the `Trainer`.
This is because the `Trainer` instance is passed to the callback during initialization.

```python theme={null}
from transformers.integrations import WandbCallback
import pandas as pd


def decode_predictions(tokenizer, predictions):
    labels = tokenizer.batch_decode(predictions.label_ids)
    logits = predictions.predictions.argmax(axis=-1)
    prediction_text = tokenizer.batch_decode(logits)
    return {"labels": labels, "predictions": prediction_text}


class WandbPredictionProgressCallback(WandbCallback):
    """Custom WandbCallback to log model predictions during training.

    This callback logs model predictions and labels to a wandb.Table at each
    logging step during training. It allows to visualize the
    model predictions as the training progresses.

    Attributes:
        trainer (Trainer): The Hugging Face Trainer instance.
        tokenizer (AutoTokenizer): The tokenizer associated with the model.
        sample_dataset (Dataset): A subset of the validation dataset
          for generating predictions.
        num_samples (int, optional): Number of samples to select from
          the validation dataset for generating predictions. Defaults to 100.
        freq (int, optional): Frequency of logging. Defaults to 2.
    """

    def __init__(self, trainer, tokenizer, val_dataset, num_samples=100, freq=2):
        """Initializes the WandbPredictionProgressCallback instance.

        Args:
            trainer (Trainer): The Hugging Face Trainer instance.
            tokenizer (AutoTokenizer): The tokenizer associated
              with the model.
            val_dataset (Dataset): The validation dataset.
            num_samples (int, optional): Number of samples to select from
              the validation dataset for generating predictions.
              Defaults to 100.
            freq (int, optional): Frequency of logging. Defaults to 2.
        """
        super().__init__()
        self.trainer = trainer
        self.tokenizer = tokenizer
        self.sample_dataset = val_dataset.select(range(num_samples))
        self.freq = freq

    def on_evaluate(self, args, state, control, **kwargs):
        super().on_evaluate(args, state, control, **kwargs)
        # control the frequency of logging by logging the predictions
        # every `freq` epochs
        if state.epoch % self.freq == 0:
            # generate predictions
            predictions = self.trainer.predict(self.sample_dataset)
            # decode predictions and labels
            predictions = decode_predictions(self.tokenizer, predictions)
            # add predictions to a wandb.Table
            predictions_df = pd.DataFrame(predictions)
            predictions_df["epoch"] = state.epoch
            records_table = self._wandb.Table(dataframe=predictions_df)
            # log the table to wandb
            self._wandb.log({"sample_predictions": records_table})


# First, instantiate the Trainer
trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=lm_datasets["train"],
    eval_dataset=lm_datasets["validation"],
)

# Instantiate the WandbPredictionProgressCallback
progress_callback = WandbPredictionProgressCallback(
    trainer=trainer,
    tokenizer=tokenizer,
    val_dataset=lm_dataset["validation"],
    num_samples=10,
    freq=2,
)

# Add the callback to the trainer
trainer.add_callback(progress_callback)
```

For a more detailed example please refer to this [colab](https://colab.research.google.com/github/wandb/examples/blob/master/colabs/huggingface/Custom_Progress_Callback.ipynb)

### What additional W\&B settings are available?

Further configuration of what is logged with `Trainer` is possible by setting environment variables. A full list of W\&B environment variables [can be found here](/platform/hosting/env-vars).

| Environment Variable | Usage                                                                                                                                                                                                                                                                                                                                                                          |
| -------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `WANDB_PROJECT`      | Give your project a name (`huggingface` by default)                                                                                                                                                                                                                                                                                                                            |
| `WANDB_LOG_MODEL`    | <p>Log the model checkpoint as a W\&B Artifact (`false` by default) </p><ul><li><code>false</code> (default): No model checkpointing </li><li><code>checkpoint</code>: A checkpoint will be uploaded every args.save\_steps (set in the Trainer's TrainingArguments). </li><li><code>end</code>: The final model checkpoint will be uploaded at the end of training.</li></ul> |
| `WANDB_WATCH`        | <p>Set whether you'd like to log your models gradients, parameters or neither</p><ul><li><code>false</code> (default): No gradient or parameter logging </li><li><code>gradients</code>: Log histograms of the gradients </li><li><code>all</code>: Log histograms of gradients and parameters</li></ul>                                                                       |
| `WANDB_DISABLED`     | Set to `true` to turn off logging entirely (`false` by default)                                                                                                                                                                                                                                                                                                                |
| `WANDB_QUIET`.       | Set to `true` to limit statements logged to standard output to critical statements only (`false` by default)                                                                                                                                                                                                                                                                   |
| `WANDB_SILENT`       | Set to `true` to silence the output printed by wandb (`false` by default)                                                                                                                                                                                                                                                                                                      |

<Tabs>
  <Tab title="Command Line">
    ```bash theme={null}
    WANDB_WATCH=all
    WANDB_SILENT=true
    ```
  </Tab>

  <Tab title="Notebook">
    ```notebook theme={null}
    %env WANDB_WATCH=all
    %env WANDB_SILENT=true
    ```
  </Tab>
</Tabs>

### How do I customize `wandb.init()`?

The `WandbCallback` that `Trainer` uses will call `wandb.init()` under the hood when `Trainer` is initialized. You can alternatively set up your runs manually by calling `wandb.init()` before the`Trainer` is initialized. This gives you full control over your W\&B run configuration.

An example of what you might want to pass to `init` is below. For `wandb.init()` details, see the [`wandb.init()` reference](/models/ref/python/functions/init).

```python theme={null}
wandb.init(
    project="amazon_sentiment_analysis",
    name="bert-base-high-lr",
    tags=["baseline", "high-lr"],
    group="bert",
)
```

## Additional resources

Below are 6 Transformers and W\&B related articles you might enjoy

<details>
  <summary>Hyperparameter Optimization for Hugging Face Transformers</summary>

  * Three strategies for hyperparameter optimization for Hugging Face Transformers are compared: Grid Search, Bayesian Optimization, and Population Based Training.
  * We use a standard uncased BERT model from Hugging Face transformers, and we want to fine-tune on the RTE dataset from the SuperGLUE benchmark
  * Results show that Population Based Training is the most effective approach to hyperparameter optimization of our Hugging Face transformer model.

  Read the [Hyperparameter Optimization for Hugging Face Transformers report](https://wandb.ai/amogkam/transformers/reports/Hyperparameter-Optimization-for-Hugging-Face-Transformers--VmlldzoyMTc2ODI).
</details>

<details>
  <summary>Hugging Tweets: Train a Model to Generate Tweets</summary>

  * In the article, the author demonstrates how to fine-tune a pre-trained GPT2 HuggingFace Transformer model on anyone's Tweets in five minutes.
  * The model uses the following pipeline: Downloading Tweets, Optimizing the Dataset, Initial Experiments, Comparing Losses Between Users, Fine-Tuning the Model.

  Read the full report [here](https://wandb.ai/wandb/huggingtweets/reports/HuggingTweets-Train-a-Model-to-Generate-Tweets--VmlldzoxMTY5MjI).
</details>

<details>
  <summary>Sentence Classification With Hugging Face BERT and WB</summary>

  * In this article, we'll build a sentence classifier leveraging the power of recent breakthroughs in Natural Language Processing, focusing on an application of transfer learning to NLP.
  * We'll be using The Corpus of Linguistic Acceptability (CoLA) dataset for single sentence classification, which is a set of sentences labeled as grammatically correct or incorrect that was first published in May 2018.
  * We'll use Google's BERT to create high performance models with minimal effort on a range of NLP tasks.

  Read the full report [here](https://wandb.ai/cayush/bert-finetuning/reports/Sentence-Classification-With-Huggingface-BERT-and-W-B--Vmlldzo4MDMwNA).
</details>

<details>
  <summary>A Step by Step Guide to Tracking Hugging Face Model Performance</summary>

  * We use W\&B and Hugging Face transformers to train DistilBERT, a Transformer that's 40% smaller than BERT but retains 97% of BERT's accuracy, on the GLUE benchmark
  * The GLUE benchmark is a collection of nine datasets and tasks for training NLP models

  Read the full report [here](https://wandb.ai/jxmorris12/huggingface-demo/reports/A-Step-by-Step-Guide-to-Tracking-HuggingFace-Model-Performance--VmlldzoxMDE2MTU).
</details>

<details>
  <summary>Examples of Early Stopping in HuggingFace</summary>

  * Fine-tuning a Hugging Face Transformer using Early Stopping regularization can be done natively in PyTorch or TensorFlow.
  * Using the EarlyStopping callback in TensorFlow is straightforward with the `tf.keras.callbacks.EarlyStopping`callback.
  * In PyTorch, there is not an off-the-shelf early stopping method, but there is a working early stopping hook available on GitHub Gist.

  Read the full report [here](https://wandb.ai/ayush-thakur/huggingface/reports/Early-Stopping-in-HuggingFace-Examples--Vmlldzo0MzE2MTM).
</details>

<details>
  <summary>How to Fine-Tune Hugging Face Transformers on a Custom Dataset</summary>

  We fine tune a DistilBERT transformer for sentiment analysis (binary classification) on a custom IMDB dataset.

  Read the full report [here](https://wandb.ai/ayush-thakur/huggingface/reports/How-to-Fine-Tune-HuggingFace-Transformers-on-a-Custom-Dataset--Vmlldzo0MzQ2MDc).
</details>

## Get help or request features

For any issues, questions, or feature requests for the Hugging Face W\&B integration, feel free to post in [this thread on the Hugging Face forums](https://discuss.huggingface.co/t/logging-experiment-tracking-with-w-b/498) or open an issue on the Hugging Face [Transformers GitHub repo](https://github.com/huggingface/transformers).


# PyTorch Ignite
Source: https://docs.wandb.ai/models/integrations/ignite

Integrate W&B with PyTorch Ignite to automatically log training metrics, model parameters, and experiment configs.

* See the resulting visualizations in this [example W\&B report →](https://app.wandb.ai/example-team/pytorch-ignite-example/reports/PyTorch-Ignite-with-W%26B--Vmlldzo0NzkwMg)
* Try running the code yourself in this [example hosted notebook →](https://colab.research.google.com/drive/15e-yGOvboTzXU4pe91Jg-Yr7sae3zBOJ#scrollTo=ztVifsYAmnRr)

Ignite supports W\&B handler to log metrics, model/optimizer parameters, gradients during training and validation. It can also be used to log model checkpoints to the W\&B cloud. This class is also a wrapper for the wandb module. This means that you can call any wandb function using this wrapper. See examples on how to save model parameters and gradients.

## Basic setup

```python theme={null}
from argparse import ArgumentParser
import wandb
import torch
from torch import nn
from torch.optim import SGD
from torch.utils.data import DataLoader
import torch.nn.functional as F
from torchvision.transforms import Compose, ToTensor, Normalize
from torchvision.datasets import MNIST

from ignite.engine import Events, create_supervised_trainer, create_supervised_evaluator
from ignite.metrics import Accuracy, Loss

from tqdm import tqdm


class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(1, 10, kernel_size=5)
        self.conv2 = nn.Conv2d(10, 20, kernel_size=5)
        self.conv2_drop = nn.Dropout2d()
        self.fc1 = nn.Linear(320, 50)
        self.fc2 = nn.Linear(50, 10)

    def forward(self, x):
        x = F.relu(F.max_pool2d(self.conv1(x), 2))
        x = F.relu(F.max_pool2d(self.conv2_drop(self.conv2(x)), 2))
        x = x.view(-1, 320)
        x = F.relu(self.fc1(x))
        x = F.dropout(x, training=self.training)
        x = self.fc2(x)
        return F.log_softmax(x, dim=-1)


def get_data_loaders(train_batch_size, val_batch_size):
    data_transform = Compose([ToTensor(), Normalize((0.1307,), (0.3081,))])

    train_loader = DataLoader(MNIST(download=True, root=".", transform=data_transform, train=True),
                              batch_size=train_batch_size, shuffle=True)

    val_loader = DataLoader(MNIST(download=False, root=".", transform=data_transform, train=False),
                            batch_size=val_batch_size, shuffle=False)
    return train_loader, val_loader
```

Using `WandBLogger` in ignite is a modular process. First, you create a `WandBLogger` object. Next, you attach it to a trainer or evaluator to automatically log the metrics. This example shows:

* Logs training loss, attached to the trainer object.
* Logs validation loss, attached to the evaluator.
* Logs optional Parameters, such as learning rate.
* Watches the model.

```python theme={null}
from ignite.contrib.handlers.wandb_logger import *
def run(train_batch_size, val_batch_size, epochs, lr, momentum, log_interval):
    train_loader, val_loader = get_data_loaders(train_batch_size, val_batch_size)
    model = Net()
    device = 'cpu'

    if torch.cuda.is_available():
        device = 'cuda'

    optimizer = SGD(model.parameters(), lr=lr, momentum=momentum)
    trainer = create_supervised_trainer(model, optimizer, F.nll_loss, device=device)
    evaluator = create_supervised_evaluator(model,
                                            metrics={'accuracy': Accuracy(),
                                                     'nll': Loss(F.nll_loss)},
                                            device=device)

    desc = "ITERATION - loss: {:.2f}"
    pbar = tqdm(
        initial=0, leave=False, total=len(train_loader),
        desc=desc.format(0)
    )
    #WandBlogger Object Creation
    wandb_logger = WandBLogger(
    project="pytorch-ignite-integration",
    name="cnn-mnist",
    config={"max_epochs": epochs,"batch_size":train_batch_size},
    tags=["pytorch-ignite", "mninst"]
    )

    wandb_logger.attach_output_handler(
    trainer,
    event_name=Events.ITERATION_COMPLETED,
    tag="training",
    output_transform=lambda loss: {"loss": loss}
    )

    wandb_logger.attach_output_handler(
    evaluator,
    event_name=Events.EPOCH_COMPLETED,
    tag="training",
    metric_names=["nll", "accuracy"],
    global_step_transform=lambda *_: trainer.state.iteration,
    )

    wandb_logger.attach_opt_params_handler(
    trainer,
    event_name=Events.ITERATION_STARTED,
    optimizer=optimizer,
    param_name='lr'  # optional
    )

    wandb_logger.watch(model)
```

You can optionally utilize ignite `EVENTS` to log the metrics directly to the terminal

```python theme={null}
    @trainer.on(Events.ITERATION_COMPLETED(every=log_interval))
    def log_training_loss(engine):
        pbar.desc = desc.format(engine.state.output)
        pbar.update(log_interval)

    @trainer.on(Events.EPOCH_COMPLETED)
    def log_training_results(engine):
        pbar.refresh()
        evaluator.run(train_loader)
        metrics = evaluator.state.metrics
        avg_accuracy = metrics['accuracy']
        avg_nll = metrics['nll']
        tqdm.write(
            "Training Results - Epoch: {}  Avg accuracy: {:.2f} Avg loss: {:.2f}"
            .format(engine.state.epoch, avg_accuracy, avg_nll)
        )

    @trainer.on(Events.EPOCH_COMPLETED)
    def log_validation_results(engine):
        evaluator.run(val_loader)
        metrics = evaluator.state.metrics
        avg_accuracy = metrics['accuracy']
        avg_nll = metrics['nll']
        tqdm.write(
            "Validation Results - Epoch: {}  Avg accuracy: {:.2f} Avg loss: {:.2f}"
            .format(engine.state.epoch, avg_accuracy, avg_nll))

        pbar.n = pbar.last_print_n = 0

    trainer.run(train_loader, max_epochs=epochs)
    pbar.close()


if __name__ == "__main__":
    parser = ArgumentParser()
    parser.add_argument('--batch_size', type=int, default=64,
                        help='input batch size for training (default: 64)')
    parser.add_argument('--val_batch_size', type=int, default=1000,
                        help='input batch size for validation (default: 1000)')
    parser.add_argument('--epochs', type=int, default=10,
                        help='number of epochs to train (default: 10)')
    parser.add_argument('--lr', type=float, default=0.01,
                        help='learning rate (default: 0.01)')
    parser.add_argument('--momentum', type=float, default=0.5,
                        help='SGD momentum (default: 0.5)')
    parser.add_argument('--log_interval', type=int, default=10,
                        help='how many batches to wait before logging training status')

    args = parser.parse_args()
    run(args.batch_size, args.val_batch_size, args.epochs, args.lr, args.momentum, args.log_interval)
```

This code generates these visualizations::

<Frame>
  <img alt="PyTorch Ignite training dashboard" />
</Frame>

<Frame>
  <img alt="PyTorch Ignite performance" />
</Frame>

<Frame>
  <img alt="PyTorch Ignite hyperparameter tuning results" />
</Frame>

<Frame>
  <img alt="PyTorch Ignite model comparison dashboard" />
</Frame>

Refer  to the [Ignite Docs](https://pytorch.org/ignite/contrib/handlers.html#module-ignite.contrib.handlers.wandb_logger) for more details.


# Keras
Source: https://docs.wandb.ai/models/integrations/keras

Use W&B Keras callbacks to track experiments, checkpoint models, and visualize predictions during training.

Use Keras callbacks to track experiments, log model checkpoints, and visualize model predictions. Keras callbacks are available in the `wandb.integration.keras` module with Pyhon SDK versions `0.13.4` and above.

W\&B Keras integration provides the following callbacks:

* **`WandbMetricsLogger`** : Use this callback for [Experiment Tracking](/models/track/). It logs your training and validation metrics along with system metrics to W\&B.
* **`WandbModelCheckpoint`** : Use this callback to log your model checkpoints to W\&B [Artifacts](/models/artifacts/).
* **`WandbEvalCallback`**: This base callback logs model predictions to W\&B [Tables](/models/tables/) for interactive visualization.

## Install and import Keras integration

Install the latest version of W\&B.

```bash theme={null}
pip install -U wandb
```

To use the Keras integration, import required classes from `wandb.integration.keras`:

```python theme={null}
import wandb
from wandb.integration.keras import WandbMetricsLogger, WandbModelCheckpoint, WandbEvalCallback
```

The following sections describe each callback in detail with code examples.

## Track experiments with `WandbMetricsLogger`

<ColabLink />

`wandb.integration.keras.WandbMetricsLogger()` automatically logs Keras' `logs` dictionary that callback methods such as `on_epoch_end`, `on_batch_end` etc, take as an argument.

The partial example below shows how to use `WandbMetricsLogger()` in a Keras workflow. First, compile the model with desired optimizer, loss function, and metrics. Then, initialize a W\&B run using `wandb.init()`. Finally, pass the `WandbMetricsLogger()` callback to `model.fit()`.

```python theme={null}
import wandb
from wandb.integration.keras import WandbMetricsLogger
import tensorflow as tf

model.compile(
    optimizer = "adam",
    loss = "categorical_crossentropy",
    metrics = ["accuracy", tf.keras.metrics.TopKCategoricalAccuracy(k=5, name='top@5_accuracy')]
)

# Initialize a new W&B Run
with wandb.init(config={"batch_size": 64}) as run:

    # Pass the WandbMetricsLogger to model.fit
    model.fit(
        X_train, y_train, validation_data=(X_test, y_test), callbacks=[WandbMetricsLogger()]
    )
```

The previous example logs training and validation metrics such as `loss`, `accuracy`, and `top@5_accuracy` to W\&B at the end of each epoch. It also logs:

### `WandbMetricsLogger` reference

| Parameter             | Description                                                                                                                                                                                                               |
| --------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `log_freq`            | (`epoch`, `batch`, or an `int`): if `epoch`, logs metrics at the end of each epoch. If `batch`, logs metrics at the end of each batch. If an `int`, logs metrics at the end of that many batches. Defaults to `epoch`.    |
| `initial_global_step` | (int): Use this argument to correctly log the learning rate when you resume training from some initial\_epoch, and a learning rate scheduler is used. This can be computed as step\_size \* initial\_step. Defaults to 0. |

## Checkpoint a model using `WandbModelCheckpoint`

<ColabLink />

Use `WandbModelCheckpoint` callback to save the Keras model (`SavedModel` format) or model weights periodically and uploads them to W\&B as a `wandb.Artifact` for model versioning.

This callback is subclassed from [`tf.keras.callbacks.ModelCheckpoint()`](https://www.tensorflow.org/api_docs/python/tf/keras/callbacks/ModelCheckpoint) ,thus the checkpointing logic is taken care of by the parent callback.

This callback saves:

* The model that has achieved best performance based on the monitor.
* The model at the end of every epoch regardless of the performance.
* The model at the end of the epoch or after a fixed number of training batches.
* Only model weights or the whole model.
* The model either in `SavedModel` format or in `.h5` format.

Use this callback in conjunction with `WandbMetricsLogger()`.

```python theme={null}
import wandb
from wandb.integration.keras import WandbMetricsLogger, WandbModelCheckpoint

# Initialize a new W&B Run
with wandb.init(config={"bs": 12}) as run:

    # Pass the WandbModelCheckpoint to model.fit
    model.fit(
        X_train,
        y_train,
        validation_data=(X_test, y_test),
        callbacks=[
            WandbMetricsLogger(),
            WandbModelCheckpoint("models"),
        ],
    )
```

### `WandbModelCheckpoint` reference

| Parameter                 | Description                                                                                                                                                                                                                                                                                                                                        |   |
| ------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | - |
| `filepath`                | (str): path to save the mode file.                                                                                                                                                                                                                                                                                                                 |   |
| `monitor`                 | (str): The metric name to monitor.                                                                                                                                                                                                                                                                                                                 |   |
| `verbose`                 | (int): Verbosity mode, 0 or 1. Mode 0 is silent, and mode 1 displays messages when the callback takes an action.                                                                                                                                                                                                                                   |   |
| `save_best_only`          | (Boolean): if `save_best_only=True`, it only saves the latest model or the model it considers the best, according to the defined by the `monitor` and `mode` attributes.                                                                                                                                                                           |   |
| `save_weights_only`       | (Boolean): if True, saves only the model's weights.                                                                                                                                                                                                                                                                                                |   |
| `mode`                    | (`auto`, `min`, or `max`): For `val_acc`, set it to `max`, for `val_loss`, set it to `min`, and so on                                                                                                                                                                                                                                              |   |
| `save_freq`               | ("epoch" or int): When using ‘epoch’, the callback saves the model after each epoch. When using an integer, the callback saves the model at end of this many batches. Note that when monitoring validation metrics such as `val_acc` or `val_loss`, `save_freq` must be set to "epoch" as those metrics are only available at the end of an epoch. |   |
| `options`                 | (str): Optional `tf.train.CheckpointOptions` object if `save_weights_only` is true or optional `tf.saved_model.SaveOptions` object if `save_weights_only` is false.                                                                                                                                                                                |   |
| `initial_value_threshold` | (float): Floating point initial "best" value of the metric to be monitored.                                                                                                                                                                                                                                                                        |   |

### Log checkpoints after N epochs

By default (`save_freq="epoch"`), the callback creates a checkpoint and uploads it as an artifact after each epoch. To create a checkpoint after a specific number of batches, set `save_freq` to an integer. To checkpoint after `N` epochs, compute the cardinality of the `train` dataloader and pass it to `save_freq`:

```python theme={null}
WandbModelCheckpoint(
    filepath="models/",
    save_freq=int((trainloader.cardinality()*N).numpy())
)
```

### Efficiently log checkpoints on a TPU architecture

While checkpointing on TPUs you might encounter `UnimplementedError: File system scheme '[local]' not implemented` error message. This happens because the model directory (`filepath`) must use a cloud storage bucket path (`gs://bucket-name/...`), and this bucket must be accessible from the TPU server. Instead, W\&B uses the local path for checkpointing which in turn is uploaded as an artifact.

```python theme={null}
checkpoint_options = tf.saved_model.SaveOptions(experimental_io_device="/job:localhost")

WandbModelCheckpoint(
    filepath="models/,
    options=checkpoint_options,
)
```

## Visualize model predictions using `WandbEvalCallback`

<ColabLink />

The `WandbEvalCallback()` is an abstract base class to build Keras callbacks primarily for model prediction and, secondarily, dataset visualization.

This abstract callback is agnostic with respect to the dataset and the task. To use this, inherit from this base `WandbEvalCallback()` callback class and implement the `add_ground_truth` and `add_model_prediction` methods.

The `WandbEvalCallback()` is a utility class that provides methods to:

* Create data and prediction `wandb.Table()` instances.
* Log data and prediction Tables as `wandb.Artifact()`.
* Log the data table `on_train_begin`.
* log the prediction table `on_epoch_end`.

The following example uses `WandbClfEvalCallback` for an image classification task. This example callback logs the validation data (`data_table`) to W\&B, performs inference, and logs the prediction (`pred_table`) to W\&B at the end of every epoch.

```python theme={null}
import wandb
from wandb.integration.keras import WandbMetricsLogger, WandbEvalCallback


# Implement your model prediction visualization callback
class WandbClfEvalCallback(WandbEvalCallback):
    def __init__(
        self, validation_data, data_table_columns, pred_table_columns, num_samples=100
    ):
        super().__init__(data_table_columns, pred_table_columns)

        self.x = validation_data[0]
        self.y = validation_data[1]

    def add_ground_truth(self, logs=None):
        for idx, (image, label) in enumerate(zip(self.x, self.y)):
            self.data_table.add_data(idx, wandb.Image(image), label)

    def add_model_predictions(self, epoch, logs=None):
        preds = self.model.predict(self.x, verbose=0)
        preds = tf.argmax(preds, axis=-1)

        table_idxs = self.data_table_ref.get_index()

        for idx in table_idxs:
            pred = preds[idx]
            self.pred_table.add_data(
                epoch,
                self.data_table_ref.data[idx][0],
                self.data_table_ref.data[idx][1],
                self.data_table_ref.data[idx][2],
                pred,
            )


# ...

# Initialize a new W&B Run
with wandb.init(config={"hyper": "parameter"}) as run:

    # Add the Callbacks to Model.fit
    model.fit(
        X_train,
        y_train,
        validation_data=(X_test, y_test),
        callbacks=[
            WandbMetricsLogger(),
            WandbClfEvalCallback(
                validation_data=(X_test, y_test),
                data_table_columns=["idx", "image", "label"],
                pred_table_columns=["epoch", "idx", "image", "label", "pred"],
            ),
        ],
    )
```

### `WandbEvalCallback` reference

| Parameter            | Description                                      |
| -------------------- | ------------------------------------------------ |
| `data_table_columns` | (list) List of column names for the `data_table` |
| `pred_table_columns` | (list) List of column names for the `pred_table` |

### Memory footprint details

We log the `data_table` to W\&B when the `on_train_begin` method is invoked. Once it's uploaded as a W\&B Artifact, we get a reference to this table which can be accessed using `data_table_ref` class variable. The `data_table_ref` is a 2D list that can be indexed like `self.data_table_ref[idx][n]`, where `idx` is the row number while `n` is the column number. Let's see the usage in the example below.

### Customize the callback

You can override the `on_train_begin` or `on_epoch_end` methods to have more fine-grained control. If you want to log the samples after `N` batches, you can implement `on_train_batch_end` method.

<Note>
  If you are implementing a callback for model prediction visualization by inheriting `WandbEvalCallback` and something needs to be clarified or fixed, let us know by opening an [issue](https://github.com/wandb/wandb/issues).
</Note>

## `WandbCallback` \[legacy]

Use the W\&B library `WandbCallback()` Class to automatically save all the metrics and the loss values tracked in `model.fit()`.

```python theme={null}
import wandb
from wandb.integration.keras import WandbCallback

with wandb.init(config={"hyper": "parameter"}) as run:

    # code to set up your model in Keras

    # Pass the callback to model.fit
    model.fit(
        X_train, y_train, validation_data=(X_test, y_test), callbacks=[WandbCallback()]
    )
```

You can watch the short video [Get Started with Keras and W\&B in Less Than a Minute](https://www.youtube.com/watch?ab_channel=Weights\&Biases\&v=4FjDIJ-vO_M).

For a more detailed video, watch [Integrate W\&B with Keras](https://www.youtube.com/watch?v=Bsudo7jbMow\&ab_channel=Weights%26Biases). You can review the [Colab Jupyter Notebook](https://colab.research.google.com/github/wandb/examples/blob/master/colabs/keras/Keras_pipeline_with_Weights_and_Biases.ipynb).

<Note>
  See our [example repo](https://github.com/wandb/examples) for scripts, including a [Fashion MNIST example](https://github.com/wandb/examples/blob/master/examples/keras/keras-cnn-fashion/train.py) and the [W\&B Dashboard](https://wandb.ai/wandb/keras-fashion-mnist/runs/5z1d85qs) it generates.
</Note>

The `WandbCallback` class supports a wide variety of logging configuration options: specifying a metric to monitor, tracking of weights and gradients, logging of predictions on training\_data and validation\_data, and more.

Check out the reference documentation for the `keras.WandbCallback` for full details.

The `WandbCallback`

* Automatically logs history data from any metrics collected by Keras: loss and anything passed into `keras_model.compile()`.
* Sets summary metrics for the run associated with the "best" training step, as defined by the `monitor` and `mode` attributes. This defaults to the epoch with the minimum `val_loss`. `WandbCallback` by default saves the model associated with the best `epoch`.
* Optionally logs gradient and parameter histogram.
* Optionally saves training and validation data for wandb to visualize.

### `WandbCallback` reference

| Arguments                  |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
| -------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `monitor`                  | (str) name of metric to monitor. Defaults to `val_loss`.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
| `mode`                     | (str) one of `{`auto`, `min`, `max`}`. `min` - save model when monitor is minimized `max` - save model when monitor is maximized `auto` - try to guess when to save the model (default).                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
| `save_model`               | True - save a model when monitor beats all previous epochs False - don't save models                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              |
| `save_graph`               | (boolean) if True save model graph to wandb (default to True).                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
| `save_weights_only`        | (boolean) if True, saves only the model's weights(`model.save_weights(filepath)`). Otherwise, saves the full model).                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              |
| `log_weights`              | (boolean) if True save histograms of the model's layer's weights.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |
| `log_gradients`            | (boolean) if True log histograms of the training gradients                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
| `training_data`            | (tuple) Same format `(X,y)` as passed to `model.fit`. This is needed for calculating gradients - this is mandatory if `log_gradients` is `True`.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| `validation_data`          | (tuple) Same format `(X,y)` as passed to `model.fit`. A set of data for wandb to visualize. If you set this field, every epoch, wandb makes a small number of predictions and saves the results for later visualization.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
| `generator`                | (generator) a generator that returns validation data for wandb to visualize. This generator should return tuples `(X,y)`. Either `validate_data` or generator should be set for wandb to visualize specific data examples.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
| `validation_steps`         | (int) if `validation_data` is a generator, how many steps to run the generator for the full validation set.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
| `labels`                   | (list) If you are visualizing your data with wandb this list of labels converts numeric output to understandable string if you are building a classifier with multiple classes. For a binary classifier, you can pass in a list of two labels \[`label for false`, `label for true`]. If `validate_data` and `generator` are both false, this does nothing.                                                                                                                                                                                                                                                                                                                                                       |
| `predictions`              | (int) the number of predictions to make for visualization each epoch, max is 100.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |
| `input_type`               | (string) type of the model input to help visualization. can be one of: (`image`, `images`, `segmentation_mask`).                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| `output_type`              | (string) type of the model output to help visualziation. can be one of: (`image`, `images`, `segmentation_mask`).                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |
| `log_evaluation`           | (boolean) if True, save a Table containing validation data and the model's predictions at each epoch. See `validation_indexes`, `validation_row_processor`, and `output_row_processor` for additional details.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
| `class_colors`             | (\[float, float, float]) if the input or output is a segmentation mask, an array containing an rgb tuple (range 0-1) for each class.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              |
| `log_batch_frequency`      | (integer) if None, callback logs every epoch. If set to integer, callback logs training metrics every `log_batch_frequency` batches.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              |
| `log_best_prefix`          | (string) if None, saves no extra summary metrics. If set to a string, prepends the monitored metric and epoch with the prefix and saves the results as summary metrics.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           |
| `validation_indexes`       | (\[wandb.data\_types.\_TableLinkMixin]) an ordered list of index keys to associate with each validation example. If `log_evaluation` is True and you provide `validation_indexes`, does not create a Table of validation data. Instead, associates each prediction with the row represented by the `TableLinkMixin`. To obtain a list of row keys, use `Table.get_index() `.                                                                                                                                                                                                                                                                                                                                      |
| `validation_row_processor` | (Callable) a function to apply to the validation data, commonly used to visualize the data. The function receives an `ndx` (int) and a `row` (dict). If your model has a single input, then `row["input"]` contains the input data for the row. Otherwise, it contains the names of the input slots. If your fit function takes a single target, then `row["target"]` contains the target data for the row. Otherwise, it contains the names of the output slots. For example, if your input data is a single array, to visualize the data as an Image, provide `lambda ndx, row: {"img": wandb.Image(row["input"])}` as the processor. Ignored if `log_evaluation` is False or `validation_indexes` are present. |
| `output_row_processor`     | (Callable) same as `validation_row_processor`, but applied to the model's output. `row["output"]` contains the results of the model output.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
| `infer_missing_processors` | (Boolean) Determines whether to infer `validation_row_processor` and `output_row_processor` if they are missing. Defaults to True. If you provide `labels`, W\&B attempts to infer classification-type processors where appropriate.                                                                                                                                                                                                                                                                                                                                                                                                                                                                              |
| `log_evaluation_frequency` | (int) Determines how often to log evaluation results. Defaults to `0` to log only at the end of training. Set to 1 to log every epoch, 2 to log every other epoch, and so on. Has no effect when `log_evaluation` is False.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |

## Frequently asked questions

### How do I use `Keras` multiprocessing with `wandb`?

When setting `use_multiprocessing=True`, this error may occur:

```python theme={null}
Error("You must call wandb.init() before wandb.config.batch_size")
```

To work around it:

1. In the `Sequence` class construction, add: `wandb.init(group='...')`.
2. In `main`, make sure you're using `if __name__ == "__main__":` and put the rest of your script logic inside it.


# PyTorch Lightning
Source: https://docs.wandb.ai/models/integrations/lightning

Use W&B with PyTorch Lightning through the built-in WandbLogger for experiment tracking and model checkpointing.

PyTorch Lightning provides a lightweight wrapper for organizing your PyTorch code and easily adding advanced features such as distributed training and 16-bit precision. W\&B provides a lightweight wrapper for logging your ML experiments. But you don't need to combine the two yourself: W\&B is incorporated directly into the PyTorch Lightning library via the [`WandbLogger`](https://lightning.ai/docs/pytorch/stable/api/lightning.pytorch.loggers.wandb.html#module-lightning.pytorch.loggers.wandb).

## Integrate with Lightning

<Tabs>
  <Tab title="PyTorch Logger">
    ```python theme={null}
    from lightning.pytorch.loggers import WandbLogger
    from lightning.pytorch import Trainer

    wandb_logger = WandbLogger(log_model="all")
    trainer = Trainer(logger=wandb_logger)
    ```

    <Note>
      **Using wandb.log():** The `WandbLogger` logs to W\&B using the Trainer's `global_step`. If you make additional calls to `wandb.log()` directly in your code, **do not** use the `step` argument in `wandb.log()`.

      Instead, log the Trainer's `global_step` like your other metrics:

      ```python theme={null}
      wandb.log({"accuracy":0.99, "trainer/global_step": step})
      ```
    </Note>
  </Tab>

  <Tab title="Fabric Logger">
    ```python theme={null}
    import lightning as L
    from wandb.integration.lightning.fabric import WandbLogger

    wandb_logger = WandbLogger(log_model="all")
    fabric = L.Fabric(loggers=[wandb_logger])
    fabric.launch()
    fabric.log_dict({"important_metric": important_metric})
    ```
  </Tab>
</Tabs>

<Frame>
  <img alt="Interactive dashboards" />
</Frame>

### Sign up and create an API key

An API key authenticates your machine to W\&B. You can generate an API key from your user profile.

<Note>
  For a more streamlined approach, create an API key by going directly to [User Settings](https://wandb.ai/settings). Copy the newly created API key immediately and save it in a secure location such as a password manager.
</Note>

1. Click your user profile icon in the upper right corner.
2. Select **User Settings**, then scroll to the **API Keys** section.

### Install the `wandb` library and log in

To install the `wandb` library locally and log in:

<Tabs>
  <Tab title="Command Line">
    1. Set the `WANDB_API_KEY` [environment variable](/models/track/environment-variables/) to your API key.

       ```bash theme={null}
       export WANDB_API_KEY=<your_api_key>
       ```

    2. Install the `wandb` library and log in.

       ```shell theme={null}
       pip install wandb

       wandb login
       ```
  </Tab>

  <Tab title="Python">
    ```bash theme={null}
    pip install wandb
    ```

    ```python theme={null}
    import wandb
    wandb.login()
    ```
  </Tab>

  <Tab title="Python notebook">
    ```notebook theme={null}
    !pip install wandb

    import wandb
    wandb.login()
    ```
  </Tab>
</Tabs>

## Use PyTorch Lightning's `WandbLogger`

PyTorch Lightning has multiple `WandbLogger` classes to log metrics and model weights, media, and more.

* [`PyTorch`](https://lightning.ai/docs/pytorch/stable/api/lightning.pytorch.loggers.wandb.html#module-lightning.pytorch.loggers.wandb)
* [`Fabric`](https://lightning.ai/docs/pytorch/stable/api/lightning.pytorch.loggers.wandb.html#module-lightning.pytorch.loggers.wandb)

To integrate with Lightning, instantiate the `WandbLogger` and pass it to Lightning's `Trainer` or `Fabric`.

<Tabs>
  <Tab title="PyTorch Logger">
    ```python theme={null}
    trainer = Trainer(logger=wandb_logger)
    ```
  </Tab>

  <Tab title="Fabric Logger">
    ```python theme={null}
    fabric = L.Fabric(loggers=[wandb_logger])
    fabric.launch()
    fabric.log_dict({
        "important_metric": important_metric
    })
    ```
  </Tab>
</Tabs>

### Common logger arguments

Below are some of the most used parameters in `WandbLogger`. Review the PyTorch Lightning documentation for details about all logger arguments.

* [`PyTorch`](https://lightning.ai/docs/pytorch/stable/api/lightning.pytorch.loggers.wandb.html#module-lightning.pytorch.loggers.wandb)
* [`Fabric`](https://lightning.ai/docs/pytorch/stable/api/lightning.pytorch.loggers.wandb.html#module-lightning.pytorch.loggers.wandb)

| Parameter   | Description                                                                   |
| ----------- | ----------------------------------------------------------------------------- |
| `project`   | Define what wandb Project to log to                                           |
| `name`      | Give a name to your wandb run                                                 |
| `log_model` | Log all models if `log_model="all"` or at end of training if `log_model=True` |
| `save_dir`  | Path where data is saved                                                      |

## Log your hyperparameters

<Tabs>
  <Tab title="PyTorch Logger">
    ```python theme={null}
    class LitModule(LightningModule):
        def __init__(self, *args, **kwarg):
            self.save_hyperparameters()
    ```
  </Tab>

  <Tab title="Fabric Logger">
    ```python theme={null}
    wandb_logger.log_hyperparams(
        {
            "hyperparameter_1": hyperparameter_1,
            "hyperparameter_2": hyperparameter_2,
        }
    )
    ```
  </Tab>
</Tabs>

## Log additional config parameters

```python theme={null}
# add one parameter
wandb_logger.experiment.config["key"] = value

# add multiple parameters
wandb_logger.experiment.config.update({key1: val1, key2: val2})

# use directly wandb module
wandb.config["key"] = value
wandb.config.update()
```

## Log gradients, parameter histogram and model topology

You can pass your model object to `wandblogger.watch()` to monitor your models's gradients and parameters as you train. See the PyTorch Lightning `WandbLogger` documentation

## Log metrics

<Tabs>
  <Tab title="PyTorch Logger">
    You can log your metrics to W\&B when using the `WandbLogger` by calling `self.log('my_metric_name', metric_vale)` within your `LightningModule`, such as in your `training_step` or `validation_step methods.`

    The code snippet below shows how to define your `LightningModule` to log your metrics and your `LightningModule` hyperparameters. This example uses the [`torchmetrics`](https://github.com/Lightning-AI/torchmetrics) library to calculate your metrics

    ```python theme={null}
    import torch
    from torch.nn import Linear, CrossEntropyLoss, functional as F
    from torch.optim import Adam
    from torchmetrics.functional import accuracy
    from lightning.pytorch import LightningModule


    class My_LitModule(LightningModule):
        def __init__(self, n_classes=10, n_layer_1=128, n_layer_2=256, lr=1e-3):
            """method used to define the model parameters"""
            super().__init__()

            # mnist images are (1, 28, 28) (channels, width, height)
            self.layer_1 = Linear(28 * 28, n_layer_1)
            self.layer_2 = Linear(n_layer_1, n_layer_2)
            self.layer_3 = Linear(n_layer_2, n_classes)

            self.loss = CrossEntropyLoss()
            self.lr = lr

            # save hyper-parameters to self.hparams (auto-logged by W&B)
            self.save_hyperparameters()

        def forward(self, x):
            """method used for inference input -> output"""

            # (b, 1, 28, 28) -> (b, 1*28*28)
            batch_size, channels, width, height = x.size()
            x = x.view(batch_size, -1)

            # let's do 3 x (linear + relu)
            x = F.relu(self.layer_1(x))
            x = F.relu(self.layer_2(x))
            x = self.layer_3(x)
            return x

        def training_step(self, batch, batch_idx):
            """needs to return a loss from a single batch"""
            _, loss, acc = self._get_preds_loss_accuracy(batch)

            # Log loss and metric
            self.log("train_loss", loss)
            self.log("train_accuracy", acc)
            return loss

        def validation_step(self, batch, batch_idx):
            """used for logging metrics"""
            preds, loss, acc = self._get_preds_loss_accuracy(batch)

            # Log loss and metric
            self.log("val_loss", loss)
            self.log("val_accuracy", acc)
            return preds

        def configure_optimizers(self):
            """defines model optimizer"""
            return Adam(self.parameters(), lr=self.lr)

        def _get_preds_loss_accuracy(self, batch):
            """convenience function since train/valid/test steps are similar"""
            x, y = batch
            logits = self(x)
            preds = torch.argmax(logits, dim=1)
            loss = self.loss(logits, y)
            acc = accuracy(preds, y)
            return preds, loss, acc
    ```
  </Tab>

  <Tab title="Fabric Logger">
    ```python theme={null}
    import lightning as L
    import torch
    import torchvision as tv
    from wandb.integration.lightning.fabric import WandbLogger
    import wandb

    fabric = L.Fabric(loggers=[wandb_logger])
    fabric.launch()

    model = tv.models.resnet18()
    optimizer = torch.optim.SGD(model.parameters(), lr=lr)
    model, optimizer = fabric.setup(model, optimizer)

    train_dataloader = fabric.setup_dataloaders(
        torch.utils.data.DataLoader(train_dataset, batch_size=batch_size)
    )

    model.train()
    for epoch in range(num_epochs):
        for batch in train_dataloader:
            optimizer.zero_grad()
            loss = model(batch)
            loss.backward()
            optimizer.step()
            fabric.log_dict({"loss": loss})
    ```
  </Tab>
</Tabs>

## Log the min/max of a metric

Using wandb's [`define_metric`](/models/ref/python/experiments/run#define_metric) function you can define whether you'd like your W\&B summary metric to display the min, max, mean or best value for that metric. If `define`\_`metric` \_ isn't used, then the last value logged with appear in your summary metrics. See the `define_metric` [reference docs here](/models/ref/python/experiments/run#define_metric) and the [guide here](/models/track/log/customize-logging-axes/) for more.

To tell W\&B to keep track of the max validation accuracy in the W\&B summary metric, call `wandb.define_metric()` only once, at the beginning of training:

<Tabs>
  <Tab title="PyTorch Logger">
    ```python theme={null}
    class My_LitModule(LightningModule):
        ...

        def validation_step(self, batch, batch_idx):
            if trainer.global_step == 0:
                wandb.define_metric("val_accuracy", summary="max")

            preds, loss, acc = self._get_preds_loss_accuracy(batch)

            # Log loss and metric
            self.log("val_loss", loss)
            self.log("val_accuracy", acc)
            return preds
    ```
  </Tab>

  <Tab title="Fabric Logger">
    ```python theme={null}
    wandb.define_metric("val_accuracy", summary="max")
    fabric = L.Fabric(loggers=[wandb_logger])
    fabric.launch()
    fabric.log_dict({"val_accuracy": val_accuracy})
    ```
  </Tab>
</Tabs>

## Checkpoint a model

To save model checkpoints as W\&B [Artifacts](/models/artifacts/),
use the Lightning [`ModelCheckpoint`](https://lightning.ai/docs/pytorch/stable/api/lightning.pytorch.callbacks.ModelCheckpoint.html) callback and set the `log_model` argument in the `WandbLogger`.

<Tabs>
  <Tab title="PyTorch Logger">
    ```python theme={null}
    trainer = Trainer(logger=wandb_logger, callbacks=[checkpoint_callback])
    ```
  </Tab>

  <Tab title="Fabric Logger">
    ```python theme={null}
    fabric = L.Fabric(loggers=[wandb_logger], callbacks=[checkpoint_callback])
    ```
  </Tab>
</Tabs>

The *latest* and *best* aliases are automatically set to easily retrieve a model checkpoint from a W\&B [Artifact](/models/artifacts/):

```python theme={null}
# reference can be retrieved in artifacts panel
# "VERSION" can be a version (ex: "v2") or an alias ("latest or "best")
checkpoint_reference = "USER/PROJECT/MODEL-RUN_ID:VERSION"
```

<Tabs>
  <Tab title="Via Logger">
    ```python theme={null}
    # download checkpoint locally (if not already cached)
    wandb_logger.download_artifact(checkpoint_reference, artifact_type="model")
    ```
  </Tab>

  <Tab title="Via wandb">
    ```python theme={null}
    # download checkpoint locally (if not already cached)
    run = wandb.init(project="MNIST")
    artifact = run.use_artifact(checkpoint_reference, type="model")
    artifact_dir = artifact.download()
    ```
  </Tab>
</Tabs>

<Tabs>
  <Tab title="PyTorch Logger">
    ```python theme={null}
    # load checkpoint
    model = LitModule.load_from_checkpoint(Path(artifact_dir) / "model.ckpt")
    ```
  </Tab>

  <Tab title="Fabric Logger">
    ```python theme={null}
    # Request the raw checkpoint
    full_checkpoint = fabric.load(Path(artifact_dir) / "model.ckpt")

    model.load_state_dict(full_checkpoint["model"])
    optimizer.load_state_dict(full_checkpoint["optimizer"])
    ```
  </Tab>
</Tabs>

The model checkpoints you log are viewable through the [W\&B Artifacts](/models/artifacts/) UI, and include the full model lineage (see an example model checkpoint in the UI [here](https://wandb.ai/wandb/arttest/artifacts/model/iv3_trained/5334ab69740f9dda4fed/lineage?_gl=1*yyql5q*_ga*MTQxOTYyNzExOS4xNjg0NDYyNzk1*_ga_JH1SJHJQXJ*MTY5MjMwNzI2Mi4yNjkuMS4xNjkyMzA5NjM2LjM3LjAuMA..)).

To bookmark your best model checkpoints and centralize them across your team, you can link them to the [W\&B Model Registry](/models).

Here you can organize your best models by task, manage model lifecycle, facilitate easy tracking and auditing throughout the ML lifecycle, and [automate](/models/automations/) downstream actions with webhooks or jobs.

## Log images, text, and more

The `WandbLogger` has `log_image`, `log_text` and `log_table` methods for logging media.

You can also directly call `wandb.log()` or `trainer.logger.experiment.log()` to log other media types such as Audio, Molecules, Point Clouds, 3D Objects and more.

<Tabs>
  <Tab title="Log Images">
    ```python theme={null}
    # using tensors, numpy arrays or PIL images
    wandb_logger.log_image(key="samples", images=[img1, img2])

    # adding captions
    wandb_logger.log_image(key="samples", images=[img1, img2], caption=["tree", "person"])

    # using file path
    wandb_logger.log_image(key="samples", images=["img_1.jpg", "img_2.jpg"])

    # using .log in the trainer
    trainer.logger.experiment.log(
        {"samples": [wandb.Image(img, caption=caption) for (img, caption) in my_images]},
        step=current_trainer_global_step,
    )
    ```
  </Tab>

  <Tab title="Log Text">
    ```python theme={null}
    # data should be a list of lists
    columns = ["input", "label", "prediction"]
    my_data = [["cheese", "english", "english"], ["fromage", "french", "spanish"]]

    # using columns and data
    wandb_logger.log_text(key="my_samples", columns=columns, data=my_data)

    # using a pandas DataFrame
    wandb_logger.log_text(key="my_samples", dataframe=my_dataframe)
    ```
  </Tab>

  <Tab title="Log Tables">
    ```python theme={null}
    # log a W&B Table that has a text caption, an image and audio
    columns = ["caption", "image", "sound"]

    # data should be a list of lists
    my_data = [
        ["cheese", wandb.Image(img_1), wandb.Audio(snd_1)],
        ["wine", wandb.Image(img_2), wandb.Audio(snd_2)],
    ]

    # log the Table
    wandb_logger.log_table(key="my_samples", columns=columns, data=data)
    ```
  </Tab>
</Tabs>

You can use Lightning's Callbacks system to control when you log to W\&B via the `WandbLogger`, in this example we log a sample of our validation images and predictions:

```python theme={null}
import torch
import wandb
import lightning.pytorch as pl
from lightning.pytorch.loggers import WandbLogger

# or
# from wandb.integration.lightning.fabric import WandbLogger


class LogPredictionSamplesCallback(Callback):
    def on_validation_batch_end(
        self, trainer, pl_module, outputs, batch, batch_idx, dataloader_idx
    ):
        """Called when the validation batch ends."""

        # `outputs` comes from `LightningModule.validation_step`
        # which corresponds to our model predictions in this case

        # Let's log 20 sample image predictions from the first batch
        if batch_idx == 0:
            n = 20
            x, y = batch
            images = [img for img in x[:n]]
            captions = [
                f"Ground Truth: {y_i} - Prediction: {y_pred}"
                for y_i, y_pred in zip(y[:n], outputs[:n])
            ]

            # Option 1: log images with `WandbLogger.log_image`
            wandb_logger.log_image(key="sample_images", images=images, caption=captions)

            # Option 2: log images and predictions as a W&B Table
            columns = ["image", "ground truth", "prediction"]
            data = [
                [wandb.Image(x_i), y_i, y_pred] or x_i,
                y_i,
                y_pred in list(zip(x[:n], y[:n], outputs[:n])),
            ]
            wandb_logger.log_table(key="sample_table", columns=columns, data=data)


trainer = pl.Trainer(callbacks=[LogPredictionSamplesCallback()])
```

## Use multiple GPUs with Lightning and W\&B

PyTorch Lightning has Multi-GPU support through their DDP Interface. However, PyTorch Lightning's design requires you to be careful about how you instantiate our GPUs.

Lightning assumes that each GPU (or Rank) in your training loop must be instantiated in exactly the same way - with the same initial conditions. However, only rank 0 process gets access to the `wandb.run` object, and for non-zero rank processes: `wandb.run = None`. This could cause your non-zero processes to fail. Such a situation can put you in a **deadlock** because rank 0 process will wait for the non-zero rank processes to join, which have already crashed.

For this reason, be careful about how we set up your training code. The recommended way to set it up would be to have your code be independent of the `wandb.run` object.

```python theme={null}
class MNISTClassifier(pl.LightningModule):
    def __init__(self):
        super(MNISTClassifier, self).__init__()

        self.model = nn.Sequential(
            nn.Flatten(),
            nn.Linear(28 * 28, 128),
            nn.ReLU(),
            nn.Linear(128, 10),
        )

        self.loss = nn.CrossEntropyLoss()

    def forward(self, x):
        return self.model(x)

    def training_step(self, batch, batch_idx):
        x, y = batch
        y_hat = self.forward(x)
        loss = self.loss(y_hat, y)

        self.log("train/loss", loss)
        return {"train_loss": loss}

    def validation_step(self, batch, batch_idx):
        x, y = batch
        y_hat = self.forward(x)
        loss = self.loss(y_hat, y)

        self.log("val/loss", loss)
        return {"val_loss": loss}

    def configure_optimizers(self):
        return torch.optim.Adam(self.parameters(), lr=0.001)


def main():
    # Setting all the random seeds to the same value.
    # This is important in a distributed training setting.
    # Each rank will get its own set of initial weights.
    # If they don't match up, the gradients will not match either,
    # leading to training that may not converge.
    pl.seed_everything(1)

    train_loader = DataLoader(train_dataset, batch_size=64, shuffle=True, num_workers=4)
    val_loader = DataLoader(val_dataset, batch_size=64, shuffle=False, num_workers=4)

    model = MNISTClassifier()
    wandb_logger = WandbLogger(project="<project_name>")
    callbacks = [
        ModelCheckpoint(
            dirpath="checkpoints",
            every_n_train_steps=100,
        ),
    ]
    trainer = pl.Trainer(
        max_epochs=3, gpus=2, logger=wandb_logger, strategy="ddp", callbacks=callbacks
    )
    trainer.fit(model, train_loader, val_loader)
```

## Examples

You can follow along in a [video tutorial with a Colab notebook](https://wandb.me/lit-colab).

## Frequently asked questions

### How does W\&B integrate with Lightning?

The core integration is based on the [Lightning `loggers` API](https://lightning.ai/docs/pytorch/stable/extensions/logging.html), which lets you write much of your logging code in a framework-agnostic way. `Logger`s are passed to the [Lightning `Trainer`](https://lightning.ai/docs/pytorch/stable/common/trainer.html) and are triggered based on that API's rich [hook-and-callback system](https://lightning.ai/docs/pytorch/stable/extensions/callbacks.html). This keeps your research code well-separated from engineering and logging code.

### What does the integration log without any additional code?

We'll save your model checkpoints to W\&B, where you can view them or download them for use in future runs. We'll also capture [system metrics](/models/ref/python/experiments/system-metrics), like GPU usage and network I/O, environment information, like hardware and OS information, [code state](/models/app/features/panels/code/) (including git commit and diff patch, notebook contents and session history), and anything printed to the standard out.

### What if I need to use `wandb.run` in my training setup?

You need to expand the scope of the variable you need to access yourself. In other words, make sure that the initial conditions are the same on all processes.

```python theme={null}
if os.environ.get("LOCAL_RANK", None) is None:
    os.environ["WANDB_DIR"] = wandb.run.dir
```

If they are, you can use `os.environ["WANDB_DIR"]` to set up the model checkpoints directory. This way, any non-zero rank process can access `wandb.run.dir`.


# OpenAI Gym
Source: https://docs.wandb.ai/models/integrations/openai-gym

Integrate W&B with OpenAI Gym to track reinforcement learning experiments and record episode performance videos.

<Note>
  "The team that has been maintaining Gym since 2021 has moved all future development to [Gymnasium](https://github.com/Farama-Foundation/Gymnasium), a drop in replacement for Gym (import gymnasium as gym), and Gym will not be receiving any future updates." ([Source](https://github.com/openai/gym#the-team-that-has-been-maintaining-gym-since-2021-has-moved-all-future-development-to-gymnasium-a-drop-in-replacement-for-gym-import-gymnasium-as-gym-and-gym-will-not-be-receiving-any-future-updates-please-switch-over-to-gymnasium-as-soon-as-youre-able-to-do-so-if-youd-like-to-read-more-about-the-story-behind-this-switch-please-check-out-this-blog-post))

  Since Gym is no longer an actively maintained project, try out our integration with Gymnasium.
</Note>

If you're using [OpenAI Gym](https://github.com/openai/gym), W\&B automatically logs videos of your environment generated by `gym.wrappers.Monitor`. Just set the `monitor_gym` keyword argument to [`wandb.init()`](/models/ref/python/functions/init) to `True` or call `wandb.gym.monitor()`.

Our gym integration is very light. We simply [look at the name of the video file](https://github.com/wandb/wandb/blob/master/wandb/integration/gym/__init__.py#L15) being logged from `gym` and name it after that or fall back to `"videos"` if we don't find a match. If you want more control, you can always just manually [log a video](/models/track/log/media/).

The [OpenRL Benchmark](https://wandb.me/openrl-benchmark-report) by[ CleanRL](https://github.com/vwxyzjn/cleanrl) uses this integration for its OpenAI Gym examples. You can find source code (including [the specific code used for specific runs](https://wandb.ai/cleanrl/cleanrl.benchmark/runs/2jrqfugg/code?workspace=user-costa-huang)) that demonstrates how to use gym with

<Frame>
  <img alt="OpenAI Gym dashboard" />
</Frame>


# PyTorch
Source: https://docs.wandb.ai/models/integrations/pytorch

Integrate W&B with PyTorch for experiment tracking, dataset versioning, and logging of metrics, gradients, and models.

<ColabLink />

Use [W\&B](https://wandb.ai) for machine learning experiment tracking, dataset versioning, and project collaboration.

<Frame>
  <img alt="Benefits of using W&B" />
</Frame>

## What this notebook covers

We show you how to integrate W\&B with your PyTorch code to add experiment tracking to your pipeline.

<Frame>
  <img alt="PyTorch and W&B integration diagram" />
</Frame>

```python theme={null}
# import the library
import wandb

# capture a dictionary of hyperparameters with config
config = {
    "learning_rate": 0.001,
    "epochs": 100,
    "batch_size": 128
}

# start a new experiment
with wandb.init(project="new-sota-model", config=config) as run:

    # set up model and data
    model, dataloader = get_model(), get_data()

    # optional: track gradients
    run.watch(model)

    for batch in dataloader:
    metrics = model.training_step()
    # log metrics inside your training loop to visualize model performance
    run.log(metrics)

    # optional: save model at the end
    model.to_onnx()
    run.save("model.onnx")
```

Follow along with a [video tutorial](https://wandb.me/pytorch-video).

**Note**: Sections starting with *Step* are all you need to integrate W\&B in an existing pipeline. The rest just loads data and defines a model.

## Install, import, and log in

```python theme={null}
import os
import random

import numpy as np
import torch
import torch.nn as nn
import torchvision
import torchvision.transforms as transforms
from tqdm.auto import tqdm

# Ensure deterministic behavior
torch.backends.cudnn.deterministic = True
random.seed(hash("setting random seeds") % 2**32 - 1)
np.random.seed(hash("improves reproducibility") % 2**32 - 1)
torch.manual_seed(hash("by removing stochasticity") % 2**32 - 1)
torch.cuda.manual_seed_all(hash("so runs are repeatable") % 2**32 - 1)

# Device configuration
device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")

# remove slow mirror from list of MNIST mirrors
torchvision.datasets.MNIST.mirrors = [mirror for mirror in torchvision.datasets.MNIST.mirrors
                                      if not mirror.startswith("http://yann.lecun.com")]
```

### Step 0: Install W\&B

To get started, we'll need to get the library.
`wandb` is easily installed using `pip`.

```python theme={null}
!pip install wandb onnx -Uq
```

### Step 1: Import W\&B and Login

In order to log data to our web service,
you'll need to log in.

If this is your first time using W\&B,
you'll need to sign up for a free account at the link that appears.

```
import wandb

wandb.login()
```

## Define the experiment and pipeline

### Track metadata and hyperparameters with `wandb.init()`

Programmatically, the first thing we do is define our experiment:
what are the hyperparameters? what metadata is associated with this run?

It's a pretty common workflow to store this information in a `config` dictionary
(or similar object)
and then access it as needed.

For this example, we're only letting a few hyperparameters vary
and hand-coding the rest.
But any part of your model can be part of the `config`.

We also include some metadata: we're using the MNIST dataset and a convolutional
architecture. If we later work with, say,
fully connected architectures on CIFAR in the same project,
this will help us separate our runs.

```python theme={null}
config = dict(
    epochs=5,
    classes=10,
    kernels=[16, 32],
    batch_size=128,
    learning_rate=0.005,
    dataset="MNIST",
    architecture="CNN")
```

Now, let's define the overall pipeline,
which is pretty typical for model-training:

1. we first `make` a model, plus associated data and optimizer, then
2. we `train` the model accordingly and finally
3. `test` it to see how training went.

We'll implement these functions below.

```python theme={null}
def model_pipeline(hyperparameters):

    # tell wandb to get started
    with wandb.init(project="pytorch-demo", config=hyperparameters) as run:
        # access all HPs through run.config, so logging matches execution.
        config = run.config

        # make the model, data, and optimization problem
        model, train_loader, test_loader, criterion, optimizer = make(config)
        print(model)

        # and use them to train the model
        train(model, train_loader, criterion, optimizer, config)

        # and test its final performance
        test(model, test_loader)

    return model
```

The only difference here from a standard pipeline
is that it all occurs inside the context of `wandb.init()`.
Calling this function sets up a line of communication
between your code and our servers.

Passing the `config` dictionary to `wandb.init()`
immediately logs all that information to us,
so you'll always know what hyperparameter values
you set your experiment to use.

To ensure the values you chose and logged are always the ones that get used
in your model, we recommend using the `run.config` copy of your object.
Check the definition of `make` below to see some examples.

> *Side Note*: We take care to run our code in separate processes,
> so that any issues on our end
> (such as if a giant sea monster attacks our data centers)
> don't crash your code.
> Once the issue is resolved, such as when the Kraken returns to the deep,
> you can log the data with `wandb sync`.

```python theme={null}
def make(config):
    # Make the data
    train, test = get_data(train=True), get_data(train=False)
    train_loader = make_loader(train, batch_size=config.batch_size)
    test_loader = make_loader(test, batch_size=config.batch_size)

    # Make the model
    model = ConvNet(config.kernels, config.classes).to(device)

    # Make the loss and optimizer
    criterion = nn.CrossEntropyLoss()
    optimizer = torch.optim.Adam(
        model.parameters(), lr=config.learning_rate)
    
    return model, train_loader, test_loader, criterion, optimizer
```

### Define the data loading and model

Now, we need to specify how the data is loaded and what the model looks like.

This part is very important, but it's
no different from what it would be without `wandb`,
so we won't dwell on it.

```python theme={null}
def get_data(slice=5, train=True):
    full_dataset = torchvision.datasets.MNIST(root=".",
                                              train=train, 
                                              transform=transforms.ToTensor(),
                                              download=True)
    #  equiv to slicing with [::slice] 
    sub_dataset = torch.utils.data.Subset(
      full_dataset, indices=range(0, len(full_dataset), slice))
    
    return sub_dataset


def make_loader(dataset, batch_size):
    loader = torch.utils.data.DataLoader(dataset=dataset,
                                         batch_size=batch_size, 
                                         shuffle=True,
                                         pin_memory=True, num_workers=2)
    return loader
```

Defining the model is normally the fun part.

But nothing changes with `wandb`,
so we're gonna stick with a standard ConvNet architecture.

Don't be afraid to mess around with this and try some experiments --
all your results will be logged on [wandb.ai](https://wandb.ai).

```python theme={null}
# Conventional and convolutional neural network

class ConvNet(nn.Module):
    def __init__(self, kernels, classes=10):
        super(ConvNet, self).__init__()
        
        self.layer1 = nn.Sequential(
            nn.Conv2d(1, kernels[0], kernel_size=5, stride=1, padding=2),
            nn.ReLU(),
            nn.MaxPool2d(kernel_size=2, stride=2))
        self.layer2 = nn.Sequential(
            nn.Conv2d(16, kernels[1], kernel_size=5, stride=1, padding=2),
            nn.ReLU(),
            nn.MaxPool2d(kernel_size=2, stride=2))
        self.fc = nn.Linear(7 * 7 * kernels[-1], classes)
        
    def forward(self, x):
        out = self.layer1(x)
        out = self.layer2(out)
        out = out.reshape(out.size(0), -1)
        out = self.fc(out)
        return out
```

### Define training logic

Moving on in our `model_pipeline`, it's time to specify how we `train`.

Two `wandb` functions come into play here: `watch` and `log`.

## Track gradients with `run.watch()` and everything else with `run.log()`

`run.watch()` will log the gradients and the parameters of your model,
every `log_freq` steps of training.

All you need to do is call it before you start training.

The rest of the training code remains the same:
we iterate over epochs and batches,
running forward and backward passes
and applying our `optimizer`.

```python theme={null}
def train(model, loader, criterion, optimizer, config):
    # Tell wandb to watch what the model gets up to: gradients, weights, and more.
    run = wandb.init(project="pytorch-demo", config=config)
    run.watch(model, criterion, log="all", log_freq=10)

    # Run training and track with wandb
    total_batches = len(loader) * config.epochs
    example_ct = 0  # number of examples seen
    batch_ct = 0
    for epoch in tqdm(range(config.epochs)):
        for _, (images, labels) in enumerate(loader):

            loss = train_batch(images, labels, model, optimizer, criterion)
            example_ct +=  len(images)
            batch_ct += 1

            # Report metrics every 25th batch
            if ((batch_ct + 1) % 25) == 0:
                train_log(loss, example_ct, epoch)


def train_batch(images, labels, model, optimizer, criterion):
    images, labels = images.to(device), labels.to(device)
    
    # Forward pass ➡
    outputs = model(images)
    loss = criterion(outputs, labels)
    
    # Backward pass ⬅
    optimizer.zero_grad()
    loss.backward()

    # Step with optimizer
    optimizer.step()

    return loss
```

The only difference is in the logging code:
where previously you might have reported metrics by printing to the terminal,
now you pass the same information to `run.log()`.

`run.log()` expects a dictionary with strings as keys.
These strings identify the objects being logged, which make up the values.
You can also optionally log which `step` of training you're on.

> *Side Note*: I like to use the number of examples the model has seen,
> since this makes for easier comparison across batch sizes,
> but you can use raw steps or batch count. For longer training runs, it can also make sense to log by `epoch`.

```python theme={null}
def train_log(loss, example_ct, epoch):
    with wandb.init(project="pytorch-demo") as run:
        # Log the loss and epoch number
        # This is where we log the metrics to W&B
        run.log({"epoch": epoch, "loss": loss}, step=example_ct)
        print(f"Loss after {str(example_ct).zfill(5)} examples: {loss:.3f}")
```

### Define testing logic

Once the model is done training, we want to test it:
run it against some fresh data from production, perhaps,
or apply it to some hand-curated examples.

## (Optional) Call `run.save()`

This is also a great time to save the model's architecture
and final parameters to disk.
For maximum compatibility, we'll `export` our model in the
[Open Neural Network eXchange (ONNX) format](https://onnx.ai/).

Passing that filename to `run.save()` ensures that the model parameters
are saved to W\&B's servers: no more losing track of which `.h5` or `.pb`
corresponds to which training runs.

For more advanced `wandb` features for storing, versioning, and distributing
models, check out our [Artifacts tools](https://www.wandb.com/artifacts).

```python theme={null}
def test(model, test_loader):
    model.eval()

    with wandb.init(project="pytorch-demo") as run:
        # Run the model on some test examples
        with torch.no_grad():
            correct, total = 0, 0
            for images, labels in test_loader:
                images, labels = images.to(device), labels.to(device)
                outputs = model(images)
                _, predicted = torch.max(outputs.data, 1)
                total += labels.size(0)
                correct += (predicted == labels).sum().item()

            print(f"Accuracy of the model on the {total} " +
                f"test images: {correct / total:%}")
            
            run.log({"test_accuracy": correct / total})

        # Save the model in the exchangeable ONNX format
        torch.onnx.export(model, images, "model.onnx")
        run.save("model.onnx")
```

### Run training and watch your metrics live on wandb.ai

Now that we've defined the whole pipeline and slipped in
those few lines of W\&B code,
we're ready to run our fully tracked experiment.

We'll report a few links to you:
our documentation,
the Project page, which organizes all the runs in a project, and
the Run page, where this run's results will be stored.

Navigate to the Run page and check out these tabs:

1. **Charts**, where the model gradients, parameter values, and loss are logged throughout training
2. **System**, which contains a variety of system metrics, including Disk I/O utilization, CPU and GPU metrics (watch that temperature soar), and more
3. **Logs**, which has a copy of anything pushed to standard out during training
4. **Files**, where, once training is complete, you can click on the `model.onnx` to view our network with the [Netron model viewer](https://github.com/lutzroeder/netron).

Once the run in finished, when the `with wandb.init()` block exits,
we'll also print a summary of the results in the cell output.

```python theme={null}
# Build, train and analyze the model with the pipeline
model = model_pipeline(config)
```

### Test Hyperparameters with Sweeps

We only looked at a single set of hyperparameters in this example.
But an important part of most ML workflows is iterating over
a number of hyperparameters.

You can use W\&B Sweeps to automate hyperparameter testing and explore the space of possible models and optimization strategies.

Check out a [Colab notebook demonstrating hyperparameter optimization using W\&B Sweeps](https://wandb.me/sweeps-colab).

Running a hyperparameter sweep with W\&B is very easy. There are just 3 simple steps:

1. **Define the sweep:** We do this by creating a dictionary or a [YAML file](/models/sweeps/define-sweep-configuration/) that specifies the parameters to search through, the search strategy, the optimization metric et all.

2. **Initialize the sweep:**
   `sweep_id = wandb.sweep(sweep_config)`

3. **Run the sweep agent:**
   `wandb.agent(sweep_id, function=train)`

That's all there is to running a hyperparameter sweep.

<Frame>
  <img alt="PyTorch training dashboard" />
</Frame>

## Example gallery

Explore examples of projects tracked and visualized with W\&B in our [Gallery →](https://app.wandb.ai/gallery).

## Advanced setup

1. [Environment variables](/platform/hosting/env-vars/): Set API keys in environment variables so you can run training on a managed cluster.
2. [Offline mode](/support/models/articles/how-do-i-run-wandb-offline): Use `dryrun` mode to train offline and sync results later.
3. [On-prem](/platform/hosting/hosting-options/self-managed): Install W\&B in a private cloud or air-gapped servers in your own infrastructure. We have local installations for everyone from academics to enterprise teams.
4. [Sweeps](/models/sweeps/): Set up hyperparameter search quickly with our lightweight tool for tuning.


# PyTorch Geometric
Source: https://docs.wandb.ai/models/integrations/pytorch-geometric

Integrate W&B with PyTorch Geometric for graph visualization and experiment tracking in geometric deep learning.

[PyTorch Geometric](https://github.com/pyg-team/pytorch_geometric) or PyG is one of the most popular libraries for geometric deep learning and W\&B works extremely well with it for visualizing graphs and tracking experiments.

After you have installed PyTorch Geometric, follow these steps to get started.

## Sign up and create an API key

An API key authenticates your machine to W\&B. You can generate an API key from your user profile.

<Note>
  For a more streamlined approach, create an API key by going directly to [User Settings](https://wandb.ai/settings). Copy the newly created API key immediately and save it in a secure location such as a password manager.
</Note>

1. Click your user profile icon in the upper right corner.
2. Select **User Settings**, then scroll to the **API Keys** section.

## Install the `wandb` library and log in

To install the `wandb` library locally and log in:

<Tabs>
  <Tab title="Command Line">
    1. Set the `WANDB_API_KEY` [environment variable](/models/track/environment-variables/) to your API key.

       ```bash theme={null}
       export WANDB_API_KEY=<your_api_key>
       ```

    2. Install the `wandb` library and log in.

       ```shell theme={null}
       pip install wandb

       wandb login
       ```
  </Tab>

  <Tab title="Python">
    ```bash theme={null}
    pip install wandb
    ```

    ```python theme={null}
    import wandb
    wandb.login()
    ```
  </Tab>

  <Tab title="Python notebook">
    ```notebook theme={null}
    !pip install wandb

    import wandb
    wandb.login()
    ```
  </Tab>
</Tabs>

## Visualize the graphs

You can save details about the input graphs including number of edges, number of nodes and more. W\&B supports logging plotly charts and HTML panels so any visualizations you create for your graph can then also be logged to W\&B.

### Use PyVis

The following snippet shows how you could do that with PyVis and HTML.

```python theme={null}
from pyvis.network import Network
import wandb

with wandb.init(project=’graph_vis’) as run:
    net = Network(height="750px", width="100%", bgcolor="#222222", font_color="white")

    # Add the edges from the PyG graph to the PyVis network
    for e in tqdm(g.edge_index.T):
        src = e[0].item()
        dst = e[1].item()

        net.add_node(dst)
        net.add_node(src)
        
        net.add_edge(src, dst, value=0.1)

    # Save the PyVis visualisation to a HTML file
    net.show("graph.html")
    run.log({"eda/graph": wandb.Html("graph.html")})
```

<Frame>
  <img alt="Interactive graph visualization" />
</Frame>

### Use Plotly

To use plotly to create a graph visualization, first you need to convert the PyG graph to a networkx object. Following this you will need to create Plotly scatter plots for both nodes and edges. The snippet below can be used for this task.

```python theme={null}
def create_vis(graph):
    G = to_networkx(graph)
    pos = nx.spring_layout(G)

    edge_x = []
    edge_y = []
    for edge in G.edges():
        x0, y0 = pos[edge[0]]
        x1, y1 = pos[edge[1]]
        edge_x.append(x0)
        edge_x.append(x1)
        edge_x.append(None)
        edge_y.append(y0)
        edge_y.append(y1)
        edge_y.append(None)

    edge_trace = go.Scatter(
        x=edge_x, y=edge_y,
        line=dict(width=0.5, color='#888'),
        hoverinfo='none',
        mode='lines'
    )

    node_x = []
    node_y = []
    for node in G.nodes():
        x, y = pos[node]
        node_x.append(x)
        node_y.append(y)

    node_trace = go.Scatter(
        x=node_x, y=node_y,
        mode='markers',
        hoverinfo='text',
        line_width=2
    )

    fig = go.Figure(data=[edge_trace, node_trace], layout=go.Layout())

    return fig


with wandb.init(project=’visualize_graph’) as run:
    run.log({‘graph’: wandb.Plotly(create_vis(graph))})
```

<Frame>
  <img alt="A visualization created using the example function and logged inside a W&B Table." />
</Frame>

## Log metrics

You can use W\&B to track your experiments and related metrics, such as loss functions, accuracy, and more. Add the following line to your training loop:

```python theme={null}
with wandb.init(project="my_project", entity="my_entity") as run:
    run.log({
        'train/loss': training_loss,
        'train/acc': training_acc,
        'val/loss': validation_loss,
        'val/acc': validation_acc
        })
```

<Frame>
  <img alt="hits@K metrics over epochs" />
</Frame>

## More resources

* [Recommending Amazon Products using Graph Neural Networks in PyTorch Geometric](https://wandb.ai/manan-goel/gnn-recommender/reports/Recommending-Amazon-Products-using-Graph-Neural-Networks-in-PyTorch-Geometric--VmlldzozMTA3MzYw#what-does-the-data-look-like?)
* [Point Cloud Classification using PyTorch Geometric](https://wandb.ai/geekyrakshit/pyg-point-cloud/reports/Point-Cloud-Classification-using-PyTorch-Geometric--VmlldzozMTExMTE3)
* [Point Cloud Segmentation using PyTorch Geometric](https://wandb.ai/wandb/point-cloud-segmentation/reports/Point-Cloud-Segmentation-using-Dynamic-Graph-CNN--VmlldzozMTk5MDcy)


# Hugging Face Simple Transformers
Source: https://docs.wandb.ai/models/integrations/simpletransformers

How to integrate W&B with the Transformers library by Hugging Face.

This library is based on the Transformers library by Hugging Face. Simple Transformers lets you quickly train and evaluate Transformer models. Only 3 lines of code are needed to initialize a model, train the model, and evaluate a model. It supports Sequence Classification, Token Classification (NER),Question Answering,Language Model Fine-Tuning, Language Model Training, Language Generation, T5 Model, Seq2Seq Tasks , Multi-Modal Classification and Conversational AI.

To use W\&B for visualizing model training. To use this, set a project name for W\&B in the `wandb_project` attribute of the `args` dictionary. This logs all hyperparameter values, training losses, and evaluation metrics to the given project.

```python theme={null}
model = ClassificationModel('roberta', 'roberta-base', args={'wandb_project': 'project-name'})
```

Any additional arguments that go into `wandb.init()` can be passed as `wandb_kwargs`.

## Structure

The library is designed to have a separate class for every NLP task. The classes that provide similar functionality are grouped together.

* `simpletransformers.classification` - Includes all Classification models.
  * `ClassificationModel`
  * `MultiLabelClassificationModel`
* `simpletransformers.ner` - Includes all Named Entity Recognition models.
  * `NERModel`
* `simpletransformers.question_answering` - Includes all Question Answering models.
  * `QuestionAnsweringModel`

Here are some minimal examples

## MultiLabel Classification

```text theme={null}
  model = MultiLabelClassificationModel("distilbert","distilbert-base-uncased",num_labels=6,
    args={"reprocess_input_data": True, "overwrite_output_dir": True, "num_train_epochs":epochs,'learning_rate':learning_rate,
                'wandb_project': "simpletransformers"},
  )
   # Train the model
  model.train_model(train_df)

  # Evaluate the model
  result, model_outputs, wrong_predictions = model.eval_model(eval_df)
```

## Question answering

```text theme={null}
  train_args = {
    'learning_rate': wandb.config.learning_rate,
    'num_train_epochs': 2,
    'max_seq_length': 128,
    'doc_stride': 64,
    'overwrite_output_dir': True,
    'reprocess_input_data': False,
    'train_batch_size': 2,
    'fp16': False,
    'wandb_project': "simpletransformers"
}

model = QuestionAnsweringModel('distilbert', 'distilbert-base-cased', args=train_args)
model.train_model(train_data)
```

SimpleTransformers provides classes as well as training scripts for all common natural language tasks. Here is the complete list of global arguments that are supported by the library, with their default arguments.

```text theme={null}
global_args = {
  "adam_epsilon": 1e-8,
  "best_model_dir": "outputs/best_model",
  "cache_dir": "cache_dir/",
  "config": {},
  "do_lower_case": False,
  "early_stopping_consider_epochs": False,
  "early_stopping_delta": 0,
  "early_stopping_metric": "eval_loss",
  "early_stopping_metric_minimize": True,
  "early_stopping_patience": 3,
  "encoding": None,
  "eval_batch_size": 8,
  "evaluate_during_training": False,
  "evaluate_during_training_silent": True,
  "evaluate_during_training_steps": 2000,
  "evaluate_during_training_verbose": False,
  "fp16": True,
  "fp16_opt_level": "O1",
  "gradient_accumulation_steps": 1,
  "learning_rate": 4e-5,
  "local_rank": -1,
  "logging_steps": 50,
  "manual_seed": None,
  "max_grad_norm": 1.0,
  "max_seq_length": 128,
  "multiprocessing_chunksize": 500,
  "n_gpu": 1,
  "no_cache": False,
  "no_save": False,
  "num_train_epochs": 1,
  "output_dir": "outputs/",
  "overwrite_output_dir": False,
  "process_count": cpu_count() - 2 if cpu_count() > 2 else 1,
  "reprocess_input_data": True,
  "save_best_model": True,
  "save_eval_checkpoints": True,
  "save_model_every_epoch": True,
  "save_steps": 2000,
  "save_optimizer_and_scheduler": True,
  "silent": False,
  "tensorboard_dir": None,
  "train_batch_size": 8,
  "use_cached_eval_features": False,
  "use_early_stopping": False,
  "use_multiprocessing": True,
  "wandb_kwargs": {},
  "wandb_project": None,
  "warmup_ratio": 0.06,
  "warmup_steps": 0,
  "weight_decay": 0,
}
```

Refer to [simpletransformers on github](https://github.com/ThilinaRajapakse/simpletransformers) for more detailed documentation.

Checkout [this W\&B report](https://app.wandb.ai/cayush/simpletransformers/reports/Using-simpleTransformer-on-common-NLP-applications---Vmlldzo4Njk2NA) that covers training transformers on some the most popular GLUE benchmark datasets. [Try it out yourself on colab](https://colab.research.google.com/drive/1oXROllqMqVvBFcPgTKJRboTq96uWuqSz?usp=sharing).


# TensorFlow
Source: https://docs.wandb.ai/models/integrations/tensorflow

Integrate W&B with TensorFlow for logging custom metrics, using estimator hooks, and TensorBoard log synchronization.

<ColabLink />

## Get started

If you're already using TensorBoard, it's easy to integrate with wandb.

```python theme={null}
import tensorflow as tf
import wandb
```

## Log custom metrics

If you need to log additional custom metrics that aren't being logged to TensorBoard, you can call `run.log()` in your code `run.log({"custom": 0.8}) `

Setting the step argument in `run.log()` is turned off when syncing Tensorboard. If you'd like to set a different step count, you can log the metrics with a step metric as:

```python theme={null}
with wandb.init(config=tf.flags.FLAGS, sync_tensorboard=True) as run:
    run.log({"custom": 0.8, "global_step":global_step}, step=global_step)
```

## TensorFlow estimators hook

If you want more control over what gets logged, wandb also provides a hook for TensorFlow estimators. It will log all `tf.summary` values in the graph.

```python theme={null}
import tensorflow as tf
import wandb

run = wandb.init(config=tf.FLAGS)

estimator.train(hooks=[wandb.tensorflow.WandbHook(steps_per_log=1000)])
run.finish()
```

## Log manually

The simplest way to log metrics in TensorFlow is by logging `tf.summary` with the TensorFlow logger:

```python theme={null}
import wandb
run = wandb.init(config=tf.flags.FLAGS, sync_tensorboard=True)
with tf.Session() as sess:
    # ...
    wandb.tensorflow.log(tf.summary.merge_all())
```

With TensorFlow 2, the recommended way of training a model with a custom loop is via using `tf.GradientTape`. You can read more in the [TensorFlow custom training walkthrough](https://www.tensorflow.org/tutorials/customization/custom_training_walkthrough). If you want to incorporate `wandb` to log metrics in your custom TensorFlow training loops you can follow this snippet:

```python theme={null}
    with tf.GradientTape() as tape:
        # Get the probabilities
        predictions = model(features)
        # Calculate the loss
        loss = loss_func(labels, predictions)

    # Log your metrics
    run.log("loss": loss.numpy())
    # Get the gradients
    gradients = tape.gradient(loss, model.trainable_variables)
    # Update the weights
    optimizer.apply_gradients(zip(gradients, model.trainable_variables))
```

A [full example of customizing training loops in TensorFlow 2](https://www.wandb.com/articles/wandb-customizing-training-loops-in-tensorflow-2) is available.

## How is W\&B different from TensorBoard?

When the cofounders started working on W\&B, they were inspired to build a tool for the frustrated TensorBoard users at OpenAI. Here are a few things we've focused on improving:

1. **Reproduce models**: W\&B is good for experimentation, exploration, and reproducing models later. We capture not just the metrics, but also the hyperparameters and version of the code, and we can save your version-control status and model checkpoints for you so your project is reproducible.
2. **Automatic organization**: Whether you're picking up a project from a collaborator, coming back from a vacation, or dusting off an old project, W\&B makes it easy to see all the models that have been tried so no one wastes hours, GPU cycles, or carbon re-running experiments.
3. **Fast, flexible integration**: Add W\&B to your project in 5 minutes. Install our free open-source Python package and add a couple of lines to your code, and every time you run your model you'll have nice logged metrics and records.
4. **Persistent, centralized dashboard**: No matter where you train your models, whether on your local machine, in a shared lab cluster, or on spot instances in the cloud, your results are shared to the same centralized dashboard. You don't need to spend your time copying and organizing TensorBoard files from different machines.
5. **Powerful tables**: Search, filter, sort, and group results from different models. It's easy to look over thousands of model versions and find the best performing models for different tasks. TensorBoard isn't built to work well on large projects.
6. **Tools for collaboration**: Use W\&B to organize complex machine learning projects. It's easy to share a link to W\&B, and you can use private teams to have everyone sending results to a shared project. We also support collaboration via reports— add interactive visualizations and describe your work in markdown. This is a great way to keep a work log, share findings with your supervisor, or present findings to your lab or team.

Get started with a [free account](https://wandb.ai)

## Examples

We've created a few examples for you to see how the integration works:

* [Example on Github](https://github.com/wandb/examples/blob/master/examples/tensorflow/tf-estimator-mnist/mnist.py): MNIST example Using TensorFlow Estimators
* [Example on Github](https://github.com/wandb/examples/blob/master/examples/tensorflow/tf-cnn-fashion/train.py): Fashion MNIST example Using Raw TensorFlow
* [Wandb Dashboard](https://app.wandb.ai/l2k2/examples-tf-estimator-mnist/runs/p0ifowcb): View result on W\&B
* Customizing Training Loops in TensorFlow 2 - [Article](https://www.wandb.com/articles/wandb-customizing-training-loops-in-tensorflow-2) | [Dashboard](https://app.wandb.ai/sayakpaul/custom_training_loops_tf)


# PyTorch torchtune
Source: https://docs.wandb.ai/models/integrations/torchtune

Use W&B logging in PyTorch torchtune for tracking LLM fine-tuning experiments with the WandBLogger metric logger.

<ColabLink />

[torchtune](https://meta-pytorch.org/torchtune/stable/index.html) is a PyTorch-based library designed to streamline the authoring, fine-tuning, and experimentation processes for large language models (LLMs). Additionally, torchtune has built-in support for [logging with W\&B](https://meta-pytorch.org/torchtune/stable/deep_dives/wandb_logging.html), enhancing tracking and visualization of training processes.

<Frame>
  <img alt="TorchTune training dashboard" />
</Frame>

Check the W\&B blog post on [Fine-tuning Mistral 7B using torchtune](https://wandb.ai/capecape/torchtune-mistral/reports/torchtune-The-new-PyTorch-LLM-fine-tuning-library---Vmlldzo3NTUwNjM0).

## W\&B logging at your fingertips

<Tabs>
  <Tab title="Command line">
    Override command line arguments at launch:

    ```bash theme={null}
    tune run lora_finetune_single_device --config llama3/8B_lora_single_device \
      metric_logger._component_=torchtune.utils.metric_logging.WandBLogger \
      metric_logger.project="llama3_lora" \
      log_every_n_steps=5
    ```
  </Tab>

  <Tab title="Recipe">
    Enable W\&B logging on the recipe's config:

    ```yaml theme={null}
    # inside llama3/8B_lora_single_device.yaml
    metric_logger:
      _component_: torchtune.utils.metric_logging.WandBLogger
      project: llama3_lora
    log_every_n_steps: 5
    ```
  </Tab>
</Tabs>

## Use the W\&B metric logger

Enable W\&B logging on the recipe's config file by modifying the `metric_logger` section. Change the `_component_` to `torchtune.utils.metric_logging.WandBLogger` class. You can also pass a `project` name and `log_every_n_steps` to customize the logging behavior.

You can also pass any other `kwargs` as you would to the [wandb.init()](/models/ref/python/functions/init) method. For example, if you are working on a team, you can pass the `entity` argument to the `WandBLogger` class to specify the team name.

<Tabs>
  <Tab title="Recipe">
    ```yaml theme={null}
    # inside llama3/8B_lora_single_device.yaml
    metric_logger:
      _component_: torchtune.utils.metric_logging.WandBLogger
      project: llama3_lora
      entity: my_project
      job_type: lora_finetune_single_device
      group: my_awesome_experiments
    log_every_n_steps: 5
    ```
  </Tab>

  <Tab title="Command Line">
    ```shell theme={null}
    tune run lora_finetune_single_device --config llama3/8B_lora_single_device \
      metric_logger._component_=torchtune.utils.metric_logging.WandBLogger \
      metric_logger.project="llama3_lora" \
      metric_logger.entity="my_project" \
      metric_logger.job_type="lora_finetune_single_device" \
      metric_logger.group="my_awesome_experiments" \
      log_every_n_steps=5
    ```
  </Tab>
</Tabs>

## What is logged?

You can explore the W\&B dashboard to see the logged metrics. By default W\&B logs all of the hyperparameters from the config file and the launch overrides.

W\&B captures the resolved config on the **Overview** tab. W\&B also stores the config in YAML format on the [Files tab](https://wandb.ai/capecape/torchtune/runs/joyknwwa/files).

<Frame>
  <img alt="TorchTune configuration" />
</Frame>

### Logged metrics

Each recipe has its own training loop. Check each individual recipe to see its logged metrics, which include these by default:

| Metric              | Description                                                                                                                                                                                                                                                            |
| ------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `loss`              | The loss of the model                                                                                                                                                                                                                                                  |
| `lr`                | The learning rate                                                                                                                                                                                                                                                      |
| `tokens_per_second` | The tokens per second of the model                                                                                                                                                                                                                                     |
| `grad_norm`         | The gradient norm of the model                                                                                                                                                                                                                                         |
| `global_step`       | Corresponds to the current step in the training loop. Takes into account gradient accumulation, basically every time an optimizer step is taken, the model is updated, the gradients are accumulated and the model is updated once every `gradient_accumulation_steps` |

<Note>
  `global_step` is not the same as the number of training steps. It corresponds to the current step in the training loop. Takes into account gradient accumulation, basically every time an optimizer step is taken the `global_step` is incremented by 1. For example, if the dataloader has 10 batches, gradient accumulation steps is 2 and run for 3 epochs, the optimizer will step 15 times, in this case `global_step` will range from 1 to 15.
</Note>

The streamlined design of torchtune allows to easily add custom metrics or modify the existing ones. It suffices to modify the corresponding [recipe file](https://github.com/meta-pytorch/torchtune/tree/main/recipes), for example, computing one could log `current_epoch` as a percentage of the total number of epochs as following:

```python theme={null}
# inside `train.py` function in the recipe file
self._metric_logger.log_dict(
    {"current_epoch": self.epochs * self.global_step / self._steps_per_epoch},
    step=self.global_step,
)
```

<Note>
  This is a fast evolving library, the current metrics are subject to change. If you want to add a custom metric, you should modify the recipe and call the corresponding `self._metric_logger.*` function.
</Note>

## Save and load checkpoints

The torchtune library supports various [checkpoint formats](https://meta-pytorch.org/torchtune/stable/deep_dives/checkpointer.html). Depending on the origin of the model you are using, you should switch to the appropriate [checkpointer class](https://meta-pytorch.org/torchtune/stable/deep_dives/checkpointer.html).

If you want to save the model checkpoints to [W\&B Artifacts](/models/artifacts/), the simplest solution is to override the `save_checkpoint` functions inside the corresponding recipe.

Here is an example of how you can override the `save_checkpoint` function to save the model checkpoints to W\&B Artifacts.

```python theme={null}
def save_checkpoint(self, epoch: int) -> None:
    ...
    ## Let's save the checkpoint to W&B
    ## depending on the Checkpointer Class the file will be named differently
    ## Here is an example for the full_finetune case
    checkpoint_file = Path.joinpath(
        self._checkpointer._output_dir, f"torchtune_model_{epoch}"
    ).with_suffix(".pt")
    wandb_artifact = wandb.Artifact(
        name=f"torchtune_model_{epoch}",
        type="model",
        # description of the model checkpoint
        description="Model checkpoint",
        # you can add whatever metadata you want as a dict
        metadata={
            utils.SEED_KEY: self.seed,
            utils.EPOCHS_KEY: self.epochs_run,
            utils.TOTAL_EPOCHS_KEY: self.total_epochs,
            utils.MAX_STEPS_KEY: self.max_steps_per_epoch,
        },
    )
    wandb_artifact.add_file(checkpoint_file)
    wandb.log_artifact(wandb_artifact)
```


# XGBoost
Source: https://docs.wandb.ai/models/integrations/xgboost

Integrate W&B with XGBoost to log gradient boosting metrics, feature importance, and model performance automatically.

<ColabLink />

The `wandb` library has a `WandbCallback` callback for logging metrics, configs and saved boosters from training with XGBoost. Here you can see a [live W\&B Dashboard](https://wandb.ai/morg/credit_scorecard) with outputs from the XGBoost `WandbCallback`.

<Frame>
  <img alt="W&B Dashboard using XGBoost" />
</Frame>

## Get started

Logging XGBoost metrics, configs and booster models to W\&B is as easy as passing the `WandbCallback` to XGBoost:

```python theme={null}
from wandb.integration.xgboost import WandbCallback
import xgboost as XGBClassifier

...
# Start a wandb run
with wandb.init() as run:
  # Pass WandbCallback to the model
  bst = XGBClassifier()
  bst.fit(X_train, y_train, callbacks=[WandbCallback(log_model=True)])
```

You can open [this notebook](https://wandb.me/xgboost) for a comprehensive look at logging with XGBoost and W\&B

## `WandbCallback` reference

### Functionality

Passing `WandbCallback` to a XGBoost model will:

* log the booster model configuration to W\&B
* log evaluation metrics collected by XGBoost, such as rmse, accuracy etc to W\&B
* log training metrics collected by XGBoost (if you provide data to eval\_set)
* log the best score and the best iteration
* save and upload your trained model to W\&B Artifacts (when `log_model = True`)
* log feature importance plot when `log_feature_importance=True` (default).
* Capture the best eval metric in `wandb.Run.summary` when `define_metric=True` (default).

### Arguments

* `log_model`: (boolean) if True save and upload the model to W\&B Artifacts

* `log_feature_importance`: (boolean) if True log a feature importance bar plot

* `importance_type`: (str) one of `{weight, gain, cover, total_gain, total_cover}` for tree model. weight for linear model.

* `define_metric`: (boolean) if True (default) capture model performance at the best step, instead of the last step, of training in your `run.summary`.

You can review the [source code for WandbCallback](https://github.com/wandb/wandb/blob/main/wandb/integration/xgboost/xgboost.py).

For additional examples, check out the [repository of examples on GitHub](https://github.com/wandb/examples/tree/master/examples/boosting-algorithms).

## Tune your hyperparameters with Sweeps

Attaining the maximum performance out of models requires tuning hyperparameters, like tree depth and learning rate. W\&B [Sweeps](/models/sweeps/) is a powerful toolkit for configuring, orchestrating, and analyzing large hyperparameter testing experiments.

<ColabLink />

You can also try this [XGBoost & Sweeps Python script](https://github.com/wandb/examples/blob/master/examples/wandb-sweeps/sweeps-xgboost/xgboost_tune.py).

<Frame>
  <img alt="XGBoost performance comparison" />
</Frame>


# LLM Evaluation Jobs
Source: https://docs.wandb.ai/models/launch

Evaluate model checkpoints or hosted API models within W&B and analyze the results using automatically generated leaderboards.

[LLM Evaluation Jobs](/models/launch) is a benchmarking framework for evaluating an LLM model's performance, using infrastructure managed by CoreWeave. Choose from a comprehensive suite of modern, industry-standard [model evaluation benchmarks](/models/launch/evaluations), then view, analyze, and share the results using automatic leaderboards and charts in W\&B Models. LLM Evaluation Jobs removes the complexity of deploying and maintaining GPU infrastructure yourself.

<Note>
  LLM Evaluation Jobs is in **Preview** for [W\&B Multi-tenant Cloud](/platform/hosting/hosting-options/multi_tenant_cloud). Compute is free during the preview period. [Learn more](/models/launch#pricing)
</Note>

## How it works

Evaluate a model checkpoint or a publicly accessible hosted OpenAI-compatible model in just a few steps:

1. Set up an evaluation job in W\&B Models. Define its benchmarks and configuration, such as whether to generate a leaderboard.
2. Launch the evaluation job.
3. View and analyze the results and leaderboard.

Each time you launch an evaluation job with the same destination project, the project's leaderboard updates automatically.

<Frame>
  <img alt="Example evaluation job leaderboard" />
</Frame>

## Next steps

* Browse the [Evaluation benchmark catalog](/models/launch/evaluations)
* [Evaluate a model checkpoint](/models/launch/evaluate-model-checkpoint)
* [Evaluate an API-hosted model](/models/launch/evaluate-hosted-model)

## More details

### Pricing

LLM Evaluation Jobs evaluates a model checkpoint or hosted API against popular benchmarks on fully-managed CoreWeave compute, with no infrastructure to manage. You pay only for resources consumed, not idle time. Pricing has two components: compute and storage. Compute is free during public preview, and we will announce pricing at general availability. Stored results include metrics and per-example traces saved in Models runs. Storage is billed monthly based on data volume. During the preview period, LLM Evaluation Jobs is available for Multi-tenant Cloud only. See the [Pricing](https://wandb.ai/pricing) page for details.

### Job limits

An individual evaluation job has these limits:

* The maximum size for a model to evaluate is 86 GB, including context.
* Each job is limited to two GPUs.

### Requirements

* To evaluate a model checkpoint, the model weights must be packaged as a VLLM-compatible artifact. See [Example: Prepare a model](/models/launch/evaluate-model-checkpoint#example-prepare-a-model) for details and example code.
* To evaluate an OpenAI-compatible model, it must be accessible at a public URL, and an organization or team admin must configure a team secret with the API key for authentication.
* Certain benchmarks use OpenAI models for scoring. To run these benchmarks, an organization or team admin must configure team secrets with the required API keys. See the [Evaluation benchmark catalog](/models/launch/evaluations) to determine whether a benchmark has this requirement.
* Certain benchmarks require access to gated datasets in Hugging Face. To run one of these benchmarks, an organization or team admin must request access to the gated dataset in Hugging Face, generate a Hugging Face user access token, and configure it as a team secret. See the [Evaluation benchmark catalog](/models/launch/evaluations) to determine whether a benchmark has this requirement.

For more details and instructions for meeting these requirements, see:

* [Evaluate a model checkpoint](/models/launch/evaluate-model-checkpoint)
* [Evaluate a hosted API model](/models/launch/evaluate-hosted-model)


# Evaluate a hosted API model
Source: https://docs.wandb.ai/models/launch/evaluate-hosted-model

Evaluate a hosted API model using infrastructure managed by CoreWeave

<Note>
  LLM Evaluation Jobs is in **Preview** for [W\&B Multi-tenant Cloud](/platform/hosting/hosting-options/multi_tenant_cloud). Compute is free during the preview period. [Learn more](/models/launch#pricing)
</Note>

This page shows how to use [LLM Evaluation Jobs](/models/launch) to run a series of evaluation benchmarks on a hosted API model at a publicly accessible URL, using infrastructure managed by CoreWeave. Running these benchmarks helps you compare model performance, validate model quality, and publish results to a shared leaderboard without managing your own evaluation infrastructure. To evaluate a model checkpoint saved as an artifact in W\&B Models, see [Evaluate a model checkpoint](/models/launch/evaluate-model-checkpoint) instead.

## Prerequisites

Before you create an evaluation job, complete the following:

1. Review the [requirements and limitations](/models/launch#more-details) for LLM Evaluation Jobs.
2. To run certain benchmarks, a team admin must add the required API keys as team-scoped secrets. Any team member can specify the secret when configuring an evaluation job.
   * An **OpenAI API key**: Used by benchmarks that use OpenAI models for scoring. Required if the field **Scorer API key** appears after you select a benchmark. The secret must be named `OPENAI_API_KEY`.
   * A **Hugging Face user access token**: Required for certain benchmarks like `lingoly` and `lingoly2` that require access to one or more gated Hugging Face datasets. Required if the field **Hugging Face Token** appears after you select a benchmark. The API key must have access to the relevant dataset. See the Hugging Face documentation for [User access tokens](https://huggingface.co/docs/hub/en/security-tokens) and [accessing gated datasets](https://huggingface.co/docs/hub/en/datasets-gated#access-gated-datasets-as-a-user).
   * To evaluate a model provided by [Serverless Inference](/inference), an organization or team admin must create `WANDB_API_KEY` with any value. The secret isn't used for authentication.
3. The model to evaluate must be available at a publicly accessible URL. An organization or team admin must create a team-scoped secret with the API key for authentication.
4. Create a new [W\&B project](/models/track/project-page) for the evaluation results. From the project sidebar, click **Create new project**.
5. Review the documentation for a given benchmark to understand how it works and learn about specific requirements. The [Available evaluation benchmarks](/models/launch/evaluations) reference includes relevant links.

## Evaluate your model

Follow these steps to set up and launch an evaluation job. When you finish, your benchmark runs are queued on CoreWeave-managed infrastructure and their results appear in the destination W\&B project you specify.

1. Log in to W\&B, then click **Launch** in the project sidebar. The **LLM Evaluation Jobs** page displays.
2. Click **Evaluate hosted API model** to set up the evaluation.
3. Select a destination project to save the evaluation results to.
4. In the **Model** section, specify the base URL and model name to evaluate, and select the API key to use for authentication. Provide the model name in OpenAI-compatible format defined by the [AI Security Institute](https://inspect.aisi.org.uk/providers.html#openai-api). For example, specify an OpenAI model in the following syntax, where `[MODEL-NAME]` is the name of the model: `openai/[MODEL-NAME]`. For a list of hosted model providers and models, see [AI Security Institute's model provider reference](https://inspect.aisi.org.uk/providers.html).
   * To evaluate a model provided by [Serverless Inference](/inference), set the base URL to `https://api.inference.wandb.ai/v1` and specify the model name in the following syntax, where `[MODEL-ID]` is the model ID: `openai-api/wandb/[MODEL-ID]`. Refer to the [Inference model catalog](/inference/models) for details.
   * To use the [OpenRouter](https://inspect.aisi.org.uk/providers.html#openrouter) provider, prefix the model name with `openrouter` in the following syntax, where `[MODEL-NAME]` is the name of the model: `openrouter/[MODEL-NAME]`.
   * To evaluate a custom OpenAPI-compliant model, specify the model name in the following syntax, where `[MODEL-NAME]` is the name of the model: `openai-api/wandb/[MODEL-NAME]`.
5. Click **Select evaluations**, then select up to four benchmarks to run.
6. If you select benchmarks that use OpenAI models for scoring, the **Scorer API key** field displays. Click it, then select the `OPENAI_API_KEY` secret. A team admin can create a secret from this drawer by clicking **Create secret**.
7. If you select benchmarks that require access to gated datasets in Hugging Face, a **Hugging Face token** field displays. [Request access to the relevant dataset](https://huggingface.co/docs/hub/en/datasets-gated#access-gated-datasets-as-a-user), then select the secret that contains the Hugging Face user access token.
8. Optional: Set **Sample limit** to a positive integer to limit the maximum number of benchmark samples to evaluate. Otherwise, all samples in the task are included.
9. To create a leaderboard automatically, click **Publish results to leaderboard**. The leaderboard displays all evaluations together in a workspace panel, and you can also share it in a report.
10. Click **Launch** to launch the evaluation job.
11. Click the circular arrow icon at the top of the page to open the recent run modal. Evaluation jobs appear with your other recent runs. Click the name of a finished run to open it in single-run view, or click the **Leaderboard** link to open the leaderboard directly. For details, see [View the results](#view-the-results).

This example job runs the `simpleqa` benchmark against the OpenAI model `o4-mini`:

<Frame>
  <img alt="Example hosted model evaluation job" />
</Frame>

If you published results to a leaderboard, you can compare evaluations side by side. This example leaderboard visualizes the performance of several OpenAI models together:

<Frame>
  <img alt="Example leaderboard visualizing the performance of several hosted models" />
</Frame>

## Review evaluation results

Review your evaluation job results in W\&B Models in the destination project's workspace.

1. Click the circular arrow icon at the top of the page to open the recent run modal, where evaluation jobs appear with other runs in the project. If the evaluation job has a leaderboard, click **Leaderboard** to open the leaderboard in full screen, or click a run name to open it in the project in single-run view.
2. View the evaluation job's traces in the **Evaluations** section of a workspace or in the **Traces** tab of the **Weave** sidebar panel.
3. Click the **Overview** tab to view detailed information about the evaluation job, including its configuration and summary metrics.
4. Click the **Logs** tab to view, search, or download the evaluation job's debug logs.
5. Click the **Files** tab to browse, view, or download the evaluation job's files, including code, log, configuration, and other output files.

## Customize a leaderboard

The leaderboard shows results for all evaluation jobs sent to a given project, with one row per benchmark per evaluation job. Columns display details like the trace, input values, and output values for the evaluation job. For more information about leaderboards, see [Leaderboards in Weave](/weave/guides/core-types/leaderboards).

<Tip>To give feedback about a result from the leaderboard, click the emoji icon or the chat icon in the **Feedback** column.</Tip>

* By default, all evaluation jobs are displayed. Filter or search for an evaluation job using the run selector at the left.
* By default, evaluation jobs are ungrouped. To group by one or more columns, click the **Group** icon. You can show or hide a group, or expand a group to view its runs.
* By default, all operations are displayed. To display only a single operation, click **All ops** and select an operation.
* To sort by a column, click the column heading. To customize the display of columns, click **Columns**.
  * By default, headers are organized in a single level. You can increase the header depth to organize related headers together.
  * Select or deselect individual columns to show or hide them, or show or hide all columns with a click.
  * Pin columns to display them before unpinned columns.

## Export a leaderboard

To export a leaderboard:

1. Click the download icon, located near the **Columns** button.
2. To optimize the export size, only the trace roots are exported by default. To export full traces, turn off **Trace roots only**.
3. To optimize the export size, feedback and costs are not exported by default. To include them in the export, toggle **Feedback** or **Costs**.
4. By default, the export is in JSONL format. To customize the format, click **Export to file** and select a format.
5. To export the leaderboard in your browser, click **Export**.
6. To export the leaderboard programmatically, select **Python** or **cURL**, then click **Copy** and run the script or command.

## Re-run an evaluation job

Depending on your situation, there are multiple ways to re-run an evaluation job or view its configuration.

* To re-run the last evaluation job again, follow the steps in [Evaluate your model](#evaluate-your-model). Select the destination project, then the model artifact details and the selected benchmarks you selected last time are populated automatically. Optionally, make adjustments, then launch the evaluation job.
* To re-run an evaluation job from the project's **Runs** tab or run selector, hover over the run name and click the play icon. The job configuration drawer displays with the settings pre-populated. Optionally adjust the settings, then click **Launch**.
* To re-run an evaluation job from a different project, import its configuration:
  1. Follow the steps in [Evaluate your model](#evaluate-your-model). After you select the destination project, click **Import configuration**.
  2. Select the project that contains the evaluation job to import, then select the evaluation job run. The job configuration drawer displays with the settings pre-populated.
  3. Optionally adjust the configuration.
  4. Click **Launch**.

## Export an evaluation job configuration

Export an evaluation job's configuration from the run's **Files** tab.

1. Open the run in single-run view.
2. Click the **Files** tab.
3. Click the download button next to `config.yaml` to download it locally.


# Evaluate a model checkpoint
Source: https://docs.wandb.ai/models/launch/evaluate-model-checkpoint

Evaluate a VLLM-compatible model checkpoint using infrastructure managed by CoreWeave

<Note>
  LLM Evaluation Jobs is in **Preview** for [W\&B Multi-tenant Cloud](/platform/hosting/hosting-options/multi_tenant_cloud). Compute is free during the preview period. [Learn more](/models/launch#pricing)
</Note>

This page shows how to use [LLM Evaluation Jobs](/models/launch) to run a series of evaluation benchmarks on a fine-tuned model in W\&B Models, using infrastructure managed by CoreWeave. To evaluate a hosted API model served at a publicly accessible URL, see [Evaluate an API-hosted model](/models/launch/evaluate-hosted-model) instead, or run a small benchmark against a public OpenAI model endpoint with a streamlined [Quickstart](/models/launch#quickstart).

## Prerequisites

Before you evaluate a model checkpoint, complete the following:

1. Review the [requirements and limitations](/models/launch#more-details) for LLM Evaluation Jobs.
2. To run certain benchmarks, a team admin must add the required API keys as [team-scoped secrets](/platform/secrets#add-a-secret). Any team member can specify the secret when configuring an evaluation job. See [Evaluation model catalog](/models/launch/evaluations) for requirements.
   * An **OpenAPI API key**: used by benchmarks that use OpenAI models for scoring. Required if the field **Scorer API key** appears after you select a benchmark. The secret must be named `OPENAI_API_KEY`.
   * A **Hugging Face user access token**: required for certain benchmarks like `lingoly` and `lingoly2` that require access to one or more gated Hugging Face datasets. Required if the field **Hugging Face token** appears after selecting a benchmark. The API key must have access to the relevant dataset. See the Hugging Face documentation for [User access tokens](https://huggingface.co/docs/hub/en/security-tokens) and [accessing gated datasets](https://huggingface.co/docs/hub/en/datasets-gated#access-gated-datasets-as-a-user).
3. Create a new [W\&B project](/models/track/project-page) for the evaluation results. From the project sidebar, click **Create new project**.
4. Package the model in VLLM-compatible format and save it as an artifact in W\&B Models. An attempt to benchmark any other type of artifact fails. For one approach, see the following [Example: Prepare a model](#example-prepare-a-model) section.
5. Review the documentation for a given benchmark to understand how it works and learn about specific requirements. For convenience, the [Available evaluation benchmarks](/models/launch/evaluations) reference includes relevant links.

## Evaluate your model

After you complete the prerequisites, follow these steps to set up and launch an evaluation job:

1. Log in to W\&B, then click **Launch** in the project sidebar. The **LLM Evaluation Jobs** page displays.
2. Click **Evaluate model checkpoint** to set up the evaluation job.
3. Select a destination project to save the evaluation results to.
4. In the **Model artifact** section, specify the project, artifact, and version of the prepared model to evaluate.
5. Click **Evaluations**, then select up to four benchmarks.
6. If you select benchmarks that use OpenAI models for scoring, the **Scorer API key** field displays. Click it, then select the `OPENAI_API_KEY` secret. For convenience, a team admin can create a secret from this drawer by clicking **Create secret**.
7. If you select benchmarks that require access to gated datasets in Hugging Face, a **Hugging Face token** field displays. [Request access to the relevant dataset](https://huggingface.co/docs/hub/en/datasets-gated#access-gated-datasets-as-a-user), then select the secret that contains the Hugging Face user access token.
8. Optional: Set **Sample limit** to a positive integer to limit the maximum number of benchmark samples to evaluate. Otherwise, the job includes all samples in the task.
9. To create a leaderboard automatically, click **Publish results to leaderboard**. The leaderboard displays all evaluations together in a workspace panel, and you can also share it in a report.
10. Click **Launch** to launch the evaluation job.
11. Click the circular arrow icon at the top of the page to open the recent run modal. Evaluation jobs appear with your other recent runs. Click the name of a finished run to open it in single-run view, or click the **Leaderboard** link to open the leaderboard directly. For details, see [View the results](#view-the-results).

<Tip>
  After you evaluate your first model, many fields are pre-filled with the most recent values when you configure your next evaluation job.
</Tip>

This example evaluation job runs two benchmarks against an artifact:

<Frame>
  <img alt="Example model checkpoint evaluation job" />
</Frame>

This example leaderboard visualizes the performance of several models together:

<Frame>
  <img alt="Example leaderboard visualizing the performance of several models against several benchmark tasks" />
</Frame>

## Review evaluation results

Review your evaluation job results in W\&B Models in the destination project's workspace.

1. Click the circular arrow icon at the top of the page to open the recent run modal, where evaluation jobs appear with other runs in the project. If the evaluation job has a leaderboard, click **Leaderboard** to open the leaderboard in full screen, or click a run name to open it in the project in single-run view.
2. View the evaluation job's traces in the **Evaluations** section of a workspace or in the **Traces** tab of the **Weave** sidebar panel.
3. Click the **Overview** tab to view detailed information about the evaluation job, including its configuration and summary metrics.
4. Click the **Logs** tab to view, search, or download the evaluation job's debug logs.
5. Click the **Files** tab to browse, view, or download the evaluation job's files, including code, log, configuration, and other output files.

## Customize a leaderboard

The leaderboard shows results for all evaluation jobs sent to a given project, with one row per benchmark per evaluation job. Columns display details like the trace, input values, and output values for the evaluation job. For more information about leaderboards, see [Leaderboards in Weave](/weave/guides/core-types/leaderboards).

<Tip>To give feedback about a result from the leaderboard, click the emoji icon or the chat icon in the **Feedback** column.</Tip>

* By default, all evaluation jobs are displayed. Filter or search for an evaluation job using the run selector at the left.
* By default, evaluation jobs are ungrouped. To group by one or more columns, click the **Group** icon. You can show or hide a group, or expand a group to view its runs.
* By default, all operations are displayed. To display only a single operation, click **All ops** and select an operation.
* To sort by a column, click the column heading. To customize the display of columns, click **Columns**.
  * By default, headers are organized in a single level. You can increase the header depth to organize related headers together.
  * Select or deselect individual columns to show or hide them, or show or hide all columns with a click.
  * Pin columns to display them before unpinned columns.

## Export a leaderboard

To export a leaderboard:

1. Click the download icon, located near the **Columns** button.
2. To optimize the export size, only the trace roots are exported by default. To export full traces, turn off **Trace roots only**.
3. To optimize the export size, feedback and costs are not exported by default. To include them in the export, toggle **Feedback** or **Costs**.
4. By default, the export is in JSONL format. To customize the format, click **Export to file** and select a format.
5. To export the leaderboard in your browser, click **Export**.
6. To export the leaderboard programmatically, select **Python** or **cURL**, then click **Copy** and run the script or command.

## Re-run an evaluation job

Depending on your situation, there are multiple ways to re-run an evaluation job or view its configuration.

* To re-run the last evaluation job again, follow the steps in [Evaluate your model](#evaluate-your-model). Select the destination project, then the model artifact details and the selected benchmarks you selected last time are populated automatically. Optionally, make adjustments, then launch the evaluation job.
* To re-run an evaluation job from the project's **Runs** tab or run selector, hover over the run name and click the play icon. The job configuration drawer displays with the settings pre-populated. Optionally adjust the settings, then click **Launch**.
* To re-run an evaluation job from a different project, import its configuration:
  1. Follow the steps in [Evaluate your model](#evaluate-your-model). After you select the destination project, click **Import configuration**.
  2. Select the project that contains the evaluation job to import, then select the evaluation job run. The job configuration drawer displays with the settings pre-populated.
  3. Optionally adjust the configuration.
  4. Click **Launch**.

## Export an evaluation job configuration

Export an evaluation job's configuration from the run's **Files** tab.

1. Open the run in single-run view.
2. Click the **Files** tab.
3. Click the download button next to `config.yaml` to download it locally.

## Example: Prepare a model

Before you can evaluate a model checkpoint, you must package it in VLLM-compatible format and save it as an artifact in W\&B Models. This example shows one way to do this:

```python lines theme={null}
import os
from transformers import AutoTokenizer, AutoModelForCausalLM

# Load your model
model_name = "your-model-name"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name)

# Save in vLLM-compatible format
save_dir = "path/to/save"
tokenizer.save_pretrained(save_dir)
model.save_pretrained(save_dir)

# Save to W&B Models
import wandb
wandb_run = wandb.init(entity="your-entity-name", project="your-project-name")
artifact = wandb.Artifact(name="your-artifact-name")
artifact.add_dir(save_dir)
logged_artifact = wandb_run.log_artifact(artifact)
logged_artifact.wait()
wandb.finish()
```


# Evaluation benchmark catalog
Source: https://docs.wandb.ai/models/launch/evaluations

Browse the evaluation benchmarks available through LLM Evaluation Jobs


<Note>
  LLM Evaluation Jobs is in **Preview** for [W\&B Multi-tenant Cloud](/platform/hosting/hosting-options/multi_tenant_cloud). Compute is free during the preview period. [Learn more](/models/launch#pricing)
</Note>

This page catalogs the evaluation benchmarks available through [LLM Evaluation Jobs](/models/launch), organized by category. Use it to discover which benchmarks you can run, identify their task IDs, and check whether a benchmark requires additional credentials.

Some benchmarks require additional credentials. A team admin must add these credentials as [team-scoped secrets](/platform/secrets#add-a-secret) before any team member can use the benchmarks in an evaluation job:

* If a benchmark has `Yes` in the **OpenAI Scorer** column, the benchmark uses OpenAI models for scoring. An organization or team admin must add an OpenAI API key as a team secret. When you configure an evaluation job with a benchmark that has this requirement, set the **Scorer API key** field to the secret.
* If a benchmark has a link in the **Gated HF Dataset** column, the benchmark requires access to a gated Hugging Face dataset. An organization or team admin must request access to the dataset in Hugging Face. The admin then creates a Hugging Face user access token and configures a team secret with that token. When you configure a benchmark with this requirement, set the **Hugging Face Token** field to the secret.

## Knowledge

Evaluate factual knowledge across domains such as science, language, and general reasoning.

| Evaluation                                                                                    | Task ID             | <div>OpenAI Scorer</div> | Gated HF Dataset | Description                                                 |
| --------------------------------------------------------------------------------------------- | ------------------- | ------------------------ | ---------------- | ----------------------------------------------------------- |
| [BoolQ](https://github.com/google-research-datasets/boolean-questions)                        | `boolq`             |                          |                  | Boolean yes/no questions from natural language queries      |
| [GPQA Diamond](https://arxiv.org/abs/2311.12022)                                              | `gpqa_diamond`      |                          |                  | Graduate-level science questions (highest quality subset)   |
| [HLE](https://arxiv.org/abs/2501.14249)                                                       | `hle`               |                          | Yes              | Human-level evaluation benchmark                            |
| [Lingoly](https://arxiv.org/abs/2406.06196)                                                   | `lingoly`           |                          | Yes              | Linguistics olympiad problems                               |
| [Lingoly Too](https://arxiv.org/abs/2503.02972)                                               | `lingoly_too`       |                          | Yes              | Extended linguistics challenge problems                     |
| [MMIU](https://arxiv.org/abs/2408.02718)                                                      | `mmiu`              |                          |                  | Massive Multitask Language Understanding benchmark          |
| [MMLU (0-shot)](https://github.com/hendrycks/test)                                            | `mmlu_0_shot`       |                          |                  | Massive Multitask Language Understanding without examples   |
| [MMLU (5-shot)](https://github.com/hendrycks/test)                                            | `mmlu_5_shot`       |                          |                  | Massive Multitask Language Understanding with 5 examples    |
| [MMLU-Pro](https://arxiv.org/abs/2406.01574)                                                  | `mmlu_pro`          |                          |                  | More challenging version of MMLU                            |
| [ONET M6](https://github.com/UKGovernmentBEIS/inspect_evals/tree/main/src/inspect_evals/onet) | `onet_m6`           |                          |                  | Occupational knowledge benchmark                            |
| [PAWS](https://github.com/google-research-datasets/paws)                                      | `paws`              |                          |                  | Paraphrase adversarial word substitution                    |
| [SevenLLM MCQ (English)](https://arxiv.org/abs/2405.03446)                                    | `sevenllm_mcq_en`   |                          |                  | Multiple choice questions in English                        |
| [SevenLLM MCQ (Chinese)](https://arxiv.org/abs/2405.03446)                                    | `sevenllm_mcq_zh`   |                          |                  | Multiple choice questions in Chinese                        |
| [SevenLLM QA (English)](https://arxiv.org/abs/2405.03446)                                     | `sevenllm_qa_en`    |                          |                  | Question answering in English                               |
| [SevenLLM QA (Chinese)](https://arxiv.org/abs/2405.03446)                                     | `sevenllm_qa_zh`    |                          |                  | Question answering in Chinese                               |
| [SimpleQA](https://openai.com/index/introducing-simpleqa/)                                    | `simpleqa`          | Yes                      |                  | Straightforward factual question answering                  |
| [SimpleQA Verified](https://openai.com/index/introducing-simpleqa/)                           | `simpleqa_verified` |                          |                  | Verified subset of SimpleQA with validated answers          |
| [WorldSense](https://github.com/facebookresearch/worldsense)                                  | `worldsense`        |                          |                  | Evaluates understanding of world knowledge and common sense |

## Reasoning

Evaluate logical thinking, problem-solving, and common-sense reasoning capabilities.

| Evaluation                                                          | Task ID          | OpenAI Scorer | Gated HF Dataset | Description                                                                 |
| ------------------------------------------------------------------- | ---------------- | ------------- | ---------------- | --------------------------------------------------------------------------- |
| [AGIE AQUA-RAT](https://arxiv.org/abs/1705.04146)                   | `agie_aqua_rat`  |               |                  | Algebraic question answering with rationales                                |
| [AGIE LogiQA (English)](https://arxiv.org/abs/2007.08124)           | `agie_logiqa_en` |               |                  | Logical reasoning questions in English                                      |
| [AGIE LSAT Analytical Reasoning](https://www.lsac.org/)             | `agie_lsat_ar`   |               |                  | LSAT analytical reasoning (logic games) problems                            |
| [AGIE LSAT Logical Reasoning](https://www.lsac.org/)                | `agie_lsat_lr`   |               |                  | LSAT logical reasoning questions                                            |
| [ARC Challenge](https://huggingface.co/datasets/allenai/ai2_arc)    | `arc_challenge`  |               |                  | Challenging science questions requiring reasoning (AI2 Reasoning Challenge) |
| [ARC Easy](https://huggingface.co/datasets/allenai/ai2_arc)         | `arc_easy`       |               |                  | Easier set of science questions from the ARC dataset                        |
| [BBH](https://github.com/suzgunmirac/BIG-Bench-Hard)                | `bbh`            |               |                  | BIG-Bench Hard: challenging tasks from BIG-Bench                            |
| [CoCoNot](https://arxiv.org/abs/2310.03697)                         | `coconot`        |               |                  | Counterfactual commonsense reasoning benchmark                              |
| [CommonsenseQA](https://huggingface.co/datasets/tau/commonsense_qa) | `commonsense_qa` |               |                  | Commonsense reasoning questions                                             |
| [HellaSwag](https://arxiv.org/abs/1905.07830)                       | `hellaswag`      |               |                  | Commonsense natural language inference                                      |
| [MUSR](https://arxiv.org/abs/2310.16049)                            | `musr`           |               |                  | Multi-step reasoning benchmark                                              |
| [PIQA](https://yonatanbisk.com/piqa/)                               | `piqa`           |               |                  | Physical commonsense reasoning                                              |
| [WinoGrande](https://winogrande.allenai.org/)                       | `winogrande`     |               |                  | Commonsense reasoning via pronoun resolution                                |

## Math

Evaluate mathematical problem-solving across difficulty levels, from grade school to competition-level problems.

| Evaluation                                                                              | Task ID                    | OpenAI Scorer | Gated HF Dataset | Description                                                      |
| --------------------------------------------------------------------------------------- | -------------------------- | ------------- | ---------------- | ---------------------------------------------------------------- |
| [AGIE Math](https://arxiv.org/abs/2410.12211)                                           | `agie_math`                |               |                  | Advanced mathematical reasoning from AGIE benchmark suite        |
| [AGIE SAT Math](https://satsuite.collegeboard.org/sat)                                  | `agie_sat_math`            |               |                  | SAT mathematics questions                                        |
| [AIME 2024](https://artofproblemsolving.com/wiki/index.php/AIME_Problems_and_Solutions) | `aime2024`                 |               |                  | American Invitational Mathematics Examination problems from 2024 |
| [AIME 2025](https://artofproblemsolving.com/wiki/index.php/AIME_Problems_and_Solutions) | `aime2025`                 |               |                  | American Invitational Mathematics Examination problems from 2025 |
| [GSM8K](https://github.com/openai/grade-school-math)                                    | `gsm8k`                    |               |                  | Grade School Math 8K: multi-step math word problems              |
| [InfiniteBench Math Calc](https://arxiv.org/abs/2402.13718)                             | `infinite_bench_math_calc` |               |                  | Mathematical calculations in long contexts                       |
| [InfiniteBench Math Find](https://arxiv.org/abs/2402.13718)                             | `infinite_bench_math_find` |               |                  | Finding mathematical patterns in long contexts                   |
| [MATH](https://github.com/hendrycks/math)                                               | `math`                     |               |                  | Competition-level mathematics problems                           |
| [MGSM](https://github.com/google-research/url-nlp/tree/main/mgsm)                       | `mgsm`                     |               |                  | Multilingual Grade School Math                                   |

## Code

Evaluate programming and software development capabilities such as debugging, code execution prediction, and function calling.

| Evaluation                                                                                 | Task ID                     | OpenAI Scorer | Gated HF Dataset | Description                                                                             |
| ------------------------------------------------------------------------------------------ | --------------------------- | ------------- | ---------------- | --------------------------------------------------------------------------------------- |
| [BFCL](https://gorilla.cs.berkeley.edu/blogs/8_berkeley_function_calling_leaderboard.html) | `bfcl`                      |               |                  | Berkeley Function Calling Leaderboard: tests function calling and tool use capabilities |
| [InfiniteBench Code Debug](https://arxiv.org/abs/2402.13718)                               | `infinite_bench_code_debug` |               |                  | Long-context code debugging tasks                                                       |
| [InfiniteBench Code Run](https://arxiv.org/abs/2402.13718)                                 | `infinite_bench_code_run`   |               |                  | Long-context code execution prediction                                                  |

## Reading

Evaluate reading comprehension and information extraction from complex texts.

| Evaluation                                                                                                  | Task ID                       | OpenAI Scorer | Gated HF Dataset | Description                                                                              |
| ----------------------------------------------------------------------------------------------------------- | ----------------------------- | ------------- | ---------------- | ---------------------------------------------------------------------------------------- |
| [AGIE LSAT Reading Comprehension](https://www.lsac.org/)                                                    | `agie_lsat_rc`                |               |                  | LSAT reading comprehension passages and questions                                        |
| [AGIE SAT English](https://satsuite.collegeboard.org/sat)                                                   | `agie_sat_en`                 |               |                  | SAT reading and writing questions with passages                                          |
| [AGIE SAT English (No Passage)](https://satsuite.collegeboard.org/sat)                                      | `agie_sat_en_without_passage` |               |                  | SAT English questions without accompanying passages                                      |
| [DROP](https://github.com/allenai/allennlp-reading-comprehension/blob/master/allennlp_rc/eval/drop_eval.py) | `drop`                        |               |                  | Discrete Reasoning Over Paragraphs: reading comprehension requiring numerical reasoning  |
| [RACE-H](https://www.cs.cmu.edu/~glai1/data/race/)                                                          | `race_h`                      |               |                  | Reading comprehension from English exams (high difficulty)                               |
| [SQuAD](https://rajpurkar.github.io/SQuAD-explorer/)                                                        | `squad`                       |               |                  | Stanford Question Answering Dataset: extractive question answering on Wikipedia articles |

## Long context

Evaluate the ability to process and reason over extended contexts, including retrieval and pattern recognition.

| Evaluation                                                                  | Task ID                              | OpenAI Scorer | Gated HF Dataset | Description                                       |
| --------------------------------------------------------------------------- | ------------------------------------ | ------------- | ---------------- | ------------------------------------------------- |
| [InfiniteBench KV Retrieval](https://arxiv.org/abs/2402.13718)              | `infinite_bench_kv_retrieval`        |               |                  | Key-value retrieval in long contexts              |
| [InfiniteBench LongBook (English)](https://arxiv.org/abs/2402.13718)        | `infinite_bench_longbook_choice_eng` |               |                  | Multiple choice questions on long books           |
| [InfiniteBench LongDialogue QA (English)](https://arxiv.org/abs/2402.13718) | `infinite_bench_longdialogue_qa_eng` |               |                  | Question answering over long dialogues            |
| [InfiniteBench Number String](https://arxiv.org/abs/2402.13718)             | `infinite_bench_number_string`       |               |                  | Number pattern recognition in long sequences      |
| [InfiniteBench Passkey](https://arxiv.org/abs/2402.13718)                   | `infinite_bench_passkey`             |               |                  | Retrieval of information from long context        |
| [NIAH](https://arxiv.org/abs/2406.07230)                                    | `niah`                               |               |                  | Needle in a Haystack: long-context retrieval test |

## Safety

Evaluate alignment, bias detection, harmful content resistance, and truthfulness.

| Evaluation                                                                                                   | Task ID                         | OpenAI Scorer | Gated HF Dataset | Description                                                           |
| ------------------------------------------------------------------------------------------------------------ | ------------------------------- | ------------- | ---------------- | --------------------------------------------------------------------- |
| [AgentHarm](https://arxiv.org/abs/2410.09024)                                                                | `agentharm`                     | Yes           |                  | Tests model resistance to harmful agent behavior and misuse scenarios |
| [AgentHarm Benign](https://arxiv.org/abs/2410.09024)                                                         | `agentharm_benign`              | Yes           |                  | Benign baseline for AgentHarm to measure false positive rates         |
| [Agentic Misalignment](https://arxiv.org/abs/2510.05179)                                                     | `agentic_misalignment`          |               |                  | Evaluates potential misalignment in agentic behavior                  |
| [AHB](https://arxiv.org/abs/2503.04804)                                                                      | `ahb`                           |               |                  | Agent Harmful Behavior: tests resistance to harmful agentic actions   |
| [AIRBench](https://arxiv.org/abs/2410.02407)                                                                 | `air_bench`                     |               |                  | Tests adversarial instruction resistance                              |
| [BBEH](https://arxiv.org/abs/2502.19187)                                                                     | `bbeh`                          |               |                  | Bias Benchmark for Evaluating Harmful behavior                        |
| [BBEH Mini](https://arxiv.org/abs/2502.19187)                                                                | `bbeh_mini`                     |               |                  | Smaller version of BBEH benchmark                                     |
| [BBQ](https://arxiv.org/abs/2110.08193)                                                                      | `bbq`                           |               |                  | Bias Benchmark for Question Answering                                 |
| [BOLD](https://arxiv.org/abs/2101.11718)                                                                     | `bold`                          |               |                  | Bias in Open-Ended Language Generation Dataset                        |
| [CYSE3 Visual Prompt Injection](https://arxiv.org/abs/2408.01605)                                            | `cyse3_visual_prompt_injection` |               |                  | Tests resistance to visual prompt injection attacks                   |
| [Make Me Pay](https://arxiv.org/abs/2410.08691)                                                              | `make_me_pay`                   |               |                  | Tests resistance to financial scam and fraud scenarios                |
| [MASK](https://arxiv.org/abs/2503.03750)                                                                     | `mask`                          | Yes           | Yes              | Tests model's handling of sensitive information                       |
| [Personality BFI](https://github.com/UKGovernmentBEIS/inspect_evals/tree/main/src/inspect_evals/personality) | `personality_BFI`               |               |                  | Big Five personality trait assessment                                 |
| [Personality TRAIT](https://arxiv.org/abs/2406.14703)                                                        | `personality_TRAIT`             |               | Yes              | Comprehensive personality trait evaluation                            |
| SOSBench                                                                                                     | `sosbench`                      | Yes           |                  | Safety and oversight stress test                                      |
| [StereoSet](https://github.com/moinnadeem/StereoSet)                                                         | `stereoset`                     |               |                  | Measures stereotypical biases in language models                      |
| [StrongREJECT](https://arxiv.org/abs/2402.10260)                                                             | `strong_reject`                 |               |                  | Tests model's ability to reject harmful requests                      |
| [Sycophancy](https://arxiv.org/abs/2310.13548)                                                               | `sycophancy`                    |               |                  | Evaluates tendency toward sycophantic behavior                        |
| [TruthfulQA](https://github.com/sylinrl/TruthfulQA)                                                          | `truthfulqa`                    |               |                  | Tests model truthfulness and resistance to falsehoods                 |
| [UCCB](https://huggingface.co/datasets/CraneAILabs/UCCB)                                                     | `uccb`                          |               |                  | Unsafe Content Classification Benchmark                               |
| [WMDP Bio](https://www.wmdp.ai/)                                                                             | `wmdp_bio`                      |               |                  | Tests hazardous knowledge in biology                                  |
| [WMDP Chem](https://www.wmdp.ai/)                                                                            | `wmdp_chem`                     |               |                  | Tests hazardous knowledge in chemistry                                |
| [WMDP Cyber](https://www.wmdp.ai/)                                                                           | `wmdp_cyber`                    |               |                  | Tests hazardous knowledge in cybersecurity                            |
| [XSTest](https://arxiv.org/abs/2308.01263)                                                                   | `xstest`                        | Yes           |                  | Exaggerated safety test for over-refusal detection                    |

## Domain-specific

Evaluate specialized knowledge in medicine, chemistry, law, biology, and other professional fields.

| Evaluation                                                     | Task ID                       | OpenAI Scorer | Gated HF Dataset | Description                                           |
| -------------------------------------------------------------- | ----------------------------- | ------------- | ---------------- | ----------------------------------------------------- |
| [ChemBench](https://arxiv.org/abs/2404.01475)                  | `chembench`                   |               |                  | Chemistry knowledge and problem-solving benchmark     |
| [HealthBench](https://arxiv.org/abs/2406.09746)                | `healthbench`                 | Yes           |                  | Healthcare and medical knowledge evaluation           |
| [HealthBench Consensus](https://arxiv.org/abs/2406.09746)      | `healthbench_consensus`       | Yes           |                  | Healthcare questions with expert consensus            |
| [HealthBench Hard](https://arxiv.org/abs/2406.09746)           | `healthbench_hard`            | Yes           |                  | Challenging healthcare scenarios                      |
| [LabBench Cloning Scenarios](https://arxiv.org/abs/2407.10362) | `lab_bench_cloning_scenarios` |               |                  | Laboratory experiment planning and cloning            |
| [LabBench DBQA](https://arxiv.org/abs/2407.10362)              | `lab_bench_dbqa`              |               |                  | Database question answering for lab scenarios         |
| [LabBench FigQA](https://arxiv.org/abs/2407.10362)             | `lab_bench_figqa`             |               |                  | Figure interpretation in scientific contexts          |
| [LabBench LitQA](https://arxiv.org/abs/2407.10362)             | `lab_bench_litqa`             |               |                  | Literature-based question answering for research      |
| [LabBench ProtocolQA](https://arxiv.org/abs/2407.10362)        | `lab_bench_protocolqa`        |               |                  | Laboratory protocol understanding                     |
| [LabBench SeqQA](https://arxiv.org/abs/2407.10362)             | `lab_bench_seqqa`             |               |                  | Biological sequence analysis questions                |
| [LabBench SuppQA](https://arxiv.org/abs/2407.10362)            | `lab_bench_suppqa`            |               |                  | Supplementary material interpretation                 |
| [LabBench TableQA](https://arxiv.org/abs/2407.10362)           | `lab_bench_tableqa`           |               |                  | Table interpretation in scientific papers             |
| [MedQA](https://github.com/jind11/MedQA)                       | `medqa`                       |               |                  | Medical licensing exam questions                      |
| [PubMedQA](https://pubmedqa.github.io/)                        | `pubmedqa`                    |               |                  | Biomedical question answering from research abstracts |
| [SEC-QA v1](https://arxiv.org/abs/2406.14806)                  | `sec_qa_v1`                   |               |                  | SEC filing question answering                         |
| [SEC-QA v1 (5-shot)](https://arxiv.org/abs/2406.14806)         | `sec_qa_v1_5_shot`            |               |                  | SEC-QA with 5 examples                                |
| [SEC-QA v2](https://arxiv.org/abs/2406.14806)                  | `sec_qa_v2`                   |               |                  | Updated SEC filing benchmark                          |
| [SEC-QA v2 (5-shot)](https://arxiv.org/abs/2406.14806)         | `sec_qa_v2_5_shot`            |               |                  | SEC-QA v2 with 5 examples                             |

## Multimodal

Evaluate vision and language understanding combining visual and textual inputs.

| Evaluation                                                              | Task ID                                      | OpenAI Scorer | Gated HF Dataset | Description                                                           |
| ----------------------------------------------------------------------- | -------------------------------------------- | ------------- | ---------------- | --------------------------------------------------------------------- |
| [DocVQA](https://www.docvqa.org/)                                       | `docvqa`                                     |               |                  | Document Visual Question Answering: questions about document images   |
| [MathVista](https://mathvista.github.io/)                               | `mathvista`                                  |               |                  | Mathematical reasoning with visual contexts combining vision and math |
| [MMMU Multiple Choice](https://mmmu-benchmark.github.io/)               | `mmmu_multiple_choice`                       |               |                  | Multimodal understanding with multiple choice format                  |
| [MMMU Open](https://mmmu-benchmark.github.io/)                          | `mmmu_open`                                  |               |                  | Multimodal understanding with open-ended responses                    |
| [V\*Star Bench Attribute Recognition](https://arxiv.org/abs/2411.10006) | `vstar_bench_attribute_recognition`          |               |                  | Visual attribute recognition tasks                                    |
| [V\*Star Bench Spatial Relationship](https://arxiv.org/abs/2411.10006)  | `vstar_bench_spatial_relationship_reasoning` |               |                  | Spatial reasoning with visual inputs                                  |

## Instruction following

Evaluate adherence to specific instructions and formatting requirements.

| Evaluation                                 | Task ID  | OpenAI Scorer | Gated HF Dataset | Description                                      |
| ------------------------------------------ | -------- | ------------- | ---------------- | ------------------------------------------------ |
| [IFEval](https://arxiv.org/abs/2311.07911) | `ifeval` |               |                  | Tests precise instruction-following capabilities |

## System

Basic system validation and pre-flight checks.

| Evaluation                                                                                 | Task ID      | OpenAI Scorer | Gated HF Dataset | Description                            |
| ------------------------------------------------------------------------------------------ | ------------ | ------------- | ---------------- | -------------------------------------- |
| [Pre-Flight](https://ukgovernmentbeis.github.io/inspect_evals/evals/knowledge/pre_flight/) | `pre_flight` |               |                  | Basic system check and validation test |

## Next steps

* [Evaluate a model checkpoint](/models/launch/evaluate-model-checkpoint)
* [Evaluate a hosted API model](/models/launch/evaluate-hosted-model)
* View details about specific benchmarks at [AISI Inspect Evals](https://inspect.aisi.org.uk/evals/)


# Registry overview
Source: https://docs.wandb.ai/models/registry

Use W&B Registry to manage and share artifact versions across your organization

<ColabLink />

W\&B Registry is a curated central repository of [W\&B Artifact versions](/models/artifacts/create-a-new-artifact-version) within your organization. Users who [have permission](/models/registry/configure_registry/) within your organization can [download and use artifacts](/models/registry/download_use_artifact/), share, and collaboratively manage the lifecycle of all artifacts, regardless of the team that the user belongs to.

Use the Registry to track artifact versions, audit the history of an artifact's usage and changes, ensure governance and compliance of your artifacts, and [automate downstream processes such as model CI/CD](/models/automations/).

In summary, use W\&B Registry to:

* [Promote](/models/registry/link_version/) artifact versions that satisfy a machine learning task to other users in your organization.
* Organize [artifacts with tags](/models/registry/organize-with-tags/) so that you can find or reference specific artifacts.
* Track an [artifact’s lineage](/models/registry/lineage/) and audit the history of changes.
* [Automate](/models/automations/) downstream processes such as model CI/CD.
* [Manage who in your organization](/models/registry/configure_registry/) can access artifacts in each registry.

The following image shows the W\&B Registry landing page. A registry called `Model` is starred. Two collections are shown `DemoModels` and `Zoo_Classifier_Models`.

<Frame>
  <img alt="W&B Registry" />
</Frame>

## Learn the basics

Each organization initially contains two registries that you can use to organize your model and dataset artifacts called **Models** and **Datasets**, respectively. You can create [additional registries to organize other artifact types based on your organization's needs](/models/registry/create_registry).

Each [*registry*](/models/registry/configure_registry/) consists of one or more [*collections*](/models/registry/create_collection/). Each collection represents a distinct task or use case.

To add an artifact to a registry, you first log a [specific artifact version to W\&B](/models/artifacts/create-a-new-artifact-version/). Each time you log an artifact, W\&B automatically assigns a version to that artifact. Artifact versions use 0 indexing, so the first version is `v0`, the second version is `v1`, and so on.

Once you log an artifact to W\&B, you can then link that specific artifact version to a collection in the registry.

<Note>
  The term "link" refers to pointers that connect where W\&B stores the artifact and where the artifact is accessible in the registry. W\&B does not duplicate artifacts when you link an artifact to a collection.
</Note>

As an example, the following code example logs and links a model artifact called `"my_model.txt"` to a collection named `"first-collection"` within a registry called `"model"`:

1. Initialize a W\&B Run with `wandb.init()`.
2. Log the artifact to W\&B with `wandb.Run.log()`.
3. Specify the name of the collection and registry to link your artifact version to.
4. Link the artifact to the collection using `wandb.Run.link_artifact()`.

Save this Python code to a script and run it. W\&B Python SDK version 0.18.6 or newer is required.

```python title="hello_collection.py" theme={null}
import wandb
import random

# Initialize a W&B Run to track the artifact
with wandb.init(project="registry_quickstart") as run:
    # Create a simulated model file so that you can log it
    with open("my_model.txt", "w") as f:
        f.write("Model: " + str(random.random()))

    # Log the artifact to W&B
    logged_artifact = run.log_artifact(
        artifact_or_path="./my_model.txt", 
        name="gemma-finetuned", 
        type="model" # Specifies artifact type
    )

    # Specify the name of the collection and registry
    # you want to publish the artifact to
    COLLECTION_NAME = "first-collection"
    REGISTRY_NAME = "model"

    # Link the artifact to the registry
    run.link_artifact(
        artifact=logged_artifact, 
        target_path=f"wandb-registry-{REGISTRY_NAME}/{COLLECTION_NAME}"
    )
```

W\&B automatically creates a collection for you if the collection you specify in the returned run object's `wandb.Run.link_artifact(target_path = "")` method does not exist within the registry you specify.

Continuing from the previous example, after you run the script, navigate to W\&B Registry to view artifact versions that you and other members of your organization publish. Select **Registry** in the project sidebar below **Applications**. Select the `"Model"` registry. Within the registry, you should see the `"first-collection"` collection with your linked artifact version.

Once you link an artifact version to a collection within a registry, members of your organization can [view](/models/registry/lineage), [download](/models/registry/download_use_artifact), [organize](/models/registry/organize-with-tags), and manage your artifact versions, create downstream automations, and more if they have the proper permissions.

<Note>
  If an artifact version logs metrics (such as by using `wandb.Run.log_artifact()`), you can view metrics for that version from its details page, and you can compare metrics across artifact versions from the collection's page. Refer to [View linked artifacts in a registry](/models/registry/link_version/#view-linked-artifacts-in-a-registry).
</Note>

## Enable W\&B Registry

Based on your deployment type, satisfy the following conditions to enable W\&B Registry:

| Deployment type    | How to enable                                                                                                                                                            |
| ------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| Multi-tenant Cloud | No action required. W\&B Registry is available on the W\&B App.                                                                                                          |
| Dedicated Cloud    | Contact your account team to enable W\&B Registry for your deployment.                                                                                                   |
| Self-Managed       | For Server v0.71.0 or newer, no action required. Registry is enabled by default. For older supported Server versions, contact your W\&B account team to enable Registry. |

## Resources to get started

Depending on your use case, explore the following resources to get started with the W\&B Registry:

* Check out the tutorial video:
  * [Getting started with Registry from W\&B](https://www.youtube.com/watch?v=p4XkVOsjIeM)
* Take the W\&B [Model CI/CD](https://www.wandb.courses/courses/enterprise-model-management) course and learn how to:
  * Use W\&B Registry to manage and version your artifacts, track lineage, and promote models through different lifecycle stages.
  * Automate your model management workflows using webhooks.
  * Integrate the registry with external ML systems and tools for model evaluation, monitoring, and deployment.


# Reference an artifact version with aliases
Source: https://docs.wandb.ai/models/registry/aliases

Use default, custom, and protected aliases to reference specific artifact versions in W&B Registry.

Reference a specific [artifact version](/models/artifacts/create-a-new-artifact-version/) with one or more aliases. [W\&B automatically assigns aliases](/models/registry/aliases/#default-aliases) to each artifact you link with the same name. You can also [create one or more custom aliases](/models/registry/aliases/#custom-aliases) to reference a specific artifact version.

Aliases appear as rectangles with the name of that alias in the rectangle in the Registry UI. If an [alias is protected](/models/registry/aliases/#protected-aliases), it appears as a gray rectangle with a lock icon. Otherwise, the alias appears as an orange rectangle. Aliases are not shared across registries.

<Note>
  **When to use an alias versus using a tag**

  Use an alias to reference a specific artifact version. Each alias within a collection is unique. Only one artifact version can have a specific alias at a time.

  Use tags to organize and group artifact versions or collections based on a common theme. Multiple artifact versions and collections can share the same tag.
</Note>

When you add an alias to an artifact version, you can optionally start a [Registry automation](/models/automations/automation-events/#registry) to notify a Slack channel or trigger a webhook. If the automation calls a webhook that needs an access token or other sensitive value in the request, store those strings as [team secrets](/platform/secrets) and select them when you configure the webhook for the automation.

## Default aliases

W\&B automatically assigns the following aliases to each artifact version you link with the same name:

* The `latest` alias to the most recent artifact version you link to a collection.
* A unique version number. W\&B counts each artifact version (zero indexing) you link. W\&B uses the count number to assign a unique version number to that artifact.

For example, if you link an artifact named `zoo_model` three times, W\&B creates three aliases `v0`, `v1`, and `v2` respectively. `v2` also has the `latest` alias.

## Custom aliases

Create one or more custom aliases for a specific artifact versions based on your unique use case. For example:

* You might use aliases such as `dataset_version_v0`, `dataset_version_v1`, and `dataset_version_v2` to identify which dataset a model was trained on.
* You might use a `best_model` alias to keep track of the best performing artifact model version.

Any user with a [**Member** or **Admin** registry role](/models/registry/configure_registry/#registry-roles) on a registry can add or remove a custom alias from a linked artifact in that registry. Users with the [**Restricted Viewer** or **Viewer** roles](/models/registry/configure_registry/#registry-roles) cannot add or remove aliases.

<Note>
  [Protected aliases](/models/registry/aliases/#protected-aliases) provide a way to label and identify which artifact versions to protect from modification or deletion.
</Note>

You can create a custom alias with the W\&B Registry or the Python SDK. Based on your use case, click on a tab below that best fits your needs.

<Tabs>
  <Tab title="W&B Registry">
    1. Navigate to the W\&B Registry.
    2. Click the **View details** button in a collection.
    3. Within the **Versions** section, click the **View** button for a specific artifact version.
    4. Click the **+** button to add one or more aliases next to the **Aliases** field.
  </Tab>

  <Tab title="Python SDK">
    When you link an artifact version to a collection with the Python SDK you can optionally provide a list of one or more aliases as an argument to the `alias` parameter in [`link_artifact()`](/models/ref/python/experiments/run.md/#link_artifact). W\&B creates an alias ([non protected alias](#custom-aliases)) for you if the alias you provide does not already exist.

    The following code snippet demonstrates how to link an artifact version to a collection and add aliases to that artifact version with the Python SDK. Replace values within `<>` with your own:

    ```python theme={null}
    import wandb

    # Initialize a run
    with wandb.init(entity = "<team_entity>", project = "<project_name>") as run:

        # Create an artifact object
        # The type parameter specifies both the type of the 
        # artifact object and the collection type
        artifact = wandb.Artifact(name = "<name>", type = "<type>")

        # Add the file to the artifact object. 
        # Specify the path to the file on your local machine.
        artifact.add_file(local_path = "<local_path_to_artifact>")

        # Specify the collection and registry to link the artifact to
        REGISTRY_NAME = "<registry_name>"
        COLLECTION_NAME = "<collection_name>"
        target_path=f"wandb-registry-{REGISTRY_NAME}/{COLLECTION_NAME}"

        # Link the artifact version to the collection
        # Add one or more aliases to this artifact version
        run.link_artifact(
            artifact = artifact, 
            target_path = target_path, 
            aliases = ["<alias_1>", "<alias_2>"]
            )
    ```
  </Tab>
</Tabs>

### Protected aliases

Use a [protected alias](/models/registry/aliases/#protected-aliases) to both label and identify artifact versions that should not be modified or deleted. For example, consider using a `production` protected alias to label and identify artifact versions that are in used in your organization's machine learning production pipeline.

[Registry admin](/models/registry/configure_registry/#registry-roles) users and [service accounts](/support/models/articles/what-is-a-service-account-and-why-is-it-) with the **Admin** role can create protected aliases and add or remove protected aliases from an artifact version. Users and service accounts with **Member**, **Viewer**, and **Restricted Viewer** roles cannot unlink a protected version or delete a collection that contains a protected alias. See [Configure registry access](/models/registry/configure_registry/) for details.

Common protected aliases include:

* **Production**: The artifact version is ready for production use.
* **Staging**: The artifact version is ready for testing.

#### Create a protected alias

The following steps describe how to create a protected alias in the W\&B Registry UI:

1. Navigate to the W\&B Registry.
2. Select a registry.
3. Click the gear button on the top right of the page to view the registry's settings.
4. Within the **Protected Aliases** section, click the **+** button to add one or more protected aliases.

After creation, each protected alias appears as a gray rectangle with a lock icon in the **Protected Aliases** section.

<Note>
  Unlike custom aliases that are not protected, creating protected aliases is available exclusively in the W\&B Registry UI and not programmatically with the Python SDK. To add a protected alias to an artifact version, you can use the W\&B Registry UI or the Python SDK.
</Note>

The following steps describe how to add a protected alias to an artifact version with the W\&B Registry UI:

1. Navigate to the W\&B Registry.
2. Click the **View details** button in a collection.
3. Within the **Versions** section, select the **View** button for a specific artifact version.
4. Click the **+** button to add one or more protected aliases next to the **Aliases** field.

After a protected alias is created, an admin can add it to an artifact version programmatically with the Python SDK. See the W\&B Registry and Python SDK tabs in [Create a custom alias](#custom-aliases) section above for an example on how to add a protected alias to an artifact version.

## Find existing aliases

You can find existing aliases with the [global search bar in the W\&B Registry](/models/registry/search_registry/#search-for-registry-items). To find a protected alias:

1. Navigate to the W\&B Registry.
2. Specify the search term in the search bar at the top of the page. Press Enter to search.

Search results appear below the search bar if the term you specify matches an existing registry, collection name, artifact version tag, collection tag, or alias.

## Example

<Note>
  The following code example is a continuation of this [W\&B Registry Tutorial notebook](https://colab.research.google.com/github/wandb/examples/blob/master/colabs/wandb_registry/zoo_wandb.ipynb). To use the following code, you must first [retrieve and process the Zoo dataset as described in the notebook](https://colab.research.google.com/github/wandb/examples/blob/master/colabs/wandb_registry/zoo_wandb.ipynb#scrollTo=87fecd29-8146-41e2-86fb-0bb4e3e3350a). Once you have the Zoo dataset, you can create an artifact version and add custom aliases to it.
</Note>

The following code snippet shows how to create an artifact version and add custom aliases to it. The example uses the Zoo dataset from the [UCI Machine Learning Repository](https://archive.ics.uci.edu/dataset/111/zoo) and the `Model` collection in the `Zoo_Classifier_Models` registry.

```python theme={null}
import wandb

# Initialize a run
with wandb.init(entity = "smle-reg-team-2", project = "zoo_experiment") as run:

    # Create an artifact object
    # The type parameter specifies both the type of the 
    # artifact object and the collection type
    artifact = wandb.Artifact(name = "zoo_dataset", type = "dataset")

    # Add the file to the artifact object. 
    # Specify the path to the file on your local machine.
    artifact.add_file(local_path="zoo_dataset.pt", name="zoo_dataset")
    artifact.add_file(local_path="zoo_labels.pt", name="zoo_labels")

    # Specify the collection and registry to link the artifact to
    REGISTRY_NAME = "Model"
    COLLECTION_NAME = "Zoo_Classifier_Models"
    target_path=f"wandb-registry-{REGISTRY_NAME}/{COLLECTION_NAME}"

    # Link the artifact version to the collection
    # Add one or more aliases to this artifact version
    run.link_artifact(
        artifact = artifact,
        target_path = target_path,
        aliases = ["production-us", "production-eu"]
        )
```

1. First, you create an artifact object (`wandb.Artifact()`).
2. Next, you add two dataset PyTorch tensors to the artifact object with `wandb.Artifact.add_file()`.
3. Lastly, you link the artifact version to the `Model` collection in the `Zoo_Classifier_Models` registry with `link_artifact()`. You also add two custom aliases to the artifact version by passing  `production-us` and `production-eu` as arguments to the `aliases` parameter.


# Configure registry access
Source: https://docs.wandb.ai/models/registry/configure_registry

Configure W&B Registry access by managing users and teams, assigning roles, and setting role-based permissions.

A registry admin can [configure registry roles](/models/registry/configure_registry/#configure-registry-roles), [add users](/models/registry/configure_registry/#add-a-user-or-a-team-to-a-registry), or [remove users](/models/registry/configure_registry/#remove-a-user-or-team-from-a-registry) from a registry by configuring the registry's settings.

## Manage users

### Add a user or a team

Registry admins can add individual users or entire teams to a registry. To add a user or team to a registry:

1. Navigate to the W\&B Registry.
2. Select the registry you want to add a user or team to.
3. Click on the gear icon on the upper right hand corner to access the registry settings.
4. In the **Registry access** section, click **Add access**.
5. Specify one or more user names, emails, or the team names to the **Include users and teams** field.
6. Click **Add access**.

<Frame>
  <img alt="Adding teams to registry" />
</Frame>

Learn more about [configuring user roles in a registry](/models/registry/configure_registry/#configure-registry-roles), or [registry role permissions](/models/registry/configure_registry/#registry-role-permissions) .

### Remove a user or team

A registry admin can remove individual users or entire teams from a registry. To remove a user or team from a registry:

1. Navigate to the W\&B Registry at [https://wandb.ai/registry/](https://wandb.ai/registry/).
2. Select the registry you want to remove a user from.
3. Click on the gear icon on the upper right hand corner to access the registry settings.
4. Navigate to the **Registry access** section and type in the username, email, or team you want to remove.
5. Click the **Delete** button.

<Note>
  Removing a user from a team also removes that user's access to the registry.
</Note>

### Change the owner of a registry

A registry admin can designate any member as a registry's owner, including a **Restricted Viewer** or a **Viewer**. Registry ownership is primarily for accountability purposes and does not confer any additional permissions beyond those granted by the user's assigned role.

To change the owner:

1. Navigate to the W\&B Registry at [https://wandb.ai/registry/](https://wandb.ai/registry/).
2. Select the registry you want to configure.
3. Click the gear icon on the upper right hand corner.
4. Scroll to the **Registry members and roles** section.
5. Hover over the row for a member.
6. Click the **action (<Icon icon="ellipsis" />)** menu at the end of the row, then click **Make owner**.

## Configure Registry roles

This section shows how to configure roles for Registry members. For more information about Registry roles, including the cabilities of each role, order of precedence, defaults, and more, see [Details about Registry roles](#details-about-registry-roles).

1. Navigate to the W\&B Registry at [https://wandb.ai/registry/](https://wandb.ai/registry/).
2. Select the registry you want to configure.
3. Click the gear icon on the upper right hand corner.
4. Scroll to the **Registry members and roles** section.
5. Within the **Member** field, search for the user or team you want to edit permissions for.
6. In the **Registry role** column, click the user's role.
7. From the dropdown, select the role you want to assign to the user.

## Details about Registry roles

The following sections give more information about Registry roles.

<Note>
  Your [role in a team](/platform/app/settings-page/teams/#team-roles-and-permissions) has no impact or relationship to your role in any registry.
</Note>

### Default roles

W\&B automatically assigns a default **registry role** to a user or team when they are added to a registry. This role determines what they can do in that registry.

| Entity                              | Default registry role<br />(Dedicated Cloud / Self-Managed)                 | Default registry role<br />(Multi-tenant Cloud)            |
| ----------------------------------- | --------------------------------------------------------------------------- | ---------------------------------------------------------- |
| Team                                | Restricted Viewer (Server v0.75.0+) <br />Viewer (Server v0.74.x and below) | Restricted Viewer                                          |
| User or service account (non admin) | Restricted Viewer (Server v0.75.0+) <br />Viewer (Server v0.74.x and below) | Restricted Viewer                                          |
| Service account (non admin)         | Member<sup><a href="#service_account_footnote">1</a></sup>                  | Member<sup><a href="#service_account_footnote">1</a></sup> |
| Org admin                           | Admin                                                                       | Admin                                                      |

<a>1</a>: Service accounts cannot have **Viewer** or **Restricted Viewer** roles.

A registry admin can assign or modify roles for users and teams in the registry.
See [Configure user roles in a registry](/models/registry/configure_registry/#configure-registry-roles) for more information.

### Role permissions

The following table lists each Registry role, along with the permissions provided by each role:

| Permission                                                                                           | Permission Group | Restricted Viewer<br />(Multi-tenant Cloud, by invitation) | Viewer | Member | Admin |
| ---------------------------------------------------------------------------------------------------- | ---------------- | ---------------------------------------------------------- | :----: | :----: | :---: |
| View a collection's details                                                                          | Read             | ✓                                                          |    ✓   |    ✓   |   ✓   |
| View a linked artifact's details                                                                     | Read             | ✓                                                          |    ✓   |    ✓   |   ✓   |
| Usage: Consume an artifact in a registry with use\_artifact                                          | Read             |                                                            |    ✓   |    ✓   |   ✓   |
| Download a linked artifact                                                                           | Read             |                                                            |    ✓   |    ✓   |   ✓   |
| Download files from an artifact's file viewer                                                        | Read             |                                                            |    ✓   |    ✓   |   ✓   |
| Search a registry                                                                                    | Read             | ✓                                                          |    ✓   |    ✓   |   ✓   |
| View a registry's settings and user list                                                             | Read             | ✓                                                          |    ✓   |    ✓   |   ✓   |
| Create a new automation for a collection                                                             | Create           |                                                            |        |    ✓   |   ✓   |
| Turn on Slack notifications for new version being added                                              | Create           |                                                            |        |    ✓   |   ✓   |
| Create a new collection                                                                              | Create           |                                                            |        |    ✓   |   ✓   |
| Create a new registry                                                                                | Create           |                                                            |        |    ✓   |   ✓   |
| Edit collection card (description)                                                                   | Update           |                                                            |        |    ✓   |   ✓   |
| Edit linked artifact description                                                                     | Update           |                                                            |        |    ✓   |   ✓   |
| Add or delete a collection's tag                                                                     | Update           |                                                            |        |    ✓   |   ✓   |
| Add or delete an alias from a linked artifact                                                        | Update           |                                                            |        |    ✓   |   ✓   |
| Add or delete a [protected alias](/models/registry/aliases#protected-aliases) from a linked artifact | Update           |                                                            |        |        |   ✓   |
| Create or delete a [protected alias](/models/registry/aliases#protected-aliases)                     | Update           |                                                            |        |        |   ✓   |
| Link a new artifact                                                                                  | Update           |                                                            |        |    ✓   |   ✓   |
| Edit allowed types list for a registry                                                               | Update           |                                                            |        |    ✓   |   ✓   |
| Edit registry name                                                                                   | Update           |                                                            |        |    ✓   |   ✓   |
| Delete a collection                                                                                  | Delete           |                                                            |        |    ✓   |   ✓   |
| Delete an automation                                                                                 | Delete           |                                                            |        |    ✓   |   ✓   |
| Unlink an artifact from a registry                                                                   | Delete           |                                                            |        |    ✓   |   ✓   |
| Edit accepted artifact types for a registry                                                          | Admin            |                                                            |        |        |   ✓   |
| Change registry visibility (Organization or Restricted)                                              | Admin            |                                                            |        |        |   ✓   |
| Add users to a registry                                                                              | Admin            |                                                            |        |        |   ✓   |
| Assign or change a user's role in a registry                                                         | Admin            |                                                            |        |        |   ✓   |

### Inherited Registry role

The registry's membership list shows each user's inherited (effective) registry role (in light gray) next to the role dropdown in their row.

<Frame>
  <img alt="Registry membership list showing the user's effective registry role" />
</Frame>

A user's effective role in a particular registry matches their *highest* role among their role in the organization, the registry, and the team that owns the registry, whether inherited or explicitly assigned. For example:

* A team **Admin** or organization **Admin** with the **Viewer** role in a particular registry owned by the team is effectively an **Admin** of the registry.
* A registry **Viewer** with the **Member** role in the team is effectively a **Member** of the registry.
* A team **Viewer** with the **Member** role in a particular registry is effectively a **Member** of the registry.

### Restricted Viewer role details

The **Restricted Viewer** role is Generally Available (GA). For Dedicated Cloud and Self-Managed, Server v0.75.0 or newer is required.

This role provides read-only access to registry artifacts without the ability to create, update, or delete collections, automations, or other registry resources.

Unlike a **Viewer**, a **Restricted Viewer**:

* Cannot download artifact files or access file contents.
* Cannot use artifacts with `wandb.Run.use_artifact()` in the W\&B SDK.

#### SDK compatibility

<Note>
  **SDK version requirement**

  To use the W\&B SDK to access artifacts as a **Restricted Viewer**, you must use W\&B SDK version 0.19.9 or higher. Otherwise, some SDK commands will result in permission errors.
</Note>

When a **Restricted Viewer** uses the SDK, certain functions are not available or work differently.

The following methods are not available and result in permission errors:

* [`Run.use_artifact()`](/models/ref/python/experiments/run/#method-runuse_artifact)
* [`Artifact.download()`](/models/ref/python/experiments/artifact/#method-artifactdownload)
* [`Artifact.file()`](/models/ref/python/experiments/artifact/#method-artifactfile)
* [`Artifact.files()`](/models/ref/python/experiments/artifact/#method-artifactfiles)

The following methods are limited to artifact metadata:

* [`Artifact.get_entry()`](/models/ref/python/experiments/artifact/#method-artifactget_entry)
* [`Artifact.get_path()`](/models/ref/python/experiments/artifact/#method-artifactget_path)
* [`Artifact.get()`](/models/ref/python/experiments/artifact/#method-artifactget)
* [`Artifact.verify()`](/models/ref/python/experiments/artifact/#method-artifactverify)

### Cross-registry permissions

A user can have different roles in different registries. For example, a user can be a **Restricted Viewer** in Registry A but a **Viewer** in Registry B. In this case:

* The same artifact linked to both registries will have different access levels
* In Registry A, the user is a **Restricted Viewer** and cannot download files or use the artifact
* In Registry B, the user is a **Viewer** and can download files and use the artifact
* In other words, access is determined by the registry in which the artifact is accessed


# Create a collection
Source: https://docs.wandb.ai/models/registry/create_collection

Create a collection of linked artifact versions within a W&B Registry and configure accepted artifact types.

A *collection* is a set of linked artifact versions within a registry. Each collection represents a distinct task or use case.

For example, within a registry you might have multiple collections. Each collection contains a different dataset such as MNIST, CIFAR-10, or ImageNet.

As another example, you might have a registry called "chatbot" that contains a collection for model artifacts, another collection for dataset artifacts, and another collection for fine-tuned model artifacts.

How you organize a registry and their collections is up to you.

<Note>
  If you are familiar with W\&B Model Registry, you might aware of registered models. Registered models in the Model Registry are now referred to as collections in the W\&B Registry.
</Note>

## Collection types

Each collection accepts one, and only one, *type* of artifact. The type you specify restricts what sort of artifacts you, and other members of your organization, can link to that collection.

<Note>
  You can think of artifact types similar to data types in programming languages such as Python. In this analogy, a collection can store strings, integers, or floats but not a mix of these data types.
</Note>

For example, suppose you create a collection that accepts "dataset" artifact types. This means that you can only link future artifact versions that have the type "dataset" to this collection. Similarly, you can only link artifacts of type "model" to a collection that accepts only model artifact types.

<Note>
  You specify an artifact's type when you create that artifact object. Note the `type` field in `wandb.Artifact()`:

  ```python theme={null}
  import wandb

  # Initialize a run
  with wandb.init(
    entity = "<team_entity>",
    project = "<project>"
    ) as run:

    # Create an artifact object
    artifact = wandb.Artifact(
        name="<artifact_name>", 
        type="<artifact_type>"
        )
  ```
</Note>

When you create a collection, you can select from a list of predefined artifact types. The artifact types available to you depend on the registry that the collection belongs to.

Before you link an artifact to a collection or create a new collection, [investigate the types of artifacts that collection accepts](#check-the-types-of-artifact-that-a-collection-accepts).

### Check the types of artifact that a collection accepts

Before you link to a collection, inspect the artifact type that the collection accepts. You can inspect the artifact types that collection accepts programmatically with the W\&B Python SDK or interactively with the W\&B App

<Note>
  An error message appears if you try to create link an artifact to a collection that does not accept that artifact type.
</Note>

<Tabs>
  <Tab title="W&B App">
    You can find the accepted artifact types on the registry card on the homepage or within a registry's settings page.

    For both methods, first navigate to your W\&B Registry.

    Within the homepage of the W\&B Registry, you can view the accepted artifact types by scrolling to the registry card of that registry. The gray horizontal ovals within the registry card lists the artifact types that registry accepts.

    <Frame>
      <img alt="Artifact types selection" />
    </Frame>

    For example, the following image shows multiple registry cards on the W\&B Registry homepage. Within the **Model** registry card, you can see two artifact types: **model** and **model-new**.

    To view accepted artifact types within a registry's settings page:

    1. Click on the registry card you want to view the settings for.
    2. Click on the gear icon in the upper right corner.
    3. Scroll to the **Accepted artifact types** field.
  </Tab>

  <Tab title="Python SDK (Beta)">
    Programmatically view the artifact types that a registry accepts with the W\&B Python SDK:

    ```python theme={null}
    import wandb

    registry_name = "<registry_name>"
    artifact_types = wandb.Api().project(name=f"wandb-registry-{registry_name}").artifact_types()
    print(artifact_type.name for artifact_type in artifact_types)
    ```

    <Note>
      Note that you do not initialize a run with the following code snippet. This is because it is unnecessary to create a run if you are only querying the W\&B API and not tracking an experiment, artifact and so on.
    </Note>
  </Tab>
</Tabs>

Once you know what type of artifact a collection accepts, you can [create a collection](#create-a-collection).

## Create a collection

Interactively or programmatically create a collection within a registry. You can not change the type of artifact that a collection accepts after you create it.

### Programmatically create a collection

Use the `wandb.Run.link_artifact()` method to link an artifact to a collection. Specify both the collection and the registry to the `target_path` field as a path that takes the form of:

```python theme={null}
f"wandb-registry-{registry_name}/{collection_name}"
```

Where `registry_name` is the name of the registry and `collection_name` is the name of the collection. Ensure to append the prefix `wandb-registry-` to the registry name.

<Note>
  W\&B automatically creates a collection for you if you try to link an artifact to a collection that does not exist. If you specify a collection that does exists, W\&B links the artifact to the existing collection.
</Note>

The following code snippet shows how to programmatically create a collection. Ensure to replace the values enclosed in `<>` with your own:

```python theme={null}
import wandb

# Initialize a run
with wandb.init(entity = "<team_entity>", project = "<project>") as run:

  # Create an artifact object
  artifact = wandb.Artifact(
    name = "<artifact_name>",
    type = "<artifact_type>"
    )

  registry_name = "<registry_name>"
  collection_name = "<collection_name>"
  target_path = f"wandb-registry-{registry_name}/{collection_name}"

  # Link the artifact to a collection
  run.link_artifact(artifact = artifact, target_path = target_path)
```

### Interactively create a collection

The following steps describe how to interactively create a collection using the W\&B Registry:

1. Navigate to the W\&B Registry at [https://wandb.ai/registry/](https://wandb.ai/registry/).
2. Select a registry.
3. Click on the **Create collection** button in the upper right hand corner.
4. Provide a name for your collection in the **Name** field.
5. Select a type from the **Type** dropdown. Or, if the registry enables custom artifact types, provide one or more artifact types that this collection accepts.
6. Optionally provide a description of your collection in the **Description** field.
7. Optionally add one or more tags in the **Tags** field.
8. Click **Link version**.
9. From the **Project** dropdown, select the project where your artifact is stored.
10. From the **Artifact** collection dropdown, select your artifact.
11. From the **Version** dropdown, select the artifact version you want to link to your collection.
12. Click on the **Create collection** button.

<Frame>
  <img alt="Create a new collection" />
</Frame>


# Create a registry
Source: https://docs.wandb.ai/models/registry/create_registry

Create a W&B Registry with configurable visibility and accepted artifact types using the App UI or Python SDK.

A registry offers flexibility and control over the artifact types that you can use, allows you to restrict the registry's visibility, and more.

## Create a registry

Create a registry either programmatically using the W\&B Registry UI or the W\&B Python SDK.

<Tabs>
  <Tab title=" W&B Registry UI">
    1. Navigate to the W\&B Registry at [https://wandb.ai/registry/](https://wandb.ai/registry/).
    2. Click on the **Create registry** button.
    3. Provide a name for the registry in the **Name** field.
    4. Optionally provide a description about the registry.
    5. Select who can view the registry from the **Registry visibility** dropdown. See [Registry visibility types](./configure_registry#registry-visibility-types) for more information on registry visibility options.
    6. Select either **All types** or **Specify types** from the **Accepted artifacts type** dropdown.
    7. (If you select **Specify types**) Add one or more artifact types that your registry accepts.
    8. Click on the **Create registry** button.
  </Tab>

  <Tab title="Python SDK">
    Use the [`wandb.Api().create_registry()`](/models/ref/python/#method-apicreate_registry) method to create a registry programmatically. Provide a name and [visibility](#visibility-types) for the registry for the `name` and `visibility` parameters, respectively.

    Copy and paste the code block below. Replace the values enclosed in `<>` with your own:

    ```python theme={null}
    import wandb

    registry = wandb.Api().create_registry(
        name="<registry_name>",
        visibility="< 'restricted' | 'organization' >",
    )
    ```

    See the  [`wandb.Api().create_registry()`](/models/ref/python/#method-apicreate_registry) method reference for a full list of parameters that you can provide when you create a registry.
  </Tab>
</Tabs>

<Note>
  An artifact type cannot be removed from a registry once it is saved in the registry's settings.
</Note>

For example, the following image shows a registry called `Fine_Tuned_Models` that a user is about to create. The registry is **Restricted** to only members that are manually added to the registry.

<Frame>
  <img alt="Creating a new registry" />
</Frame>

## Visibility types

The *visibility* of a registry determines who can access that registry. Restricting the visibility of a registry helps ensure that only specified members can access that registry.

There are two type registry visibility options for a registry:

| Visibility   | Description                                                |
| ------------ | ---------------------------------------------------------- |
| Restricted   | Only invited organization members can access the registry. |
| Organization | Everyone in the org can access the registry.               |

A team admin or registry admin can set the visibility of a registry.

The user who creates a registry with Restricted visibility is added to the registry automatically as its registry admin.

## Configure the visibility of a registry

A team admin or registry admin can assign the visibility of a registry during or after the creation of a registry.

To restrict the visibility of an existing registry:

1. Navigate to the W\&B Registry at [https://wandb.ai/registry/](https://wandb.ai/registry/).
2. Select a registry.
3. Click on the gear icon on the upper right hand corner.
4. From the **Registry visibility** dropdown, select the desired registry visibility.
5. If you select **Restricted visibility**:
   1. Add members of your organization that you want to have access to this registry. Scroll to the **Registry members and roles** section and click on the **Add member** button.
   2. Within the **Member** field, add the email or username of the member you want to add.
   3. Click **Add new member**.

<Frame>
  <img alt="Changing registry visibility settings from private to public or team-restricted access" />
</Frame>

See [Create a registry](./create_registry#create-a-custom-registry) for more information on how assign the visibility of a registry when a team admin creates it.


# Delete registry
Source: https://docs.wandb.ai/models/registry/delete_registry

Delete a W&B Registry as a Team admin or Registry admin using the Python SDK or the W&B App UI.

This page shows how a Team admin or Registry admin can delete a registry.

* A Team admin can delete any registry in the organization.
* A Registry admin can delete a registry that they created.

Deleting a registry also deletes collections that belong to that registry, but does not delete artifacts linked to the registry.  Such an artifact remains in the original project that the artifact was logged to.

<Tabs>
  <Tab title="Python SDK">
    Use the `wandb` API's `delete()` method to delete a registry programmatically.  The following example illustrates how to:

    1. Fetch the registry you want to delete with `api.registry()`.
    2. Call the `delete()` method on the returned registry object to delete the registry.

    ```python theme={null}
    import wandb

    # Initialize the W&B API
    api = wandb.Api()

    # Fetch the registry you want to delete
    fetched_registry = api.registry("<registry_name>")

    # Deleting a registry
    fetched_registry.delete()
    ```
  </Tab>

  <Tab title="W&B App">
    1. Navigate to the W\&B Registry at [https://wandb.ai/registry/](https://wandb.ai/registry/).
    2. Select the registry you want to delete.
    3. Click the gear icon in the upper right corner to view the registry's settings.
    4. To delete the registry, click the trash can icon in the upper right corner of the settings page.
    5. Confirm the registry to delete by entering its name in the modal that appears, then click **Delete**.
  </Tab>
</Tabs>


# Download an artifact from a registry
Source: https://docs.wandb.ai/models/registry/download_use_artifact

Download an artifact linked to a W&B Registry collection by constructing its path and using the Python SDK.

Use the W\&B Python SDK to download an artifact linked to a registry. To download and use an artifact, you need to know the name of the registry, the name of the collection, and the alias or index of the artifact version you want to download.

Once you know the properties of the artifact, you can [construct the path to the linked artifact](#construct-path-to-linked-artifact) and download the artifact. Alternatively, you can [copy and paste a pre-generated code snippet](#copy-and-paste-pre-generated-code-snippet) from the W\&B App UI to download an artifact linked to a registry.

## Construct path to linked artifact

To download an artifact linked to a registry, you must know the path of that linked artifact. The path consists of the registry name, collection name, and the alias or index of the artifact version you want to access.

Once you have the registry, collection, and alias or index of the artifact version, you can construct the path to the linked artifact using the following string template:

```python theme={null}
# Artifact name with version index specified
f"wandb-registry-{REGISTRY}/{COLLECTION}:v{INDEX}"

# Artifact name with alias specified
f"wandb-registry-{REGISTRY}/{COLLECTION}:{ALIAS}"
```

Replace the values within the curly braces `{}` with the name of the registry, collection, and the alias or index of the artifact version you want to access.

Use the `wandb.Run.use_artifact()` method to access the artifact and download its contents once you have the path of the linked artifact. The following code snippet shows how to use and download an artifact linked to the W\&B Registry. Ensure to replace values within `<>` with your own:

```python theme={null}
import wandb

REGISTRY = '<registry_name>'
COLLECTION = '<collection_name>'
ALIAS = '<artifact_alias>'

with wandb.init(entity = '<team_name>', project = '<project_name>')  as run:
    artifact_name = f"wandb-registry-{REGISTRY}/{COLLECTION}:{ALIAS}"
    # artifact_name = '<artifact_name>' # Copy and paste Full name specified in the W&B Registry UI
    fetched_artifact = run.use_artifact(artifact_or_name = artifact_name)  
    download_path = fetched_artifact.download()  
```

The `wandb.Run.use_artifact()` method both creates a [run](/models/runs) and marks the artifact you download as the input to that run.
Marking an artifact as the input to a run enables W\&B to track the lineage of that artifact.

If you do not want to create a run, you can use the `wandb.Api()` object to access the artifact:

```python theme={null}
import wandb

REGISTRY = "<registry_name>"
COLLECTION = "<collection_name>"
VERSION = "<version>"

api = wandb.Api()
artifact_name = f"wandb-registry-{REGISTRY}/{COLLECTION}:{VERSION}"
artifact = api.artifact(name = artifact_name)
```

<details>
  <summary>Example: Use and download an artifact linked to the W\&B Registry</summary>

  The following code example shows how a user can download an artifact linked to a collection called `phi3-finetuned` in the **Fine-tuned Models** registry. The alias of the artifact version is set to `production`.

  ```python theme={null}
  import wandb

  TEAM_ENTITY = "product-team-applications"
  PROJECT_NAME = "user-stories"

  REGISTRY = "Fine-tuned Models"
  COLLECTION = "phi3-finetuned"
  ALIAS = 'production'

  # Initialize a run inside the specified team and project
  with wandb.init(entity=TEAM_ENTITY, project = PROJECT_NAME) as run:

      artifact_name = f"wandb-registry-{REGISTRY}/{COLLECTION}:{ALIAS}"

      # Access an artifact and mark it as input to your run for lineage tracking
      fetched_artifact = run.use_artifact(artifact_or_name = artifact_name)  

      # Download artifact. Returns path to downloaded contents
      downloaded_path = fetched_artifact.download()  
  ```
</details>

See [`wandb.Run.use_artifact()`](/models/ref/python/experiments/run#use_artifact) and [`Artifact.download()`](/models/ref/python/experiments/artifact#download) in the API Reference for parameters and return type.

<Note>
  **Users with a personal entity that belong to multiple organizations**

  Users with a personal entity that belong to multiple organizations must also specify either the name of their organization or use a team entity when accessing artifacts linked to a registry.

  ```python theme={null}
  import wandb

  REGISTRY = "<registry_name>"
  COLLECTION = "<collection_name>"
  VERSION = "<version>"

  # Ensure you are using your team entity to instantiate the API
  api = wandb.Api(overrides={"entity": "<team-entity>"})
  artifact_name = f"wandb-registry-{REGISTRY}/{COLLECTION}:{VERSION}"
  artifact = api.artifact(name = artifact_name)

  # Construct the path
  api = wandb.Api()
  artifact_name = f"wandb-registry-{REGISTRY}/{COLLECTION}:{VERSION}"
  artifact = api.artifact(name = artifact_name)
  ```
</Note>

## Copy and paste pre-generated code snippet

W\&B creates a code snippet that you can copy and paste into your Python script, notebook, or terminal to download an artifact linked to a registry.

1. Navigate to the W\&B Registry.
2. Select the name of the registry that contains your artifact.
3. Select the name of the collection.
4. From the list of artifact versions, select the version you want to access.
5. Select the **Usage** tab.
6. Copy the code snippet shown in the **Usage API** section.
7. Paste the code snippet into your Python script, notebook, or terminal.

<Frame>
  <img alt="Step-by-step process to locate and copy usage code snippet from the registry UI" />
</Frame>


# Lineage graphs and audit history
Source: https://docs.wandb.ai/models/registry/lineage

Use lineage graphs to visualize a linked artifact's history and audit a collection's history.

Use a lineage graph to visualize a linked artifact's history. Audit a collection's history to track changes made to artifacts in that collection.

## Lineage graphs

Within a collection in the W\&B Registry, you can view a history of the artifacts that an ML experiment uses. This history is called a *lineage graph*.

A lineage graph shows:

* Artifacts used as [inputs to a run](/models/artifacts/explore-and-traverse-an-artifact-graph#track-the-input-of-a-run).
* Artifacts created as [outputs from a run](/models/artifacts/explore-and-traverse-an-artifact-graph#track-the-output-of-a-run).

In other words, a lineage graph shows the input and output of a run.

For example, the following image shows a typical lineage graph for artifacts created and used throughout an ML experiment:

<Frame>
  <img alt="Registry lineage" />
</Frame>

From left to right, the image shows:

1. Multiple runs log the `split_zoo_dataset:v0` artifact.
2. The "zesty-snowball-7" run uses the `split_zoo_dataset:v0` artifact for training.
3. The output of the "zesty-snowball-7" run is a model artifact called `zoo-qne08r7u:v0`.
4. A run called "glamorous-planet-8" uses the model artifact `zoo-qne08r7u:v0` to evaluate the model.

To view a lineage graph for an artifact in a collection:

1. Navigate to the W\&B Registry.
2. Select the collection that contains the artifact.
3. From the dropdown, select the artifact version you want to view its lineage graph.
4. Select the **Lineage** tab.
5. Select a node to view detailed information about the run or artifact.

<Info>
  See [Enable lineage graph tracking](/models/artifacts/explore-and-traverse-an-artifact-graph#enable-lineage-graph-tracking) to learn how to track the input and output of a run using the W\&B Python SDK.
</Info>

The following image shows the expanded detailed view of a run (`zesty-snowball-7`) when you select a node in the lineage graph:

<Frame>
  <img alt="Expanded lineage node" />
</Frame>

The following image shows the expanded detailed view of an artifact (`zoo-qne08r7u:v0`) when you select an artifact node in the lineage graph:

<Frame>
  <img alt="Expanded artifact node details" />
</Frame>

<Info>
  You can also view lineage graphs for artifacts you log to W\&B that are not part of a collection. See [Explore artifact graphs](/models/artifacts/explore-and-traverse-an-artifact-graph) for more information.
</Info>

### Create a custom view

Click **Custom** in the top-right corner of the lineage graph to create a custom view. You can filter and customize the lineage graph with the following options:

* **Filter by artifact type**: Filter by artifact type values logged to W\&B. For example, if you log an artifact with the type "dataset", then "dataset" is available as a filter value.
* **Filter by run job type**: Filter by run job type values logged to W\&B. For example, if you log a run with the job type "training", then "training" is available as a filter value.
* **Include extended lineage**: Display items that are not in the direct lineage for the base version.
* **Include generated artifacts**: Items created programmatically will show.
* **Expand clusters**: Similar items with similar connections will not be grouped together.
* **Downstream hops**: Descendant generations relative to the active node.
* **Upstream hops**: Ancestor generations relative to the active node.

To reset the lineage graph to its default view, click the backward arrow button.

## Audit a collection's history

View actions that members of your organization take on that collection. You can view:

* If an alias was added or removed from an artifact version.
* If an artifact version was added or removed from a collection.

For both actions, you can view the user that performed the action and the date the action occurred.

To view a collection's action history:

1. Navigate to the W\&B Registry.
2. Select the collection you want to view its action history.
3. Select the dropdown menu next to the collection name.
4. Select the **Action History** option.


# Link an artifact version to a collection
Source: https://docs.wandb.ai/models/registry/link_version

Link an artifact version to a collection in W&B Registry to share it across your organization.

To make an artifact version available to your organization, *link* it to a [collection](/models/registry/create_collection) in
the W\&B Registry. Linking moves the version from a [private, project-level scope to a shared, organization-level scope](/models/registry/create_registry#visibility-types). You
can link an artifact version [programmatically by using the W\&B Python SDK or interactively in the W\&B App](/models/registry/link_version#link-an-artifact-to-a-collection).

When you link an artifact, W\&B creates a reference between the source artifact and the collection entry. The linked version points to the source artifact version that was logged to a run within a project. You can view both the linked version in the collection and the source version in the project where it was logged.

## Link an artifact to a collection

Based on your use case, follow the instructions described in the tabs below to link an artifact version.

<Note>
  Before you start, check the following:

  * The types of artifacts that collection permits. For more information about collection types, see "Collection types" within [Create a collection](./create_collection).
  * The registry that the collection belongs to already exists. To check that the registry exists, navigate to the [Registry App and search for](/models/registry/search_registry) the name of the registry.
</Note>

<Tabs>
  <Tab title="Python SDK">
    Programmatically link an artifact version to a collection with [`wandb.Run.link_artifact()`](/models/ref/python/experiments/run#link_artifact) or
    [`wandb.Artifact.link()`](/models/ref/python/experiments/artifact#method-artifactlink).

    <Note>
      Use `wandb.Run.link_artifact()` to link an artifact version [within the context of a run](#link-an-artifact-version-within-the-context-of-a-run). Use `wandb.Artifact.link()` to link an *existing artifact version* [outside the context of a run](#link-an-artifact-version-outside-the-context-of-a-run).
    </Note>

    For both approaches, specify the name of the artifact (`wandb.Artifact(name="<name>"`), the type  of artifact (`wandb.Artifact(type="<type>"`), and the `target_path` (`wandb.Artifact(target_path="<target_path>"`)) of the collection and registry you want to link the artifact version to.

    The target path consists of the prefix `"wandb-registry"`, the name of the registry, and the name of the collection separated by a forward slashes:

    ```text theme={null}
    wandb-registry-{REGISTRY_NAME}/{COLLECTION_NAME}
    ```

    ### Link an artifact version within the context of a run

    Use `wandb.Run.link_artifact()` to link an artifact version within the context of a run. To do so,
    first initialize a run with `wandb.init()`. Next, create an artifact object and add files to it. Finally, use the `wandb.Run.link_artifact()` method to link the artifact version to the collection.

    When you use this approach, a run is created in your W\&B project. The artifact version is linked to the collection and is associated with that run.

    Copy and paste the code snippet below. Replace values enclosed in `<>` with your own:

    ```python theme={null}
    import wandb

    entity = "<team_entity>"          # Your team entity
    project = "<project_name>"        # The name of the project that contains your artifact

    # Initialize a run
    with wandb.init(entity = entity, project = project) as run:

      # Create an artifact object
      # The type parameter specifies both the type of the 
      # artifact object and the collection type
      artifact = wandb.Artifact(name = "<name>", type = "<type>")

      # Add the file to the artifact object. 
      # Specify the path to the file on your local machine.
      artifact.add_file(local_path = "<local_path_to_artifact>")

      # Specify the collection and registry to link the artifact to
      REGISTRY_NAME = "<registry_name>"  
      COLLECTION_NAME = "<collection_name>"
      target_path=f"wandb-registry-{REGISTRY_NAME}/{COLLECTION_NAME}"

      # Link the artifact to the collection
      run.link_artifact(artifact = artifact, target_path = target_path)
    ```

    ### Link an artifact version outside the context of a run

    Use `wandb.Artifact.link()` to link an existing artifact version outside the context of a run. With this approach,
    you do not need to initialize a run with `wandb.init()`. This means that a run is not created in your W\&B project. In other
    words, the artifact version is linked to the collection without being associated with a run.

    First, create an artifact object and add files to it. Next, use the `wandb.Artifact.link()` method to link the artifact version to the collection.

    Copy and paste the code snippet below. Replace values enclosed in `<>` with your own:

    ```python theme={null}
    import wandb

    # Create an artifact object
    # The type parameter specifies both the type of the 
    # artifact object and the collection type
    artifact = wandb.Artifact(name = "<name>", type = "<type>")

    # Add the file to the artifact object. 
    # Specify the path to the file on your local machine.
    artifact.add_file(local_path = "<local_path_to_artifact>")

    # Specify the collection and registry to link the artifact to
    REGISTRY_NAME = "<registry_name>"  
    COLLECTION_NAME = "<collection_name>"
    target_path=f"wandb-registry-{REGISTRY_NAME}/{COLLECTION_NAME}"

    # Link the artifact to the collection
    artifact.link(target_path = target_path)
    ```
  </Tab>

  <Tab title="W&B Registry">
    1. Navigate to the W\&B Registry.
       <Frame>
         <img alt="W&B Registry UI with project sidebar" />
       </Frame>
    2. Hover your mouse next to the name of the collection you want to link an artifact version to.
    3. Select the **action (<Icon icon="ellipsis" />)** menu next to **View details**.
    4. From the dropdown, select **Link new version**.
    5. From the sidebar that appears, select the name of a team from the **Team** dropdown.
    6. From the **Project** dropdown, select the name of the project that contains your artifact.
    7. From the **Artifact** dropdown, select the name of the artifact.
    8. From the **Version** dropdown, select the artifact version you want to link to the collection.
  </Tab>

  <Tab title="Artifact browser">
    1. Navigate to your project's artifact browser on the W\&B App at: `https://wandb.ai/<entity>/<project>/artifacts`
    2. Select the Artifacts icon in the project sidebar.
    3. Click on the artifact version you want to link to your registry.
    4. Within the **Version overview** section, click the **Link to registry** button.
    5. From the modal that appears on the right of the screen, select an artifact from the **Select a register model** menu dropdown.
    6. Click **Next step**.
    7. (Optional) Select an alias from the **Aliases** dropdown.
    8. Click **Link to registry**.
  </Tab>
</Tabs>

You can [view a linked artifact's metadata, version data, usage, lineage information](/models/registry/link_version#view-linked-artifacts-in-a-registry") and more in the Registry App.

## View linked artifacts in a registry

View information about linked artifacts such as metadata, lineage, and usage information in the W\&B Registry.

1. Navigate to the W\&B Registry.
2. Select the name of the registry that you linked the artifact to.
3. Select the name of the collection.
4. If the collection's artifacts log metrics, compare metrics across versions by clicking **Show metrics**.
5. From the list of artifact versions, select the version you want to access. Version numbers are incrementally assigned to each linked artifact version starting with `v0`.
6. To view details about an artifact version, click the version. From the tabs in this page, you can view that version's metadata (including logged metrics), lineage, and usage information.

Make note of the **Full Name** field within the **Version** tab. The full name of a linked artifact consists of the registry, collection name, and the alias or index of the artifact version.

```text title="Full name of a linked artifact" theme={null}
wandb-registry-{REGISTRY_NAME}/{COLLECTION_NAME}:v{INTEGER}
```

You need the full name of a linked artifact to access the artifact version programmatically.

## Troubleshooting

Below are some common things to double check if you are not able to link an artifact.

### Logging artifacts from a personal account

Artifacts logged to W\&B with a personal entity can not be linked to the registry. Make sure that you log artifacts using a team entity within your organization. Only artifacts logged within an organization's team can be linked to the organization's registry.

<Note>
  Ensure that you log an artifact with a team entity if you want to link that artifact to a registry.
</Note>

#### Find your team entity

W\&B uses the name of your team as the team's entity. For example, if your team is called **team-awesome**, your team entity is `team-awesome`.

You can confirm the name of your team by:

1. Navigate to your team's W\&B profile page.
2. Copy the site's URL. It has the form of `https://wandb.ai/<team>`. Where `<team>` is the both the name of your team and the team's entity.

#### Log from a team entity

1. Specify the team as the entity when you initialize a run with [`wandb.init()`](/models/ref/python/functions/init). If you do not specify the `entity` when you initialize a run, the run uses your default entity which may or may not be your team entity.

   ```python theme={null}
   import wandb   

   with wandb.init(entity='<team_entity>', project='<project_name>') as run:
       # Log your artifact here
   ```

2. Log the artifact to the run either with `wandb.Run.log_artifact()` or by creating an Artifact object and then adding files to it with:

   ```python theme={null}
   artifact = wandb.Artifact(name="<artifact_name>", type="<type>")
   ```

   To log artifacts, see [Construct artifacts](/models/artifacts/construct-an-artifact/).

3. If an artifact is logged to your personal entity, you will need to re-log it to an entity within your organization.

### Confirm the path of a registry in the W\&B App UI

There are two ways to confirm the path of a registry with the UI: create an empty collection and view the collection details or copy and paste the autogenerated code on the collection's homepage.

#### Copy and paste autogenerated code

1. Navigate to the W\&B Registry at [https://wandb.ai/registry/](https://wandb.ai/registry/).
2. Click the registry you want to link an artifact to.
3. At the top of the page, you will see an autogenerated code block.
4. Copy and paste this into your code, ensure to replace the last part of the path with the name of your collection.

<Frame>
  <img alt="Auto-generated code snippet" />
</Frame>

#### Create an empty collection

1. Navigate to the W\&B Registry at [https://wandb.ai/registry/](https://wandb.ai/registry/).
2. Click the registry you want to link an artifact to.
3. Click on the empty collection. If an empty collection does not exist, create a new collection.
4. Within the code snippet that appears, identify the `target_path` field within `.link_artifact()`.
5. (Optional) Delete the collection.

<Frame>
  <img alt="Create an empty collection" />
</Frame>

For example, after completing the steps outlined, you find the code block with the `target_path` parameter:

```python theme={null}
target_path = 
      "smle-registries-bug-bash/wandb-registry-Golden Datasets/raw_images"
```

Breaking this down into its components, you can see what you will need to use to create the path to link your artifact programmatically:

```python theme={null}
ORG_ENTITY_NAME = "smle-registries-bug-bash"
REGISTRY_NAME = "Golden Datasets"
COLLECTION_NAME = "raw_images"
```

<Note>
  Ensure that you replace the name of the collection from the temporary collection with the name of the collection that you want to link your artifact to.
</Note>


# Organize versions with tags
Source: https://docs.wandb.ai/models/registry/organize-with-tags

Use tags to organize collections or artifact versions within collections. You can add, remove, edit tags with the Python SDK or W&B App UI.

Create and add tags to organize your collections or artifact versions within your registry. Add, modify, view, or remove tags to a collection  or artifact version with the W\&B App UI or the W\&B Python SDK.

<Note>
  **When to use a tag versus using an alias**

  Use aliases when you need to reference a specific artifact version uniquely. For example, use an alias such as 'production' or 'latest' to ensure that `artifact_name:alias` always points to a single, specific version.

  Use tags when you want more flexibility for grouping or searching. Tags are ideal when multiple versions or collections can share the same label, and you don’t need the guarantee that only one version is associated with a specific identifier.
</Note>

## Add a tag to a collection

Use the W\&B App UI or Python SDK to add a tag to a collection:

<Tabs>
  <Tab title="W&B App">
    Use the W\&B App UI to add a tag to a collection:

    1. Navigate to the [W\&B Registry](https://wandb.ai/registry).
    2. Click on a registry card.
    3. Click **View details** next to the name of a collection.
    4. Within the collection card, click on the plus icon (**+**) next to the **Tags** field and type in the name of the tag.
    5. Press **Enter** on your keyboard.

    <Frame>
      <img alt="Adding tags to a Registry collection" />
    </Frame>
  </Tab>

  <Tab title="Python SDK">
    ```python theme={null}
    import wandb

    COLLECTION_TYPE = "<collection_type>"
    REGISTRY_NAME = "<registry_name>"
    COLLECTION_NAME = "<collection_name>"

    full_name = f"wandb-registry-{REGISTRY_NAME}/{COLLECTION_NAME}"

    collection = wandb.Api().artifact_collection(
      type_name = COLLECTION_TYPE, 
      name = full_name
      )

    collection.tags = ["your-tag"]
    collection.save()
    ```
  </Tab>
</Tabs>

## Update tags that belong to a collection

Update a tag programmatically by reassigning or by mutating the `tags` attribute. W\&B recommends, and it is good Python practice, that you reassign the `tags` attribute instead of in-place mutation.

For example, the following code snippet shows common ways to update a list with reassignment. For brevity, we continue the code example from the [Add a tag to a collection section](#add-a-tag-to-a-collection):

```python theme={null}
collection.tags = [*collection.tags, "new-tag", "other-tag"]
collection.tags = collection.tags + ["new-tag", "other-tag"]

collection.tags = set(collection.tags) - set(tags_to_delete)
collection.tags = []  # deletes all tags
```

The following code snippet shows how you can use in-place mutation to update tags that belong to an artifact version:

```python theme={null}
collection.tags += ["new-tag", "other-tag"]
collection.tags.append("new-tag")

collection.tags.extend(["new-tag", "other-tag"])
collection.tags[:] = ["new-tag", "other-tag"]
collection.tags.remove("existing-tag")
collection.tags.pop()
collection.tags.clear()
```

## View tags that belong to a collection

Use the W\&B App UI to view tags added to a collection:

1. Navigate to the [W\&B Registry](https://wandb.ai/registry).
2. Click on a registry card.
3. Click **View details** next to the name of a collection.

If a collection has one or more tags, you can view those tags within the collection card next to the **Tags** field.

<Frame>
  <img alt="Registry collection with selected tags" />
</Frame>

Tags added to a collection also appear next to the name of that collection.

For example, in the following image, a tag called "tag1" was added to the "zoo-dataset-tensors" collection.

<Frame>
  <img alt="Tag management" />
</Frame>

## Remove a tag from a collection

Use the W\&B App UI to remove a tag from a collection:

1. Navigate to the [W\&B Registry](https://wandb.ai/registry).
2. Click on a registry card.
3. Click **View details** next to the name of a collection.
4. Within the collection card, hover your mouse over the name of the tag you want to remove.
5. Click on the cancel button (**X** icon).

## Add a tag to an artifact version

Add a tag to an artifact version linked to a collection with the W\&B App UI or with the Python SDK.

<Tabs>
  <Tab title="W&B App">
    1. Navigate to the W\&B Registry at [https://wandb.ai/registry](https://wandb.ai/registry)
    2. Click on a registry card.
    3. Click **View details** next to the name of the collection you want to add a tag to.
    4. Scroll down to **Versions**.
    5. Click **View** next to an artifact version.
    6. Within the **Version** tab, click on the plus icon (**+**) next to the **Tags** field and type in the name of the tag.
    7. Press **Enter** on your keyboard.

    <Frame>
      <img alt="Adding tags to artifact versions" />
    </Frame>
  </Tab>

  <Tab title="Python SDK">
    Fetch the artifact version you want to add or update a tag to. Once you have the artifact version, you can access the artifact object's `tag` attribute to add or modify tags to that artifact. Pass in one or more tags as list to the artifacts `tag` attribute.

    Like other artifacts, you can fetch an artifact from W\&B without creating a run or you can create a run and fetch the artifact within that run. In either case, ensure to call the artifact object's `save` method to update the artifact on the W\&B servers.

    Copy and paste an appropriate code cells below to add or modify an artifact version's tag. Replace the values in `<>` with your own.

    The following code snippet shows how to fetch an artifact and add a tag without creating a new run:

    ```python title="Add a tag to an artifact version without creating a new run" theme={null}
    import wandb

    ARTIFACT_TYPE = "<TYPE>"
    REGISTRY_NAME = "<registry_name>"
    COLLECTION_NAME = "<collection_name>"
    VERSION = "<artifact_version>"

    artifact_name = f"wandb-registry-{REGISTRY_NAME}/{COLLECTION_NAME}:v{VERSION}"

    artifact = wandb.Api().artifact(name = artifact_name, type = ARTIFACT_TYPE)
    artifact.tags = ["tag2"] # Provide one or more tags in a list
    artifact.save()
    ```

    The following code snippet shows how to fetch an artifact and add a tag by creating a new run:

    ```python title="Add a tag to an artifact version during a run" theme={null}
    import wandb

    REGISTRY_NAME = "<registry_name>"
    COLLECTION_NAME = "<collection_name>"
    VERSION = "<artifact_version>"

    with wandb.init(entity = "<entity>", project="<project>") as run:

        artifact_name = f"wandb-registry-{REGISTRY_NAME}/{COLLECTION_NAME}:v{VERSION}"

        artifact = run.use_artifact(artifact_or_name = artifact_name)
        artifact.tags = ["tag2"] # Provide one or more tags in a list
        artifact.save()
    ```
  </Tab>
</Tabs>

## Update tags that belong to an artifact version

Update a tag programmatically by reassigning or by mutating the `tags` attribute. W\&B recommends, and it is good Python practice, that you reassign the `tags` attribute instead of in-place mutation.

For example, the following code snippet shows common ways to update a list with reassignment. For brevity, we continue the code example from the [Add a tag to an artifact version section](#add-a-tag-to-an-artifact-version):

```python theme={null}
artifact.tags = [*artifact.tags, "new-tag", "other-tag"]
artifact.tags = artifact.tags + ["new-tag", "other-tag"]

artifact.tags = set(artifact.tags) - set(tags_to_delete)
artifact.tags = []  # deletes all tags
```

The following code snippet shows how you can use in-place mutation to update tags that belong to an artifact version:

```python theme={null}
artifact.tags += ["new-tag", "other-tag"]
artifact.tags.append("new-tag")

artifact.tags.extend(["new-tag", "other-tag"])
artifact.tags[:] = ["new-tag", "other-tag"]
artifact.tags.remove("existing-tag")
artifact.tags.pop()
artifact.tags.clear()
```

## View tags that belong to an artifact version

View tags that belong to an artifact version that is linked to a registry with the W\&B App UI or with the Python SDK.

<Tabs>
  <Tab title="W&B App">
    1. Navigate to the [W\&B Registry](https://wandb.ai/registry).
    2. Click on a registry card.
    3. Click **View details** next to the name of the collection you want to add a tag to.
    4. Scroll down to **Versions** section.

    If an artifact version has one or more tags, you can view those tags within the **Tags** column.

    <Frame>
      <img alt="Artifact version with tags" />
    </Frame>
  </Tab>

  <Tab title="Python SDK">
    Fetch the artifact version to view its tags. Once you have the artifact version, you can view tags that belong to that artifact by viewing the artifact object's `tag` attribute.

    Similar to other artifacts, you can fetch an artifact from W\&B without creating a run or you can create a run and fetch the artifact within that run.

    Copy and paste an appropriate code cells below to add or modify an artifact version's tag. Replace the values in `<>` with your own.

    The following code snippet shows how to fetch and view an artifact version's tags without creating a new run:

    ```python title="Add a tag to an artifact version without creating a new run" theme={null}
    import wandb

    ARTIFACT_TYPE = "<TYPE>"
    REGISTRY_NAME = "<registry_name>"
    COLLECTION_NAME = "<collection_name>"
    VERSION = "<artifact_version>"

    artifact_name = f"wandb-registry-{REGISTRY_NAME}/{COLLECTION_NAME}:v{VERSION}"

    artifact = wandb.Api().artifact(name = artifact_name, type = artifact_type)
    print(artifact.tags)
    ```

    The following code snippet shows how to fetch and view artifact version's tags by creating a new run:

    ```python title="Add a tag to an artifact version during a run" theme={null}
    import wandb

    REGISTRY_NAME = "<registry_name>"
    COLLECTION_NAME = "<collection_name>"
    VERSION = "<artifact_version>"

    with wandb.init(entity = "<entity>", project="<project>") as run:

        artifact_name = f"wandb-registry-{REGISTRY_NAME}/{COLLECTION_NAME}:v{VERSION}"

        artifact = run.use_artifact(artifact_or_name = artifact_name)
        print(artifact.tags)
    ```
  </Tab>
</Tabs>

## Remove a tag from an artifact version

1. Navigate to the [W\&B Registry](https://wandb.ai/registry).
2. Click on a registry card.
3. Click **View details** next to the name of the collection you want to add a tag to
4. Scroll down to **Versions**
5. Click **View** next to an artifact version
6. Within the **Version** tab, hover your mouse over the name of the tag
7. Click on the cancel button (**X** icon)

## Search existing tags

Use the W\&B App UI to search existing tags in collections and artifact versions:

1. Navigate to the [W\&B Registry](https://wandb.ai/registry).
2. Click on a registry card.
3. Within the search bar, type in the name of a tag.

<Frame>
  <img alt="Tag-based search" />
</Frame>

## Find artifact versions with a specific tag

Use the W\&B Python SDK to find artifact versions that have a set of tags:

```python theme={null}
import wandb

api = wandb.Api()
tagged_artifact_versions = api.artifacts(
    type_name = "<artifact_type>",
    name = "<artifact_name>",
    tags = ["<tag_1>", "<tag_2>"]
)

for artifact_version in tagged_artifact_versions:
    print(artifact_version.tags)
```


# Annotate collections
Source: https://docs.wandb.ai/models/registry/registry_cards

Add descriptions and documentation to W&B Registry collections to help users understand their purpose and contents.

Add human-friendly text to your collections to help users understand the purpose of the collection and the artifacts it contains.

Depending on the collection, you might want to include information about the training data, model architecture, task, license, references, and deployment. The following lists some topics worth documenting in a collection:

W\&B recommends including at minimum these details:

* **Summary**: The purpose of the collection. The machine learning framework used for the machine learning experiment.
* **License**: The legal terms and permissions associated with the use of the machine learning model. It helps model users understand the legal framework under which they can utilize the model. Common licenses include Apache 2.0, MIT, and GPL.
* **References**: Citations or references to relevant research papers, datasets, or external resources.

If your collection contains training data, consider including these additional details:

* **Training data**: Describe the training data used
* **Processing**: Processing done on the training data set.
* **Data storage**: Where is that data stored and how to access it.

If your collection contains a machine learning model, consider including these additional details:

* **Architecture**: Information about the model architecture, layers, and any specific design choices.
* **Task**: The specific type of task or problem that the machine that the collection model is designed to perform. It's a categorization of the model's intended capability.
* **Deserialize the model**: Provide information on how someone on your team can load the model into memory.
* **Deployment**: Details on how and where the model is deployed and guidance on how the model is integrated into other enterprise systems, such as a workflow orchestration platforms.

## Add a description to a collection

Interactively or programmatically add a description to a collection with the W\&B Registry UI or Python SDK.

<Tabs>
  <Tab title="W&B Registry UI">
    1. Navigate to the [W\&B Registry](https://wandb.ai/registry/).
    2. Click on a collection.
    3. Select **View details** next to the name of the collection.
    4. Within the **Description** field, provide information about your collection. Format text within with [Markdown markup language](https://www.markdownguide.org/).
  </Tab>

  <Tab title="Python SDK">
    Use the [`wandb.Api().artifact_collection()`](/models/ref/python/public-api/api#artifact_collection) method to access a collection's description. Use the returned object's `description` property to add, or update, a description to the collection.

    Specify the collection's type for the `type_name` parameter and the collection's full name for the `name` parameter. A collection's name consists of the prefix “wandb-registry”, the name of the registry, and the name of the collection separated by a forward slashes:

    ```text theme={null}
    wandb-registry-{REGISTRY_NAME}/{COLLECTION_NAME}
    ```

    Copy and paste the following code snippet into your Python script or notebook. Replace values enclosed in angle brackets (`<>`) with your own.

    ```python theme={null}
    import wandb

    api = wandb.Api()

    collection = api.artifact_collection(
      type_name = "<collection_type>", 
      name = "<collection_name>"
      )


    collection.description = "This is a description."
    collection.save()  
    ```
  </Tab>
</Tabs>

For example, the following image shows a collection that documents a model's architecture, intended use, performance information and more.

<Frame>
  <img alt="Collection card" />
</Frame>


# Find registry items
Source: https://docs.wandb.ai/models/registry/search_registry

Learn how to search for registries, collections, and artifact versions in the W&B Registry using the global search bar or queries.

Use the [global search bar in the W\&B Registry](./search_registry#search-for-registry-items) to find a registry, collection, artifact version tag, collection tag, or alias. You can use queries to [filter registries, collections, and artifact versions](/models/registry/search_registry#query-registry-items) based on specific criteria using the W\&B Python SDK.

<Info>
  The syntax and available operators you can use to query W\&B Registry is similar, but not identical, to MongoDB queries.
</Info>

Only items that you have permission to view appear in the search results.

## Search for registry items

Use the W\&B App to search for a registry item:

1. Navigate to the W\&B Registry.
2. Specify the search term in the search bar at the top of the page. Press Enter to search.

Search results appear below the search bar if the term you specify matches an existing registry, collection name, artifact version tag, collection tag, or alias.

<Frame>
  <img alt="Searching within a Registry" />
</Frame>

## Query registry items

Use [`wandb.Api().registries()`](/models/ref/python/public-api/api#registries) and *query predicates* to filter registries, collections, and artifact versions. A query predicate is a condition that specifies the criteria that returned items must meet.

To create a query predicate, use a JSON-like dictionary that consists of [query name](/models/registry/search_registry#filterable-fields), one or more [operators](/models/registry/search_registry#supported-operators), and values. The following code snippet shows the general structure of a query predicate:

```python theme={null}
{
    "query_name": {
        "operator": value
    }
}
```

The following sections describe the available registry [query names](/models/registry/search_registry#filterable-fields), [supported operators](/models/registry/search_registry#supported-operators), and [example queries](/models/registry/search_registry#example-queries).

### Filterable fields

The following table lists query names you can use based on the type of item you want to filter:

|             | query name                                               |
| ----------- | -------------------------------------------------------- |
| registries  | `name`, `description`, `created_at`, `updated_at`        |
| collections | `name`, `tag`, `description`, `created_at`, `updated_at` |
| versions    | `tag`, `alias`, `created_at`, `updated_at`, `metadata`   |

### Supported operators

W\&B supports the following comparison and logical operators for filtering registry items:

#### Comparison operators

| Operator | Description              |
| -------- | ------------------------ |
| `$eq`    | Equal to                 |
| `$ne`    | Not equal to             |
| `$gt`    | Greater than             |
| `$gte`   | Greater than or equal to |
| `$lt`    | Less than                |
| `$lte`   | Less than or equal to    |

#### Logical operators

| Operator | Description                                                                                       |
| -------- | ------------------------------------------------------------------------------------------------- |
| `$and`   | Performs [AND](https://en.wikipedia.org/wiki/Logical_conjunction) logic to one or more conditions |
| `$or`    | Performs [OR](https://en.wikipedia.org/wiki/Logical_disjunction) logic to one or more conditions  |
| `$nor`   | Performs [NOR](https://en.wikipedia.org/wiki/Logical_NOR) logic to one or more conditions         |
| `$not`   | Performs [NOT](https://en.wikipedia.org/wiki/Negation) logic to a condition                       |

#### Other operators

| Operator    | Description                         |
| ----------- | ----------------------------------- |
| `$regex`    | Regular expression pattern matching |
| `$exists`   | Field exists/doesn't exist          |
| `$contains` | String contains value               |

### Example queries

The following code examples demonstrate some common search scenarios.

To use the `wandb.Api().registries()` method, first import the W\&B Python SDK ([`wandb`](/models/ref/python/)) library:

```python theme={null}
import wandb

# (Optional) Create an instance of the wandb.Api() class for readability
api = wandb.Api()
```

Filter all registries that contain the string `model`:

```python theme={null}
# Filter all registries that contain the string `model`
registry_filters = {
    "name": {"$regex": "model"}
}

# Returns an iterable of all registries that match the filters
registries = api.registries(filter=registry_filters)
```

Filter all collections, independent of registry, that contains the string `yolo` in the collection name:

```python theme={null}
# Filter all collections, independent of registry, that 
# contains the string `yolo` in the collection name
collection_filters = {
    "name": {"$regex": "yolo"}
}

# Returns an iterable of all collections that match the filters
collections = api.registries().collections(filter=collection_filters)
```

Filter all collections, independent of registry, that contains the string `yolo` in the collection name and possesses `cnn` as a tag:

```python theme={null}
# Filter all collections, independent of registry, that contains the
# string `yolo` in the collection name and possesses `cnn` as a tag
collection_filters = {
    "name": {"$regex": "yolo"},
    "tag": "cnn"
}

# Returns an iterable of all collections that match the filters
collections = api.registries().collections(filter=collection_filters)
```

Find all artifact versions that contains the string `model` and has either the tag `image-classification` or an `latest` alias:

```python theme={null}
# Find all artifact versions that contains the string `model` and 
# has either the tag `image-classification` or an `latest` alias
registry_filters = {
    "name": {"$regex": "model"}
}

# Use logical $or operator to filter artifact versions
version_filters = {
    "$or": [
        {"tag": "image-classification"},
        {"alias": "production"}
    ]
}

# Returns an iterable of all artifact versions that match the filters
artifacts = api.registries(filter=registry_filters).collections().versions(filter=version_filters)
```

Each item in the `artifacts` iterable in the previous code snippet is an instance of the `Artifact` class. This means that you can access each artifact's attributes, such as `name`, `collection`, `aliases`, `tags`, `created_at`, and more:

```python theme={null}
for art in artifacts:
    print(f"artifact name: {art.name}")
    print(f"collection artifact belongs to: { art.collection.name}")
    print(f"artifact aliases: {art.aliases}")
    print(f"tags attached to artifact: {art.tags}")
    print(f"artifact created at: {art.created_at}\n")
```

For a complete list of an artifact object's attributes, see the [Artifacts Class](/models/ref/python/experiments/artifact/) in the API Reference docs.

Filter all artifact versions, independent of registry or collection, created between 2024-01-08 and 2025-03-04 at 13:10 UTC:

```python theme={null}
# Find all artifact versions created between 2024-01-08 and 2025-03-04 at 13:10 UTC. 

artifact_filters = {
    "alias": "latest",
    "created_at" : {"$gte": "2024-01-08", "$lte": "2025-03-04 13:10:00"},
}

# Returns an iterable of all artifact versions that match the filters
artifacts = api.registries().collections().versions(filter=artifact_filters)
```

<Note>
  Specify the date and time in `YYYY-MM-DD HH:MM:SS` format for `created_at` and `updated_at` queries. You can omit the hours, minutes, and seconds if you want to filter by date only.
</Note>


# Reports overview
Source: https://docs.wandb.ai/models/reports

Project management and collaboration tools for machine learning projects

<CardGroup>
  <ColabLink />

  <TryProductLink />
</CardGroup>

Use W\&B Reports to:

* Organize Runs.
* Embed and automate visualizations.
* Describe your findings.
* Share updates with collaborators, either as a LaTeX zip file or a PDF.

The following image shows a section of a report created from metrics that were logged to W\&B over the course of training.

<Frame>
  <img alt="W&B report with benchmark results" />
</Frame>

View the report where the above image was taken from [here](https://wandb.ai/stacey/saferlife/reports/SafeLife-Benchmark-Experiments--Vmlldzo0NjE4MzM).

## How it works

Create a collaborative report with a few clicks.

1. Navigate to your W\&B project workspace in the W\&B App.
2. Click the **Create report** button in the upper right corner of your workspace.

<Frame>
  <img alt="Create report button" />
</Frame>

3. A modal titled **Create Report** will appear. Select the charts and panels you want to add to your report. (You can add or remove charts and panels later).
4. Click **Create report**.
5. Edit the report to your desired state.
6. Click **Publish to project**.
7. Click the **Share** button to share your report with collaborators.

See the [Create a report](/models/reports/create-a-report/) page for more information on how to create reports interactively and programmatically with the W\&B Python SDK.

## How to get started

Depending on your use case, explore the following resources to get started with W\&B Reports:

* Check out our [video demonstration](https://www.youtube.com/watch?v=2xeJIv_K_eI) to get an overview of W\&B Reports.
* Explore the [Reports gallery](/models/reports/reports-gallery/) for examples of live reports.
* Try the [Programmatic Workspaces](https://colab.research.google.com/github/wandb/wandb-workspaces/blob/Update-wandb-workspaces-tuturial/Workspace_tutorial.ipynb) notebook to learn how to create and customize your workspace.
* Read curated Reports in [W\&B Fully Connected](https://wandb.me/fc).

## Recommended practices and tips

For best practices and tips for Experiments and logging, see [Best Practices: Reports](https://wandb.ai/wandb/pytorch-lightning-e2e/reports/W-B-Best-Practices-Guide--VmlldzozNTU1ODY1#reports).


# Clone and export reports
Source: https://docs.wandb.ai/models/reports/clone-and-export-reports

Export W&B Reports as PDF or LaTeX files, and clone reports using the App UI or the Report and Workspace API.

<Note>
  W\&B Report and Workspace API is in Public Preview.
</Note>

This page describes how to export a W\&B Report to a portable file format. It also describes how to clone an existing report to reuse its structure as a starting point for new work.

## Export a report

Export a report as a PDF or LaTeX file to share its contents outside of W\&B or to archive a static version of your analysis. In your report, select the **action (<Icon icon="ellipsis-vertical" />)** menu. Choose **Download** and select either PDF or LaTeX output format.

## Clone a report

Clone a report to reuse an existing project's template and formatting as the basis for a new report. You can clone reports in the W\&B App UI or programmatically with the Report and Workspace API.

<Tabs>
  <Tab title="W&B App">
    In your report, select the **action (<Icon icon="ellipsis-vertical" />)** menu. Choose **Clone this report**. In the modal, pick a destination for your cloned report. Choose **Clone report**.

    <Frame>
      <img alt="Cloning reports" />
    </Frame>

    When you clone a report, you specify the destination. If you clone it to a team, all team members can view it. If you clone it to your personal account, only you can view it by default.
  </Tab>

  <Tab title="Report and Workspace API">
    Use the Report and Workspace API to load an existing report from its URL and reuse it as a template for a new report.

    Load a report from a URL to use it as a template. Replace `PROJECT` with the name of your W\&B project.

    ```python theme={null}
    report = wr.Report(
        project=PROJECT, title="Quickstart Report", description="That was easy!"
    )  # Create
    report.save()  # Save
    new_report = wr.Report.from_url(report.url)  # Load
    ```

    After you load the report, edit the content in `new_report.blocks` to customize the cloned report, then save it. Replace `ENTITY` with your W\&B entity name.

    ```python theme={null}
    pg = wr.PanelGrid(
        runsets=[
            wr.Runset(ENTITY, PROJECT, "First Run Set"),
            wr.Runset(ENTITY, PROJECT, "Elephants Only!", query="elephant"),
        ],
        panels=[
            wr.LinePlot(x="Step", y=["val_acc"], smoothing_factor=0.8),
            wr.BarPlot(metrics=["acc"]),
            wr.MediaBrowser(media_keys="img", num_columns=1),
            wr.RunComparer(diff_only="split", layout={"w": 24, "h": 9}),
        ],
    )
    new_report.blocks = (
        report.blocks[:1] + [wr.H1("Panel Grid Example"), pg] + report.blocks[1:]
    )
    new_report.save()
    ```
  </Tab>
</Tabs>


# Collaborate on reports
Source: https://docs.wandb.ai/models/reports/collaborate-on-reports

Collaborate and share W&B Reports with peers, co-workers, and your team.

This page describes ways to collaborate on reports with your team. Use these workflows to share results, gather feedback, and keep important reports accessible.

You can share a report, edit it collaboratively, add comments, or star it for quick access.

## Share a report

When viewing a report, click **Share**. You can share a report in one of the following ways:

* To share a link to the report with an email address or a username, click **Invite**. Enter an email address or username, select **Can view** or **Can edit**, then click **Invite**. If you share by email, the email address doesn't need to be a member of your organization or team.
* To generate a sharing link instead, click **Share**. Adjust the permissions for the link, then click **Copy report link**. Share the link with the member.

When viewing the report, click a panel to open it in full-screen mode. If you copy the URL from the browser and share it with another user, the panel opens directly in full-screen mode when they access the link.

## Edit a report

Multiple team members can edit the same report and publish changes when they're ready. When any team member clicks the **Edit** button to begin editing the report, W\&B automatically saves a draft. Select **Save to report** to publish your changes.

If an edit conflict occurs, such as when two team members edit the report at once, a warning notification helps you resolve any conflicts.

<Frame>
  <img alt="Report sharing modal for a report in a 'Public' project" />
</Frame>

## Comment on reports

To leave a comment on a report, click **Comment**.

To comment directly on a panel, hover over the panel, then click the comment button (<Icon icon="comment" />).

<Frame>
  <img alt="Adding a comment to a panel" />
</Frame>

## Star a report

If your team has many reports, click the open star icon (<Icon icon="star" />) at the top of a report to add it to your favorites. To remove the star, click the closed star icon (<Icon icon="star" />). When viewing your team's list of reports, click the open star in a report's row to add it to your favorites. Starred reports appear at the top of the list.

From the list of reports, you can see how many members have starred each report to gauge its popularity.


# Create a report
Source: https://docs.wandb.ai/models/reports/create-a-report

Create a W&B Report with the W&B App or programmatically.

<Note>
  W\&B Report and Workspace API is in Public Preview.
</Note>

Select a tab to learn how to create a report in the W\&B App or programmatically with the Report and Workspace API.

For an example of how to programmatically create a report, see this [Google Colab](https://colab.research.google.com/github/wandb/examples/blob/master/colabs/intro/Report_API_Quickstart.ipynb).

<Tabs>
  <Tab title="W&B App">
    1. Navigate to your project workspace in the W\&B App.

    2. In the upper-right corner of your workspace, click **Create report**.

       <Frame>
         <img alt="Create report button" />
       </Frame>

    3. A modal appears. Select the charts you want to start with. You can add or delete charts later from the report interface.

       <Frame>
         <img alt="Create report modal" />
       </Frame>

    4. Select the **Filter run sets** option to prevent new runs from being added to your report. You can toggle this option on or off. A draft report is saved automatically. Access it in the **Reports** tab.
  </Tab>

  <Tab title="Report tab">
    1. Navigate to your project workspace in the W\&B App.
    2. Select the **Reports** tab in your project.
    3. Click **Create report**.

       <Frame>
         <img alt="Create report button" />
       </Frame>
  </Tab>

  <Tab title="Report and Workspace API">
    Create a report programmatically:

    1. Install W\&B SDK (`wandb`) and Report and Workspace API (`wandb-workspaces`):
       ```bash theme={null}
       pip install wandb wandb-workspaces
       ```

    2. Import the W\&B Python SDK and the Report and Workspace API.
       ```python theme={null}
       import wandb
       import wandb_workspaces.reports.v2 as wr
       ```

    3. Create a report with `wandb_workspaces.reports.v2.Report`. Create a report instance with the Report Class Public API ([`wandb.apis.reports`](/models/ref/python/public-api/api#reports)). Specify a name for the project.
       ```python theme={null}
       report = wr.Report(project="report_standard")
       ```

    4. Save the report. Reports aren't uploaded to W\&B until you call the `.save()` method:
       ```python theme={null}
       report.save()
       ```

    For more information, see [Edit a report](/models/reports/edit-a-report).
  </Tab>
</Tabs>


# Compare runs across projects
Source: https://docs.wandb.ai/models/reports/cross-project-reports

Compare runs from two different projects with cross-project reports.

<Note>
  Watch a [video demonstrating comparing runs across projects](https://www.youtube.com/watch?v=uD4if_nGrs4) (2 min).
</Note>

Cross-project reports let you compare runs from two different projects side by side, which is useful when you want to evaluate experiments tracked under separate W\&B projects without duplicating data. Use the project selector in the run set table to pick a project.

<Frame>
  <img alt="Compare runs across different projects" />
</Frame>

The visualizations in the section pull columns from the first active run set. If you don't see the metric you're looking for in the line plot, make sure that the first run set checked in the section has that column available.

This feature supports history data on time series lines, but doesn't support pulling different summary metrics from different projects. In other words, you can't create a scatter plot from columns that are only logged in another project.

If you need to compare runs from two projects and the columns aren't working, add a tag to the runs in one project and then move those runs to the other project. You can still filter only the runs from each project, but the report includes all the columns for both sets of runs.

## View-only report links

When you need to share a report with someone outside your project (such as a stakeholder who doesn't have a W\&B account), use a view-only link. Share a view-only link to a report that is in a private project or team project.

<Frame>
  <img alt="View-only report links" />
</Frame>

View-only report links add a secret access token to the URL, so anyone who opens the link can view the page. Anyone can use the magic link to view the report without logging in first. For customers on [W\&B Local](/platform/hosting/) private cloud installations, these links remain behind your firewall, so only members of your team with access to your private instance *and* access to the view-only link can view the report.

In **view-only mode**, someone who isn't logged in can see the charts and mouse over to see tooltips of values, zoom in and out on charts, and scroll through columns in the table. When in view mode, they can't create new charts or new table queries to explore the data. View-only visitors to the report link can't click a run to get to the run page. Also, view-only visitors can't see the share modal. Instead, they see a tooltip on hover that says: `Sharing not available for view only access`.

<Info>
  The magic links are only available for "Private" and "Team" projects. For "Public" (anyone can view) or "Open" (anyone can view and contribute runs) projects, you can't turn the links on or off because the project is public, meaning it's already available to anyone with the link.
</Info>

## Send a graph to a report

If you want to preserve a chart from a workspace alongside related analysis, you can send it directly to a report. Send a graph from your workspace to a report to keep track of your progress. Click the dropdown menu on the chart or panel you'd like to copy to a report and click **Add to report** to select the destination report.


# Edit a report
Source: https://docs.wandb.ai/models/reports/edit-a-report

Edit a report interactively with the App UI or programmatically with the W&B SDK.

<Note>
  W\&B Report and Workspace API is in Public Preview.
</Note>

This page describes how to edit a report interactively with the App UI or programmatically with the W\&B SDK. It covers adding plots, run sets, code blocks, markdown, HTML elements, and rich media, as well as filtering and grouping run sets, organizing report layout, and visualizing multi-dimensional relationships.

A report's body consists of *blocks*. Blocks contain text, images, embedded visualizations, plots from experiments and runs, and panel grids.

*Panel grids* are a specific type of block that hold panels and *run sets*. Run sets are a collection of runs logged to a project in W\&B. Panels are visualizations of run set data.

<Note>
  Run the [Programmatic workspaces notebook](https://colab.research.google.com/github/wandb/wandb-workspaces/blob/Update-wandb-workspaces-tuturial/Workspace_tutorial.ipynb) for an end-to-end tutorial of creating a customized saved workspace view.
</Note>

<Note>
  To programmatically edit a report, you must have the W\&B Report and Workspace API (`wandb-workspaces`) installed along with the W\&B Python SDK:

  ```bash theme={null}
  pip install wandb wandb-workspaces
  ```
</Note>

## Add plots

Plots let you visualize run data inside a report. Each panel grid has a set of run sets and a set of panels. The run sets at the bottom of the section control what data appears on the panels in the grid. Create a new panel grid if you want to add charts that pull data from a different set of runs.

<Tabs>
  <Tab title="W&B App">
    Enter a forward slash (`/`) in the report to display a dropdown menu. Select **Add panel** to add a panel. You can add any panel that W\&B supports, including a line plot, scatter plot, or parallel coordinates chart.

    <Frame>
      <img alt="Add charts to a report" />
    </Frame>
  </Tab>

  <Tab title="Report and Workspace API">
    Add plots to a report programmatically with the SDK. Pass a list of one or more plot or chart objects to the `panels` parameter in the `PanelGrid` Public API Class. Create a plot or chart object with its associated Python Class.

    The following examples demonstrate how to create a line plot and scatter plot.

    ```python theme={null}
    import wandb
    import wandb_workspaces.reports.v2 as wr

    report = wr.Report(
        project="report-editing",
        title="Report title",
        description="Report description.",
    )

    blocks = [
        wr.PanelGrid(
            panels=[
                wr.LinePlot(x="time", y="velocity"),
                wr.ScatterPlot(x="time", y="acceleration"),
            ]
        )
    ]

    report.blocks = blocks
    report.save()
    ```

    For more information about available plots and charts you can add to a report programmatically, see `wr.panels`.
  </Tab>
</Tabs>

## Add run sets

Add run sets from projects interactively with the App UI or the W\&B SDK.

<Tabs>
  <Tab title="W&B App">
    Enter a forward slash (`/`) in the report to display a dropdown menu. From the dropdown, choose **Panel Grid**. W\&B automatically imports the run set from the project the report was created from.

    If you import a panel into a report, run names are inherited from the project. In the report, you can optionally [rename a run](/models/runs/#rename-a-run) to give the reader more context. The run is renamed only in the individual panel. If you clone the panel in the same report, the run is also renamed in the cloned panel.

    1. In the report, click the pencil icon to open the report editor.

    2. In the run set, find the run to rename. Hover over the report name and click the **action (<Icon icon="ellipsis-vertical" />)** menu. Select one of the following choices, then submit the form.

       * **Rename run for project**: Rename the run across the entire project. To generate a new random name, leave the field blank.
       * **Rename run for panel grid**: Rename the run only in the report, preserving the existing name in other contexts. Generating a new random name is not supported.

    3. Click **Publish report**.
  </Tab>

  <Tab title="Report and Workspace API">
    Add run sets from projects with the `wr.Runset()` and `wr.PanelGrid` Classes. To add a runset, follow these steps:

    1. Create a `wr.Runset()` object instance. Provide the name of the project that contains the run sets for the project parameter and the entity that owns the project for the entity parameter.
    2. Create a `wr.PanelGrid()` object instance. Pass a list of one or more runset objects to the `run sets` parameter.
    3. Store one or more `wr.PanelGrid()` object instances in a list.
    4. Update the report instance blocks attribute with the list of panel grid instances.

    Replace `[PROJECT-NAME]` with your W\&B project name and `[ENTITY-NAME]` with your W\&B entity name in the following example:

    ```python theme={null}
    import wandb
    import wandb_workspaces.reports.v2 as wr

    report = wr.Report(
        project="report-editing",
        title="Report title",
        description="Report description.",
    )

    panel_grids = wr.PanelGrid(
        runsets=[wr.RunSet(project="[PROJECT-NAME]", entity="[ENTITY-NAME]")]
    )

    report.blocks = [panel_grids]
    report.save()
    ```

    You can optionally add runsets and panels with one call to the SDK:

    ```python theme={null}
    import wandb

    report = wr.Report(
        project="report-editing",
        title="Report title",
        description="Report description.",
    )

    panel_grids = wr.PanelGrid(
        panels=[
            wr.LinePlot(
                title="line title",
                x="x",
                y=["y"],
                range_x=[0, 100],
                range_y=[0, 100],
                log_x=True,
                log_y=True,
                title_x="x axis title",
                title_y="y axis title",
                ignore_outliers=True,
                groupby="hyperparam1",
                groupby_aggfunc="mean",
                groupby_rangefunc="minmax",
                smoothing_factor=0.5,
                smoothing_type="gaussian",
                smoothing_show_original=True,
                max_runs_to_show=10,
                plot_type="stacked-area",
                font_size="large",
                legend_position="west",
            ),
            wr.ScatterPlot(
                title="scatter title",
                x="y",
                y="y",
                # z='x',
                range_x=[0, 0.0005],
                range_y=[0, 0.0005],
                # range_z=[0,1],
                log_x=False,
                log_y=False,
                # log_z=True,
                running_ymin=True,
                running_ymean=True,
                running_ymax=True,
                font_size="small",
                regression=True,
            ),
        ],
        runsets=[wr.RunSet(project="[PROJECT-NAME]", entity="[ENTITY-NAME]")],
    )


    report.blocks = [panel_grids]
    report.save()
    ```
  </Tab>
</Tabs>

## Freeze a run set

A report automatically updates run sets to show the latest data from the project. Freeze a run set when you want to preserve its state in a report at a point in time, so that later runs don't change what the report displays.

To freeze a run set when viewing a report, click the snowflake icon in its panel grid near the **Filter** button.

<Frame>
  <img alt="Freeze runset button" />
</Frame>

## Group a run set programmatically

Group runs in a run set programmatically with the [Workspace and Reports API](/models/ref/wandb_workspaces/reports). Grouping organizes related runs together in panels, which makes it easier to compare configurations and results.

You can group runs in a run set by config values, run metadata, or summary metrics. The following table lists the available grouping methods along with the available keys for each grouping method:

| Grouping method | Description                   | Available keys                                                |
| --------------- | ----------------------------- | ------------------------------------------------------------- |
| Config values   | Group runs by config values   | Values specified in config parameter in `wandb.init(config=)` |
| Run metadata    | Group runs by run metadata    | `State`, `Name`, `JobType`                                    |
| Summary metrics | Group runs by summary metrics | Values you log to a run with `wandb.Run.log()`                |

### Group runs by config values

Group runs by config values to compare runs with similar configurations. Config values are parameters you specify in your run configuration `(wandb.init(config=))`. To group runs by config values, use the `config.[KEY]` syntax, where `[KEY]` is the name of the config value you want to group by.

For example, the following code snippet first initializes a run with a config value for `group`, then groups runs in a report based on the `group` config value. Replace `[ENTITY]` and `[PROJECT]` with your W\&B entity and project names.

```python theme={null}
import wandb
import wandb_workspaces.reports.v2 as wr

entity = "[ENTITY]"
project = "[PROJECT]"

for group in ["control", "experiment_a", "experiment_b"]:
    for i in range(3):
        with wandb.init(entity=entity, project=project, group=group, config={"group": group, "run": i}, name=f"{group}_run_{i}") as run:
            # Simulate some training
            for step in range(100):
                run.log({
                    "acc": 0.5 + (step / 100) * 0.3 + (i * 0.05),
                    "loss": 1.0 - (step / 100) * 0.5
                })
```

Within your Python script or notebook, you can then group runs by the `config.group` value:

```python theme={null}
runset = wr.Runset(
  project=project,
  entity=entity,
  groupby=["config.group"]  # Group by the "group" config value
)
```

Continuing from the previous example, you can create a report with the grouped run set:

```python theme={null}
report = wr.Report(
  entity=entity,
  project=project,
  title="Grouped Runs Example",
)

report.blocks = [
  wr.PanelGrid(
      runsets=[runset],
          )
      ]

report.save()
```

### Group runs by run metadata

Group runs by a run's name (`Name`), state (`State`), or job type (`JobType`).

Continuing from the previous example, you can group your runs by their name with the following code snippet:

```python theme={null}
runset = wr.Runset(
  project=project,
  entity=entity,
  groupby=["Name"]  # Group by run names
)
```

<Note>
  The name of the run is the name you specify in the `wandb.init(name=)` parameter. If you don't specify a name, W\&B generates a random name for the run.

  You can find the name of the run in the **Overview** page of a run in the W\&B App or programmatically with `Api.runs().run.name`.
</Note>

### Group runs by summary metrics

The following examples demonstrate how to group runs by summary metrics. Summary metrics are the values you log to a run with `wandb.Run.log()`. After you log a run, you can find the names of your summary metrics in the W\&B App under the **Summary** section of a run's **Overview** page.

The syntax for grouping runs by summary metrics is `summary.[KEY]`, where `[KEY]` is the name of the summary metric you want to group by.

For example, suppose you log a summary metric called `acc`. Replace `[ENTITY]` and `[PROJECT]` with your W\&B entity and project names:

```python theme={null}
import wandb
import wandb_workspaces.reports.v2 as wr

entity = "[ENTITY]"
project = "[PROJECT]"

for group in ["control", "experiment_a", "experiment_b"]:
    for i in range(3):
        with wandb.init(entity=entity, project=project, group=group, config={"group": group, "run": i}, name=f"{group}_run_{i}") as run:
            # Simulate some training
            for step in range(100):
                run.log({
                    "acc": 0.5 + (step / 100) * 0.3 + (i * 0.05),
                    "loss": 1.0 - (step / 100) * 0.5
                })

```

You can then group runs by the `summary.acc` summary metric:

```python theme={null}
runset = wr.Runset(
  project=project,
  entity=entity,
  groupby=["summary.acc"]  # Group by summary values
)
```

## Filter a run set programmatically

Programmatically filter run sets and add them to a report with the [Workspace and Reports API](/models/ref/wandb_workspaces/reports). Filtering narrows a run set to the specific runs you want to display, based on config values, metrics, tags, or run properties.

The general syntax for a filter expression is:

```text theme={null}
Filter('[KEY]') operation [VALUE]
```

In this expression, `[KEY]` is the name of the filter, `operation` is a comparison operator (for example, `>`, `<`, `==`, `in`, `not in`, `or`, and `and`), and `[VALUE]` is the value to compare against. `Filter` is a placeholder for the type of filter you want to apply. The following table lists the available filters and their descriptions:

| Filter                   | Description               | Available keys                                                               |
| ------------------------ | ------------------------- | ---------------------------------------------------------------------------- |
| `Config('[KEY]')`        | Filter by config values   | Values specified in `config` parameter in `wandb.init(config=)`.             |
| `SummaryMetric('[KEY]')` | Filter by summary metrics | Values you log to a run with `wandb.Run.log()`.                              |
| `Tags('[KEY]')`          | Filter by tags            | Tag values that you add to your run (programmatically or with the W\&B App). |
| `Metric('[KEY]')`        | Filter by run properties  | `tags`, `state`, `displayName`, `jobType`                                    |

After you define your filters, you can create a report and pass the filtered run sets to `wr.PanelGrid(runsets=)`. See the **Report and Workspace API** tabs throughout this page for more information about how to add various elements to a report programmatically.

The following examples demonstrate how to filter run sets in a report. Replace values enclosed in brackets (for example, `[ENTITY]` and `[PROJECT]`) with your own values.

### Config filters

Filter a runset by one or more config values. Config values are parameters you specify in your run configuration (`wandb.init(config=)`).

For example, the following code snippet first initializes a run with a config value for `learning_rate` and `batch_size`, then filters runs in a report based on the `learning_rate` config value.

```python theme={null}
import wandb

config = {
    "learning_rate": 0.01,
    "batch_size": 32,
}

with wandb.init(project="[PROJECT]", entity="[ENTITY]", config=config) as run:
    # Your training code here
    pass
```

Within your Python script or notebook, you can then programmatically filter runs that have a learning rate greater than `0.01`.

```python theme={null}
import wandb_workspaces.reports.v2 as wr

runset = wr.Runset(
  entity="[ENTITY]",
  project="[PROJECT]",
  filters="Config('learning_rate') > 0.01"
)
```

You can also filter by multiple config values with the `and` operator:

```python theme={null}
runset = wr.Runset(
  entity="[ENTITY]",
  project="[PROJECT]",
  filters="Config('learning_rate') > 0.01 and Config('batch_size') == 32"
)
```

Continuing from the previous example, you can create a report with the filtered runset as follows:

```python theme={null}
report = wr.Report(
  entity="[ENTITY]",
  project="[PROJECT]",
  title="My Report"
)

report.blocks = [
  wr.PanelGrid(
      runsets=[runset],
      panels=[
          wr.LinePlot(
              x="Step",
              y=["accuracy"],
          )
      ]
  )
]

report.save()
```

### Metric filters

Filter run sets based on a run's tag (`tags`), run state (`state`), run name (`displayName`), or job type (`jobType`).

<Note>
  `Metric` filters use a different syntax than other filters. You must pass values as a list.

  ```text theme={null}
  Metric('[KEY]') operation [VALUE]
  ```
</Note>

For example, consider the following Python snippet that creates three runs and assigns each of them a name:

```python theme={null}
import wandb

with wandb.init(project="[PROJECT]", entity="[ENTITY]") as run:
    for i in range(3):
        run.name = f"run{i+1}"
        # Your training code here
        pass
```

When you create your report, you can filter runs by their display name. For example, to filter runs with names `run1`, `run2`, and `run3`, you can use the following code:

```python theme={null}
runset = wr.Runset(
  entity="[ENTITY]",
  project="[PROJECT]",
  filters="Metric('displayName') in ['run1', 'run2', 'run3']"
)
```

<Note>
  You can find the name of the run in the **Overview** page of a run in the W\&B App or programmatically with `Api.runs().run.name`.
</Note>

The following examples demonstrate how to filter a runset by the run's state (`finished`, `crashed`, or `running`):

```python theme={null}
runset = wr.Runset(
  entity="[ENTITY]",
  project="[PROJECT]",
  filters="Metric('state') in ['finished']"
)
```

```python theme={null}
runset = wr.Runset(
  entity="[ENTITY]",
  project="[PROJECT]",
  filters="Metric('state') not in ['crashed']"
)
```

### SummaryMetric filters

The following examples demonstrate how to filter a run set by summary metrics. Summary metrics are the values you log to a run with `wandb.Run.log()`. After you log a run, you can find the names of your summary metrics in the W\&B App under the **Summary** section of a run's **Overview** page.

```python theme={null}
runset = wr.Runset(
  entity="[ENTITY]",
  project="[PROJECT]",
  filters="SummaryMetric('accuracy') > 0.9"
)
```

```python theme={null}
runset = wr.Runset(
  entity="[ENTITY]",
  project="[PROJECT]",
  filters="Metric('state') in ['finished'] and SummaryMetric('train/train_loss') < 0.5"
)
```

### Tags filters

The following code snippet shows how to filter a runs set by its tags. Tags are values you add to a run (programmatically or with the W\&B App).

```python theme={null}
runset = wr.Runset(
  entity="[ENTITY]",
  project="[PROJECT]",
  filters="Tags('training') == 'training'"
)
```

## Add code blocks

Add code blocks to your report interactively with the App UI or with the W\&B SDK.

<Tabs>
  <Tab title="App UI">
    Enter a forward slash (`/`) in the report to display a dropdown menu. From the dropdown, choose **Code**.

    Select the name of the programming language on the right side of the code block to expand a dropdown. From the dropdown, select your programming language syntax. You can choose from JavaScript, Python, CSS, JSON, HTML, Markdown, and YAML.
  </Tab>

  <Tab title="Report and Workspace API">
    Use the `wr.CodeBlock` Class to create a code block programmatically. Provide the name of the language and the code you want to display for the language and code parameters, respectively.

    The following example demonstrates a list in a YAML file:

    ```python theme={null}
    import wandb
    import wandb_workspaces.reports.v2 as wr

    report = wr.Report(project="report-editing")

    report.blocks = [
        wr.CodeBlock(
            code=["this:", "- is", "- a", "cool:", "- yaml", "- file"], language="yaml"
        )
    ]

    report.save()
    ```

    This renders a code block similar to:

    ```yaml theme={null}
    this:
    - is
    - a
    cool:
    - yaml
    - file
    ```

    The following example demonstrates a Python code block:

    ```python theme={null}
    report = wr.Report(project="report-editing")


    report.blocks = [wr.CodeBlock(code=["Hello, World!"], language="python")]

    report.save()
    ```

    This renders a code block similar to:

    ```md theme={null}
    Hello, World!
    ```
  </Tab>
</Tabs>

## Add markdown

Add markdown to your report interactively with the App UI or with the W\&B SDK.

<Tabs>
  <Tab title="App UI">
    Enter a forward slash (`/`) in the report to display a dropdown menu. From the dropdown, choose **Markdown**.
  </Tab>

  <Tab title="Report and Workspace API">
    Use the `wandb.apis.reports.MarkdownBlock` Class to create a markdown block programmatically. Pass a string to the `text` parameter:

    ```python theme={null}
    import wandb
    import wandb_workspaces.reports.v2 as wr

    report = wr.Report(project="report-editing")

    report.blocks = [
        wr.MarkdownBlock(text="Markdown cell with *italics* and **bold** and $e=mc^2$")
    ]
    ```

    This renders a markdown block similar to:

    <Frame>
      <img alt="Rendered markdown block" />
    </Frame>
  </Tab>
</Tabs>

## Add HTML elements

Add HTML elements to your report interactively with the App UI or with the W\&B SDK.

<Tabs>
  <Tab title="App UI">
    Enter a forward slash (`/`) in the report to display a dropdown menu. From the dropdown, select a type of text block. For example, to create an H2 heading block, select the `Heading 2` option.
  </Tab>

  <Tab title="Report and Workspace API">
    Pass a list of one or more HTML elements to `wandb.apis.reports.blocks` attribute. The following example demonstrates how to create an H1, H2, and an unordered list:

    ```python theme={null}
    import wandb
    import wandb_workspaces.reports.v2 as wr

    report = wr.Report(project="report-editing")

    report.blocks = [
        wr.H1(text="How Programmatic Reports work"),
        wr.H2(text="Heading 2"),
        wr.UnorderedList(items=["Bullet 1", "Bullet 2"]),
    ]

    report.save()
    ```

    This renders the HTML elements to the following:

    <Frame>
      <img alt="Rendered HTML elements" />
    </Frame>
  </Tab>
</Tabs>

## Embed rich media links

Embed rich media within the report with the App UI or with the W\&B SDK.

<Tabs>
  <Tab title="App UI">
    Copy and paste URLs into reports to embed rich media within the report. The following animations demonstrate how to copy and paste URLs from Twitter, YouTube, and SoundCloud.

    ### Twitter

    Copy and paste a Tweet link URL into a report to view the Tweet within the report.

    <Frame>
      <img alt="Embedding Twitter content" />
    </Frame>

    ### Youtube

    Copy and paste a YouTube video URL link to embed a video in the report.

    <Frame>
      <img alt="Embedding YouTube videos" />
    </Frame>

    ### SoundCloud

    Copy and paste a SoundCloud link to embed an audio file into a report.

    <Frame>
      <img alt="Embedding SoundCloud audio" />
    </Frame>
  </Tab>

  <Tab title="Report and Workspace API">
    Pass a list of one or more embedded media objects to the `wandb.apis.reports.blocks` attribute. The following example demonstrates how to embed video and Twitter media into a report:

    ```python theme={null}
    import wandb
    import wandb_workspaces.reports.v2 as wr

    report = wr.Report(project="report-editing")

    report.blocks = [
        wr.Video(url="https://www.youtube.com/embed/6riDJMI-Y8U"),
        wr.Twitter(
            embed_html='<blockquote class="twitter-tweet"><p lang="en" dir="ltr">The voice of an angel, truly. <a href="https://twitter.com/hashtag/MassEffect?src=hash&amp;ref_src=twsrc%5Etfw">#MassEffect</a> <a href="https://t.co/nMev97Uw7F">pic.twitter.com/nMev97Uw7F</a></p>&mdash; Mass Effect (@masseffect) <a href="https://twitter.com/masseffect/status/1428748886655569924?ref_src=twsrc%5Etfw">August 20, 2021</a></blockquote>\n'
        ),
    ]
    report.save()
    ```
  </Tab>
</Tabs>

## Duplicate panel grids

Duplicate a panel grid to reuse its layout in the same report or in a different report. Select a panel grid and copy-paste it to duplicate it in the same report or paste it into a different report.

Highlight a whole panel grid section by selecting the drag handle in the upper right corner. Click and drag to highlight and select a region in a report such as panel grids, text, and headings.

<Frame>
  <img alt="Copying panel grids" />
</Frame>

## Delete panel grids

Select a panel grid and press `delete` on your keyboard to delete a panel grid.

<Frame>
  <img alt="Deleting panel grids" />
</Frame>

## Collapse headers to organize reports

Collapse headers in a report to hide content within a text block. When the report loads, only expanded headers show content. Collapsing headers in reports helps organize your content and prevents excessive data loading. The following gif demonstrates the process.

<Frame>
  <img alt="Collapsing headers in a report." />
</Frame>

## Visualize relationships across multiple dimensions

To compare more variables than a 2D plot can show, you can use a color gradient as an additional dimension. Using a color gradient to represent one of the variables can make patterns easier to interpret.

1. Choose a variable to represent with a color gradient, like penalty scores or learning rates. This provides a clearer understanding of how penalty (color) interacts with reward/side effects (y-axis) over training time (x-axis).
2. Highlight key trends. Hover over a specific group of runs to highlight them in the visualization.


# Embed a report
Source: https://docs.wandb.ai/models/reports/embed-reports

Embed W&B reports directly into Notion or with an HTML `iframe` element.

Embed a W\&B report in an external page or tool so that your team can view live experiment results alongside other documentation. This page describes how to generate an embed code from a report and use it in HTML, Confluence, Notion, or Gradio.

## HTML `iframe` element

Use an HTML `iframe` element to embed a report on any web page that accepts custom HTML. W\&B provides a prebuilt embed code that you can copy directly from the report.

Select the **Share** button in the upper right of a report to open a dialog, then select **Copy embed code**. The copied snippet is an `iframe` HTML element that you can paste into any page that accepts custom HTML.

<Note>
  Only *public* reports are viewable when embedded.
</Note>

<Frame>
  <img alt="Getting embed code" />
</Frame>

## Confluence

The following animation demonstrates how to insert the direct link to the report within an `iframe` cell in Confluence.

<Frame>
  <img alt="Embedding in Confluence" />
</Frame>

## Notion

The following animation demonstrates how to insert a report into a Notion document using an **Embed** block in Notion and the report's embed code.

<Frame>
  <img alt="Embedding in Notion" />
</Frame>

## Gradio

To embed a W\&B report in a Gradio app, including an app hosted on Hugging Face Spaces, use Gradio's `gr.HTML` element to render an `iframe` that points to the report URL.

```python theme={null}
import gradio as gr


def wandb_report(url):
    iframe = f'<iframe src={url} style="border:none;height:1024px;width:100%">'
    return gr.HTML(iframe)


with gr.Blocks() as demo:
    report = wandb_report(
        "https://wandb.ai/_scott/pytorch-sweeps-demo/reports/loss-22-10-07-16-00-17---VmlldzoyNzU2NzAx"
    )
demo.launch()
```


# Example reports
Source: https://docs.wandb.ai/models/reports/reports-gallery

Explore example W&B Reports showcasing use cases like experiment summaries, team collaboration, and sharing findings.

This page showcases example W\&B Reports that demonstrate common use cases, including capturing experiment notes, collaborating with colleagues, and tracking project work logs. Use these examples as inspiration for structuring your own reports.

The following sections describe common report patterns and link to example reports for each.

## Notes: Add a visualization with a quick summary

Capture an important observation, an idea for future work, or a milestone reached in the development of a project. All experiment runs in your report link to their parameters, metrics, logs, and code, so you can save the full context of your work.

Jot down some text and pull in relevant charts to illustrate your insight.

See the [What To Do When Inception-ResNet-V2 Is Too Slow](https://wandb.ai/stacey/estuary/reports/When-Inception-ResNet-V2-is-too-slow--Vmlldzo3MDcxMA) report for an example of how you can share comparisons of training time.

<Frame>
  <img alt="Quick summary notes" />
</Frame>

Save the best examples from a complex code base for quick reference and future interaction. See the [LIDAR point clouds](https://wandb.ai/stacey/lyft/reports/LIDAR-Point-Clouds-of-Driving-Scenes--Vmlldzo2MzA5Mg) report for an example of how to visualize LIDAR point clouds from the Lyft dataset and annotate with 3D bounding boxes.

<Frame>
  <img alt="Save best examples" />
</Frame>

## Collaboration: Share findings with your colleagues

Explain how to get started with a project, share what you've observed so far, and synthesize the latest findings. Your colleagues can make suggestions or discuss details using comments on any panel or at the end of the report.

Include dynamic settings so that your colleagues can explore for themselves, get additional insights, and better plan their next steps. In this example, you can visualize three types of experiments independently, compare them, or average them.

See the [SafeLife benchmark experiments](https://wandb.ai/stacey/saferlife/reports/SafeLife-Benchmark-Experiments--Vmlldzo0NjE4MzM) report for an example of how to share first runs and observations of a benchmark.

<Frame>
  <img alt="SafeLife benchmark report" />
</Frame>

<Frame>
  <img alt="Experiment comparison view" />
</Frame>

Use sliders and configurable media panels to showcase a model's results or training progress. View the [Cute Animals and Post-Modern Style Transfer: StarGAN v2 for Multi-Domain Image Synthesis](https://wandb.ai/stacey/stargan/reports/Cute-Animals-and-Post-Modern-Style-Transfer-StarGAN-v2-for-Multi-Domain-Image-Synthesis---VmlldzoxNzcwODQ) report for an example report with sliders.

<Frame>
  <img alt="StarGAN report with sliders" />
</Frame>

<Frame>
  <img alt="Interactive media panels" />
</Frame>

## Work log: Track what you've tried and plan next steps

Write down your thoughts on experiments, your findings, and any gotchas and next steps as you work through a project, and keep everything organized in one place. This lets you "document" all the important pieces beyond your scripts. See the [Who Is Them? Text Disambiguation With Transformers](https://wandb.ai/stacey/winograd/reports/Who-is-Them-Text-Disambiguation-with-Transformers--VmlldzoxMDU1NTc) report for an example of how you can report your findings.

<Frame>
  <img alt="Text disambiguation report" />
</Frame>

Tell the story of a project, which you and others can reference later to understand how and why a model was developed. See [The View from the Driver's Seat](https://wandb.ai/stacey/deep-drive/reports/The-View-from-the-Driver-s-Seat--Vmlldzo1MTg5NQ) report for how you can report your findings.

<Frame>
  <img alt="Driver's seat project report" />
</Frame>

See [Learning Dexterity End-to-End Using W\&B Reports](https://bit.ly/wandb-learning-dexterity) for an example of how the OpenAI Robotics team uses Reports to run large-scale machine learning projects.


# Tutorial: Create sweep job from project
Source: https://docs.wandb.ai/models/sweeps/existing-project

Tutorial on how to create sweep jobs from a pre-existing W&B project.

This tutorial explains how to create sweep jobs from a pre-existing W\&B project. We will use the [Fashion MNIST dataset](https://github.com/zalandoresearch/fashion-mnist) to train a PyTorch convolutional neural network how to classify images. The required code an dataset is located in the [W\&B examples repository (PyTorch CNN Fashion)](https://github.com/wandb/examples/tree/master/examples/pytorch/pytorch-cnn-fashion)

Explore the results in this [W\&B Dashboard](https://app.wandb.ai/carey/pytorch-cnn-fashion).

## 1. Create a project

First, create a baseline. Download the PyTorch MNIST dataset example model from W\&B examples GitHub repository. Next, train the model. The training script is within the `examples/pytorch/pytorch-cnn-fashion` directory.

1. Clone this repo `git clone https://github.com/wandb/examples.git`
2. Open this example `cd examples/pytorch/pytorch-cnn-fashion`
3. Run a run manually `python train.py`

Optionally explore the example appear in the W\&B App UI dashboard.

[View an example project page →](https://app.wandb.ai/carey/pytorch-cnn-fashion)

## 2. Create a sweep

From your project page, open the [Sweep tab](./visualize-sweep-results) in the project sidebar and select **Create Sweep**.

<Frame>
  <img alt="Sweep overview" />
</Frame>

The auto-generated configuration guesses values to sweep over based on the runs you have completed. Edit the configuration to specify what ranges of hyperparameters you want to try. When you launch the sweep, it starts a new process on the hosted W\&B sweep server. This centralized service coordinates the agents— the machines that are running the training jobs.

<Frame>
  <img alt="Sweep configuration" />
</Frame>

## 3. Launch agents

Next, launch an agent locally. You can launch up to 20 agents on different machines in parallel if you want to distribute the work and finish the sweep job more quickly. The agent will print out the set of parameters it’s trying next.

<Frame>
  <img alt="Launch agents" />
</Frame>

Now you're running a sweep. The following image demonstrates what the dashboard looks like as the example sweep job is running. [View an example project page →](https://app.wandb.ai/carey/pytorch-cnn-fashion)

<Frame>
  <img alt="Sweep dashboard" />
</Frame>

## Seed a new sweep with existing runs

Launch a new sweep using existing runs that you've previously logged.

1. Open your project table.
2. Select the runs you want to use with checkboxes on the left side of the table.
3. Click the dropdown to create a new sweep.

Your sweep will now be set up on our server. All you need to do is launch one or more agents to start running runs.

<Frame>
  <img alt="Seed sweep from runs" />
</Frame>

<Note>
  If you kick off the new sweep as a bayesian sweep, the selected runs will also seed the Gaussian Process.
</Note>


# Initialize a sweep
Source: https://docs.wandb.ai/models/sweeps/initialize-sweeps

Initialize a W&B Sweep using the Python SDK or CLI to start hyperparameter searches with your sweep configuration.

W\&B uses a *Sweep Controller* to manage sweeps on the cloud (standard), locally (local) across one or more machines. After a run completes, the sweep controller will issue a new set of instructions describing a new run to execute. These instructions are picked up by *agents* who actually perform the runs. In a typical W\&B Sweep, the controller lives on the W\&B server. Agents live on *your* machines.

The following code snippets demonstrate how to initialize sweeps with the CLI and within a Jupyter Notebook or Python script.

<Warning>
  1. Before you initialize a sweep, make sure you have a sweep configuration defined either in a YAML file or a nested Python dictionary object in your script. For more information, see [Define sweep configuration](/models/sweeps/define-sweep-configuration/).
  2. Both the W\&B Sweep and the W\&B Run must be in the same project. Therefore, the name you provide when you initialize W\&B ([`wandb.init()`](/models/ref/python/functions/init)) must match the name of the project you provide when you initialize a W\&B Sweep ([`wandb.sweep()`](/models/ref/python/functions/sweep)).
</Warning>

<Tabs>
  <Tab title="Python script or notebook">
    Use the W\&B SDK to initialize a sweep. Pass the sweep configuration dictionary to the `sweep` parameter. Optionally provide the name of the project for the project parameter (`project`) where you want the output of the W\&B Run to be stored. If the project is not specified, the run is put in an "Uncategorized" project.

    ```python theme={null}
    import wandb

    # Example sweep configuration
    sweep_configuration = {
        "method": "random",
        "name": "sweep",
        "metric": {"goal": "maximize", "name": "val_acc"},
        "parameters": {
            "batch_size": {"values": [16, 32, 64]},
            "epochs": {"values": [5, 10, 15]},
            "lr": {"max": 0.1, "min": 0.0001},
        },
    }

    sweep_id = wandb.sweep(sweep=sweep_configuration, project="project-name")
    ```

    The [`wandb.sweep()`](/models/ref/python/functions/sweep) function returns the sweep ID. The sweep ID includes the entity name and the project name. Make a note of the sweep ID.
  </Tab>

  <Tab title="CLI">
    Use the W\&B CLI to initialize a sweep. Provide the name of your configuration file. Optionally provide the name of the project for the `project` flag. If the project is not specified, the W\&B Run is put in an "Uncategorized" project.

    Use the [`wandb sweep`](/models/ref/cli/wandb-sweep) command to initialize a sweep. The following code example initializes a sweep for a `sweeps_demo` project and uses a `config.yaml` file for the configuration.

    ```bash theme={null}
    wandb sweep --project sweeps_demo config.yaml
    ```

    This command will print out a sweep ID. The sweep ID includes the entity name and the project name. Make a note of the sweep ID.
  </Tab>
</Tabs>


# Manage algorithms locally
Source: https://docs.wandb.ai/models/sweeps/local-controller

Search and stop algorithms locally instead of using the W&B cloud-hosted service.

The hyper-parameter controller is hosted by Weights & Biased as a cloud service by default. W\&B agents communicate with the controller to determine the next set of parameters to use for training. The controller is also responsible for running early stopping algorithms to determine which runs can be stopped.

The local controller feature allows you to run search and stop algorithms locally. The local controller gives you the ability to inspect and instrument the code to debug issues and develop new features that can be incorporated into the cloud service.

<Warning>
  This feature is offered to support faster development and debugging of new algorithms for the Sweeps tool. It is not intended for actual hyperparameter optimization workloads.
</Warning>

Before you get start, you must install the W\&B SDK(`wandb`). Type the following code snippet into your command line:

```
pip install wandb sweeps 
```

The following examples assume you already have a configuration file and a training loop defined in a python script or Jupyter Notebook. For more information about how to define a configuration file, see [Define sweep configuration](/models/sweeps/define-sweep-configuration/).

### Run the local controller from the command line

Initialize a sweep similarly to how you normally would when you use hyper-parameter controllers hosted by W\&B as a cloud service. Specify the controller flag (`controller`) to indicate you want to use the local controller for W\&B sweep jobs:

```bash theme={null}
wandb sweep --controller config.yaml
```

Alternatively, you can separate initializing a sweep and specifying that you want to use a local controller into two steps.

To separate the steps, first add the following key-value to your sweep's YAML configuration file:

```yaml theme={null}
controller:
  type: local
```

Next, initialize the sweep:

```bash theme={null}
wandb sweep config.yaml
```

`wandb sweep` generates a sweep ID. After you initialized the sweep, start a controller with [`wandb controller`](/models/ref/python/functions/controller):

```bash theme={null}
wandb controller {user}/{entity}/{sweep_id}
```

Once you have specified you want to use a local controller, start one or more Sweep agents to execute the sweep. Start a W\&B Sweep similar to how you normally would. See [Start sweep agents](/models/sweeps/start-sweep-agents/), for more information.

```bash theme={null}
wandb sweep sweep_ID
```

### Run a local controller with W\&B Python SDK

The following code snippets demonstrate how to specify and use a local controller with the W\&B Python SDK.

The simplest way to use a controller with the Python SDK is to pass the sweep ID to the [`wandb.controller()`](/models/ref/python/functions/controller) method. Next, use the return objects `run` method to start the sweep job:

```python theme={null}
sweep = wandb.controller(sweep_id)
sweep.run()
```

If you want more control of the controller loop:

```python theme={null}
import wandb

sweep = wandb.controller(sweep_id)
while not sweep.done():
    sweep.print_status()
    sweep.step()
    time.sleep(5)
```

Or even more control over the parameters served:

```python theme={null}
import wandb

sweep = wandb.controller(sweep_id)
while not sweep.done():
    params = sweep.search()
    sweep.schedule(params)
    sweep.print_status()
```

If you want to specify your sweep entirely with code you can do something like this:

```python theme={null}
import wandb

sweep = wandb.controller()
sweep.configure_search("grid")
sweep.configure_program("train-dummy.py")
sweep.configure_controller(type="local")
sweep.configure_parameter("param1", value=3)
sweep.create()
sweep.run()
```


# Parallelize agents
Source: https://docs.wandb.ai/models/sweeps/parallelize-agents

Parallelize W&B Sweep agents on multi-core or multi-GPU machine.

Parallelize your W\&B Sweep agents on a multi-core or multi-GPU machine. Before you get started, ensure you have initialized your W\&B Sweep. For more information on how to initialize a W\&B Sweep, see [Initialize sweeps](./initialize-sweeps).

### Parallelize on a multi-CPU machine

Depending on your use case, explore the following tabs to learn how to parallelize W\&B Sweep agents using the CLI or within a Jupyter Notebook.

<Tabs>
  <Tab title="CLI">
    Use the [`wandb agent`](/models/ref/cli/wandb-agent) command to parallelize your sweep agent across multiple CPUs with the terminal. Provide the sweep ID that was returned when you [initialized the sweep](./initialize-sweeps).

    1. Open more than one terminal window on your local machine.
    2. Copy and paste the code snippet below and replace `sweep_id` with your sweep ID:

    ```bash theme={null}
    wandb agent sweep_id
    ```
  </Tab>

  <Tab title="Jupyter Notebook">
    Use the W\&B Python SDK library to parallelize your W\&B Sweep agent across multiple CPUs within Jupyter Notebooks. Ensure you have the sweep ID that was returned when you [initialized the sweep](./initialize-sweeps).  In addition, provide the name of the function the sweep will execute for the `function` parameter:

    1. Open more than one Jupyter Notebook.
    2. Copy and paste the W\&B Sweep ID on multiple Jupyter Notebooks to parallelize a W\&B Sweep. For example, you can paste the following code snippet on multiple jupyter notebooks to parallelize your sweep if you have the sweep ID stored in a variable called `sweep_id` and the name of the function is `function_name`:

    ```python theme={null}
    wandb.agent(sweep_id=sweep_id, function=function_name)
    ```
  </Tab>
</Tabs>

### Parallelize on a multi-GPU machine

Follow the procedure outlined to parallelize your W\&B Sweep agent across multiple GPUs with a terminal using CUDA Toolkit:

1. Open more than one terminal window on your local machine.
2. Specify the GPU instance to use with `CUDA_VISIBLE_DEVICES` when you start a W\&B Sweep job ([`wandb agent`](/models/ref/cli/wandb-agent)). Assign `CUDA_VISIBLE_DEVICES` an integer value corresponding to the GPU instance to use.

For example, suppose you have two NVIDIA GPUs on your local machine. Open a terminal window and set `CUDA_VISIBLE_DEVICES` to `0` (`CUDA_VISIBLE_DEVICES=0`). Replace `sweep_ID` in the following example with the W\&B Sweep ID that is returned when you initialized a W\&B Sweep:

Terminal 1

```bash theme={null}
CUDA_VISIBLE_DEVICES=0 wandb agent sweep_ID
```

Open a second terminal window. Set `CUDA_VISIBLE_DEVICES` to `1` (`CUDA_VISIBLE_DEVICES=1`). Paste the same W\&B Sweep ID for the `sweep_ID` mentioned in the following code snippet:

Terminal 2

```bash theme={null}
CUDA_VISIBLE_DEVICES=1 wandb agent sweep_ID
```


# Manage sweeps
Source: https://docs.wandb.ai/models/sweeps/pause-resume-and-cancel-sweeps

Pause, resume, and cancel a W&B Sweep with the CLI.

Use the [W\&B CLI](/models/ref/cli/wandb-sweep) to pause, resume, and cancel a sweep. The CLI's `sweep` command uses flags such as `--pause` and `--resume` to control the sweep's ability to create new W\&B runs, with different effects on existing runs:

* `--pause`: When you pause a sweep, the agent creates no new runs until you resume the sweep. Existing runs continue to execute normally.
* `--resume`: When you resume a sweep, the agent continues creating new runs according to the search strategy.
* `--stop`: When you stop a sweep, the agent stops creating new runs. Existing runs continue to completion.
* `--cancel`: When you cancel a sweep, the agent immediately kills all currently executing runs and stops creating new runs.

Use the following guidance to pause, resume, and cancel a sweep. In each case, provide the sweep ID that was generated when you initialized a sweep.

### Pause a sweep

Pause a sweep so it temporarily stops creating new runs. Runs that are already executing will continue to run until completion. Use the [`wandb sweep --pause`](/models/ref/cli/wandb-sweep) command to pause a sweep. Provide the sweep ID that you want to pause.

```bash theme={null}
wandb sweep --pause entity/project/sweep_ID
```

### Resume a sweep

Resume a paused sweep with the [`wandb sweep --resume`](/models/ref/cli/wandb-sweep) command. The sweep will start creating new runs again according to its search strategy. Provide the sweep ID that you want to resume:

```bash theme={null}
wandb sweep --resume entity/project/sweep_ID
```

### Stop a sweep

Finish a sweep to stop creating new runs while letting currently executing runs finish gracefully. Use the [`wandb sweep --stop`](/models/ref/cli/wandb-sweep) command:

```bash theme={null}
wandb sweep --stop entity/project/sweep_ID
```

<Warning>
  W\&B does not terminate active [sweeps](/models/sweeps) or agents when you delete a project.
</Warning>

### Cancel a sweep

Cancel a sweep to immediately kill all running runs and stop creating new runs. This is the only sweep command that forcibly terminates existing runs. Runs are terminated abruptly; the running processes have no chance to run user-defined signal handlers. Use the [`wandb sweep --cancel`](/models/ref/cli/wandb-sweep) command to cancel a sweep. Provide the sweep ID that you want to cancel. For more on signals and sweep runs, see [Signal handling and sweep runs](/models/sweeps/signal-handling-sweep-runs).

```bash theme={null}
wandb sweep --cancel entity/project/sweep_ID
```

For a full list of CLI command options, see the [wandb sweep](/models/ref/cli/wandb-sweep) CLI Reference Guide.

<Warning>
  W\&B does not terminate active [sweeps](/models/sweeps) or agents when you delete a project.
</Warning>

## Understanding sweep and run statuses

A sweep orchestrates multiple runs to explore hyperparameter combinations. Understanding how sweep status and run status interact is crucial for effectively managing your hyperparameter optimization.

### Key differences

* **Sweep status** controls whether new runs are created (Running, Paused, Stopped, Cancelled, Finished, Failed, Crashed)
* **Run status** reflects the execution state of individual runs (Pending, Running, Finished, Failed, Crashed, Killed)

### Stop an individual run

When you [stop a run](/models/runs/stop-runs) in a sweep, the sweep agent automatically kicks off the next run in the sweep. This allows you to skip poorly performing configurations without interrupting the sweep's overall progress.

### Best practices

* Use `--pause` instead of cancel when you want to temporarily halt exploration without losing running experiments
* Monitor individual run statuses to identify systematic failures
* Use `--stop` for graceful termination when you've found satisfactory hyperparameters
* Reserve `--cancel` for emergencies when runs are consuming excessive resources or producing errors


# Signal handling and sweep runs
Source: https://docs.wandb.ai/models/sweeps/signal-handling-sweep-runs

Learn how W&B Sweeps handle UNIX signals, exit codes, and preemption in sweep runs.

This page provides details about how W\&B Sweeps handle system signals and process exit codes, to help you run sweeps reliably in preemptible environments such as SLURM, EC2 Spot, or Google Cloud preemptible VMs. These sections explain how to interrupt runs cleanly from the keyboard and give details to help you understand and predict run requeue behavior. For details about how runs are requeued when preempted, see [Resume preemptible Sweeps runs](/models/runs/resuming#resume-preemptible-sweeps-runs).

## Exit status and signals

W\&B uses the training process exit status to decide whether a run is requeued and how run state is recorded.

**Exit code contract:**

* **Exit code 0**: The run is considered to have completed successfully and is not requeued.
* **Non-zero exit code**: The run is treated as failed or preempted. When you use [`mark_preempting()`](/models/ref/python/experiments/run#mark_preempting), W\&B requeues the run so another agent (or the same agent after restart) can resume it.

This applies whether the process exits from a signal handler, from an exception, or from an explicit `sys.exit()` call. Understanding and relying on this contract is vitally important in preemptible or cluster environments.

When the process exits due to a [**catchable** signal](#catchable-signals-and-preemption), your handler can run, call [`wandb.run.mark_preempting()`](/models/ref/python/experiments/run#mark_preempting) if you want the run requeued, perform cleanup (for example, save a checkpoint), then exit with a non-zero code. A common convention is `sys.exit(128 + signum)` for termination by signal. W\&B records that exit code and the same [requeue rules](/models/runs/resuming#resume-preemptible-sweeps-runs) apply. When the process is killed by the operating system kernel with [**`SIGKILL`**](#sigkill-uncatchable), the process cannot run exit hooks, so no final summary is written and the run may appear as crashed or killed; the agent still starts the next run.

## Stale runs and server-side timeouts

If a run neither finishes nor posts new metrics for a long time (on the order of about five minutes), the W\&B server marks the run as **crashed**. That often happens when the training process hangs, stops logging, or is terminated without a clean exit (for example after `SIGKILL`). Logging metrics on a steady cadence or exiting with a defined code helps keep run state aligned with what actually happened.

## Catchable signals and preemption

You can register custom signal handlers in your training script. When a catchable signal is delivered, your handler runs; metrics already sent to W\&B are preserved, and the agent detects the process exit and starts the next run.

**Best practices:**

* Register handlers early (for example, before entering the main training loop).
* In the handler, call [`wandb.run.mark_preempting()`](/models/ref/python/experiments/run#mark_preempting) when you intend the run to be requeued after preemption, perform cleanup (for example, save a checkpoint), then exit with a non-zero code.

The following example registers handlers for `SIGUSR1` (a typical cluster preemption signal) and `SIGTERM`. It leaves `SIGINT` free for interactive use (for example, manual cancellation from the terminal). The handler calls `wandb.run.mark_preempting()` and exits using `128 + signum`:

```python theme={null}
import signal
import sys
import wandb


def signal_handler(signum, frame):
    if wandb.run is not None:
        # Optional: save a model checkpoint, flush buffers, and so on.
        print(f"Preempted with signal: {signal.Signals(signum).name}.")
        wandb.run.mark_preempting()
    sys.exit(128 + signum)


def train():
    signal.signal(signal.SIGUSR1, signal_handler)
    signal.signal(signal.SIGTERM, signal_handler)

    with wandb.init() as run:
        config = wandb.config
        for epoch in range(100):
            # Training step; wandb.log(...) as needed
            pass


if __name__ == "__main__":
    train()
```

## `SIGKILL` (uncatchable)

`SIGKILL` cannot be caught or ignored. The process terminates immediately with no chance to run handlers or atexit callbacks. W\&B cannot write a final summary for the run. The agent still recovers and continues the sweep, but run data for that run is incomplete. Use `SIGKILL` only as a last resort; prefer `SIGTERM` or `SIGINT` when you need graceful shutdown.

## Forwarding signals from agent to child

When you use the [`wandb agent`](/models/ref/cli/wandb-agent) CLI, the agent runs your training script as a **child process**. When you interrupt the **agent** (for example, with Ctrl+C or when a scheduler sends `SIGTERM` to the job), the **child** (training process) does not receive the signal by default; the training script cannot run its handler or call `mark_preempting()`. This is described in [GitHub #3667](https://github.com/wandb/wandb/issues/3667).

To let the child shut down gracefully and call `wandb.run.mark_preempting()` in a handler, run the CLI agent with `--forward-signals`:

```bash theme={null}
wandb agent --forward-signals entity/project/sweep_ID
```

Signal forwarding is **not** supported for [`wandb.agent()`](/models/ref/python/functions/agent) in the Python API. That path runs your training function in a thread, not as a separate child process, so the same forwarding behavior does not apply.

When the CLI agent receives `SIGINT` or `SIGTERM` with forwarding enabled, it relays the signal to the child so your training script's handler can run, call `wandb.run.mark_preempting()` and [`wandb.finish()`](/models/ref/python/experiments/run#finish) with a non-zero exit code if needed, and exit with a non-zero code. If you press Ctrl+C twice on the agent process, the agent receives `SIGTERM` by default. With `--forward-signals`, `SIGINT` can be forwarded to the child so your handler runs.

See the [wandb agent](/models/ref/cli/wandb-agent) CLI reference for details.

## Preemptible clusters like `SLURM`

On preemption, the **training process** must receive the signal, mark the run as preempting, and exit with a non-zero code so the run is requeued. A new agent (or the same agent after the job is requeued) can then resume the run.

**Ensure the training process receives the signal:**

1. **When the scheduler signals the agent**: Run the agent with `wandb agent --forward-signals` so that when the scheduler (or user) sends a signal to the agent, the agent forwards it to the child. The child's handler can then call `wandb.run.mark_preempting()`, [`wandb.finish(exit_code=...)`](/models/ref/python/experiments/run#finish) with a non-zero code, and `sys.exit(128 + signum)` (or another non-zero exit code).
2. **When the scheduler signals the launch script (not the agent directly)**: Have the launch script send the preemption signal directly to the training process. For example, the training script writes its process ID to a file; the launch script traps the cluster signal (for example `SIGUSR1`) and runs `kill -SIGUSR1 $(cat $PID_FILE)` so the training process's handler runs.

**In the training script:** Register a handler for the signal your cluster uses (for example `SIGTERM` or `SIGUSR1`). In the handler, call `wandb.run.mark_preempting()` if a run is active, then finish the run with a non-zero exit code and `sys.exit(128 + signum)` (or another non-zero code) so the run is requeued. See [Resume preemptible Sweeps runs](/models/runs/resuming#resume-preemptible-sweeps-runs) for when runs are requeued and how that interacts with `mark_preempting()`.

**Sweep state:** Run `wandb sweep entity/project/sweep_ID --resume` before starting the agent so the sweep is in resume mode and will hand out requeued runs.

**Multi-agent coordination:** When many agents run at once (such as SLURM array jobs), they can race to claim the same preempted run. This is a known limitation. Stagger agent startup or use external coordination mechanisms like locks to help work around this potential issue.

## `wandb sweep --cancel`

You cancel a sweep using the W\&B API, not an OS signal. Run a command like `wandb sweep --cancel entity/project/sweep_ID`. The server tells the agent to exit, and the agent then terminates running child processes and stops. There can be a short delay (on the order of the agent's API polling interval) before cancellation takes effect.

Cancellation delivers **`SIGKILL`** to runs. Child processes have no chance to run user-defined signal handlers. The same applies when you use the **Cancel** control on the Sweeps UI. Use `--cancel` when you want to stop the entire sweep and mark it cancelled. For graceful shutdown of the current run, send a catchable signal to the run (or use `--forward-signals` with the CLI agent and signal the agent). For graceful sweep completion, use [`wandb sweep --stop`](/models/sweeps/pause-resume-and-cancel-sweeps#stop-a-sweep) instead of `--cancel`.

See [Manage sweeps](/models/sweeps/pause-resume-and-cancel-sweeps) for pause, resume, stop, and cancel options.

## Killing the agent vs the run

If you send a signal to the **agent** process (not the child training process), the agent may exit while the child continues running as an orphan. The orphan may keep printing to your terminal, and the shell may not show a new prompt until you press Enter.

Unless you use `--forward-signals` with the CLI agent, stopping the agent does not guarantee the child training process stops.

To confirm the agent has exited, use an OS command like `ps -p <agent_pid>` or `pgrep -f "wandb agent"` instead of relying on prompt appearance.

## Reference: `mark_preempting()` and final run state

The table below summarizes how run state depends on **when** you call `mark_preempting()` and how the process exits. It assumes you use the [`wandb agent`](/models/ref/cli/wandb-agent) CLI with your training program as a subprocess.

| Scenario                                                         | No `mark_preempting()`           | Signal handler calls `mark_preempting()` and exits non-zero | `mark_preempting()` always called right after `init()` |
| ---------------------------------------------------------------- | -------------------------------- | ----------------------------------------------------------- | ------------------------------------------------------ |
| Run completes normally with exit code 0                          | FINISHED                         | FINISHED                                                    | FINISHED                                               |
| Run fails with non-zero exit code                                | FAILED                           | FAILED                                                      | PREEMPTED                                              |
| Run receives `SIGKILL`                                           | CRASHED after about five minutes | CRASHED after about five minutes (uncatchable)              | PREEMPTED after about five minutes                     |
| Run receives `SIGINT`                                            | KILLED                           | PREEMPTED (with a `SIGINT` handler)                         | PREEMPTED                                              |
| Run receives another signal (for example `SIGTERM` or `SIGUSR1`) | CRASHED after about five minutes | PREEMPTED (with a matching handler)                         | PREEMPTED after about five minutes                     |

If you only call `mark_preempting()` inside a signal handler, you do not cover cases where the handler never runs, such as `SIGKILL`.

If you always call `mark_preempting()` immediately after `wandb.init()`, any failure can be treated as preemption and the run may be requeued repeatedly, including for bugs or bad configuration.

For environments with a well-defined preemption signal, the usual approach is a **signal handler** that calls `mark_preempting()` and exits non-zero, not an unconditional call after `init()`.


# Start a sweep agent
Source: https://docs.wandb.ai/models/sweeps/start-sweep-agents

Start or stop a W&B Sweep Agent on one or more machines.

Start a sweep on one or more agents on one or more machines. Sweep agents use the sweep configuration defined when you [initialize a sweep](/models/sweeps/initialize-sweeps) to explore different hyperparameter combinations. W\&B creates a new run for each hyperparameter combination the sweep agent tries.

See [Manage sweeps](/models/sweeps/pause-resume-and-cancel-sweeps) to learn how to pause, resume, stop, or cancel a sweep.

<Note>
  Before you continue, make sure you:

  * Configure your training script to create and track hyperparameter combinations with W\&B. For more information, see [Add W\&B to your code](./add-w-and-b-to-your-code#training-script-with-w%26b-python-sdk).
  * Have a [configuration file](./define-sweep-configuration) defined for your sweep.
</Note>

The following code snippets demonstrate how to start an agent with the CLI and within a Jupyter Notebook or Python script. For both methods, provide the sweep ID that W\&B returns when you initialized the sweep. The sweep ID has the form:

```bash theme={null}
entity/project/sweep_ID
```

Where:

* `entity`: Your W\&B username or team name.
* `project`: The name of the project where you want W\&B to store the output of the run. If the project is not specified, W\&B puts the run in a project called "Uncategorized".
* `sweep_ID`: The pseudo random, unique ID generated by W\&B.

<Tabs>
  <Tab title="CLI">
    Use the `wandb agent` command to start a sweep. Provide the sweep ID that W\&B returns when you initialized the sweep.

    Copy and paste the code snippet below and replace `sweep_id` with your sweep ID:

    ```bash theme={null}
    wandb agent sweep_id
    ```

    For graceful shutdown when you interrupt the agent (for example, with Ctrl+C), use `wandb agent --forward-signals sweep_id` so the current run receives the signal and can shut down cleanly. See [Signal handling and sweep runs](/models/sweeps/signal-handling-sweep-runs) for details.
  </Tab>

  <Tab title="Python script or notebook">
    Use [`wandb.agent()`](/models/ref/python/functions/agent) to start a sweep. Provide the sweep ID that W\&B returns when you initialized the sweep along with the name of the function that is the entrypoint to your training script.

    Copy and paste the code snippet below and replace `<sweep_id>` with your sweep ID and `<function_name>` with the name of your training function:

    ```python theme={null}
    wandb.agent(sweep_id="<sweep_id>", function="<function_name>")
    ```

    Signal forwarding from the agent to the training run is only supported when you use the CLI (`wandb agent --forward-signals`). It is not supported for `wandb.agent()` in Python because the training function runs in a thread, not as a child process. See [Signal handling and sweep runs](/models/sweeps/signal-handling-sweep-runs) for details.

    See [Python script or notebook tab](/models/sweeps/add-w-and-b-to-your-code#python-script-or-notebook) in Add W\&B to your code for an example of how to set up your training script if you use this method.

    <Warning>
      **Multiprocessing**

      You must wrap your `wandb.agent()` and `wandb.sweep()` calls with `if __name__ == '__main__':` if you use Python standard library's `multiprocessing` or PyTorch's `pytorch.multiprocessing` package. For example:

      ```python theme={null}
      if __name__ == '__main__':
          wandb.agent(sweep_id="<sweep_id>", function="<function>", count="<count>")
      ```

      Wrapping your code with this convention ensures that it is only executed when the script is run directly, and not when it is imported as a module in a worker process.

      See [Python standard library `multiprocessing`](https://docs.python.org/3/library/multiprocessing.html#the-spawn-and-forkserver-start-methods) or [PyTorch `multiprocessing`](https://docs.pytorch.org/docs/stable/notes/multiprocessing.html#asynchronous-multiprocess-training-e-g-hogwild) for more information about multiprocessing. See [https://realpython.com/if-name-main-python/](https://realpython.com/if-name-main-python/) for information about the `if __name__ == '__main__':` convention.
    </Warning>
  </Tab>
</Tabs>

### Limit the number of runs a sweep agent tries

<Warning>
  Random and Bayesian searches will run forever. You must stop the process from the command line, within your python script, or the [Sweeps UI](./visualize-sweep-results).
</Warning>

Specify the number of runs a sweep agent should try. The following code snippets demonstrate how to set a maximum number of [W\&B Runs](/models/ref/python/experiments/run) with the CLI and within a Jupyter Notebook, Python script.

<Tabs>
  <Tab title="CLI">
    First, initialize your sweep with the [`wandb sweep`](/models/ref/cli/wandb-sweep) command. For more information, see [Initialize sweeps](./initialize-sweeps).

    ```
    wandb sweep config.yaml
    ```

    Next, pass an integer value to the count flag to set the maximum number of runs to try.

    ```python theme={null}
    NUM=10
    SWEEPID="dtzl1o7u"
    wandb agent --count $NUM $SWEEPID
    ```
  </Tab>

  <Tab title="Python script or notebook">
    First, initialize your sweep. For more information, see [Initialize sweeps](./initialize-sweeps).

    ```
    sweep_id = wandb.sweep(sweep_config)
    ```

    Next, start the sweep job. Provide the sweep ID generated from sweep initiation. Pass an integer value to the count parameter to set the maximum number of runs to try.

    ```python theme={null}
    sweep_id, count = "dtzl1o7u", 10
    wandb.agent(sweep_id, count=count)
    ```

    <Warning>
      If you start a new run after the sweep agent has finished, within the same script or notebook, then you should call `wandb.teardown()` before starting the new run.
    </Warning>
  </Tab>
</Tabs>


# Sweep configuration options
Source: https://docs.wandb.ai/models/sweeps/sweep-config-keys

Reference for all W&B Sweep configuration keys including method, metric, parameters, early termination, and command.

A sweep configuration consists of nested key-value pairs. Use top-level keys within your sweep configuration to define qualities of your sweep search such as the parameters to search through ([`parameter`](./sweep-config-keys#parameters) key), the methodology to search the parameter space ([`method`](./sweep-config-keys#method) key), and more.

The following table lists top-level sweep configuration keys and a brief description. See the respective sections for more information about each key.

| Top-level keys                        | Description                                                                           |
| ------------------------------------- | ------------------------------------------------------------------------------------- |
| `program`                             | (required) Training script to run                                                     |
| `entity`                              | The entity for this sweep                                                             |
| `project`                             | The project for this sweep                                                            |
| `description`                         | Text description of the sweep                                                         |
| `name`                                | The name of the sweep, displayed in the W\&B UI.                                      |
| [`method`](#method)                   | (required) The search strategy                                                        |
| [`metric`](#metric)                   | The metric to optimize (only used by certain search strategies and stopping criteria) |
| [`parameters`](#parameters)           | (required) Parameter bounds to search                                                 |
| [`early_terminate`](#early_terminate) | Any early stopping criteria                                                           |
| [`command`](#command)                 | Command structure for invoking and passing arguments to the training script           |
| `run_cap`                             | Maximum number of runs for this sweep                                                 |

See the [Sweep configuration](./sweep-config-keys) structure for more information on how to structure your sweep configuration.

## `metric`

Use the `metric` top-level sweep configuration key to specify the name, the goal, and the target metric to optimize.

| Key      | Description                                                                                                                                                                                                                                                                                      |
| -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `name`   | Name of the metric to optimize.                                                                                                                                                                                                                                                                  |
| `goal`   | Either `minimize` or `maximize` (Default is `minimize`).                                                                                                                                                                                                                                         |
| `target` | Goal value for the metric you are optimizing. The sweep does not create new runs when if or when a run reaches a target value that you specify. Active agents that have a run executing (when the run reaches the target) wait until the run completes before the agent stops creating new runs. |

## `parameters`

In your YAML file or Python script, specify `parameters` as a top level key. Within the `parameters` key, provide the name of a hyperparameter you want to optimize. Common hyperparameters include: learning rate, batch size, epochs, optimizers, and more. For each hyperparameter you define in your sweep configuration, specify one or more search constraints.

The following table shows supported hyperparameter search constraints. Based on your hyperparameter and use case, use one of the search constraints below to tell your sweep agent where (in the case of a distribution) or what (`value`, `values`, and so forth) to search or use.

| Search constraint | Description                                                                                                                                                      |
| ----------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `values`          | Specifies all valid values for this hyperparameter. Compatible with `grid`.                                                                                      |
| `value`           | Specifies the single valid value for this hyperparameter. Compatible with `grid`.                                                                                |
| `distribution`    | Specify a probability [distribution](#distribution-options-for-random-and-bayesian-search). See the note following this table for information on default values. |
| `probabilities`   | Specify the probability of selecting each element of `values` when using `random`.                                                                               |
| `min`, `max`      | (`int`or `float`) Maximum and minimum values. If `int`, for `int_uniform` -distributed hyperparameters. If `float`, for `uniform` -distributed hyperparameters.  |
| `mu`              | (`float`) Mean parameter for `normal` - or `lognormal` -distributed hyperparameters.                                                                             |
| `sigma`           | (`float`) Standard deviation parameter for `normal` - or `lognormal` -distributed hyperparameters.                                                               |
| `q`               | (`float`) Quantization step size for quantized hyperparameters.                                                                                                  |
| `parameters`      | Nest other parameters inside a root level parameter.                                                                                                             |

<Note>
  W\&B sets the following distributions based on the following conditions if a [distribution](#distribution-options-for-random-and-bayesian-search) is not specified:

  * `categorical` if you specify `values`
  * `int_uniform` if you specify `max` and `min` as integers
  * `uniform` if you specify `max` and `min` as floats
  * `constant` if you provide a set to `value`
</Note>

## `method`

Specify the hyperparameter search strategy with the `method` key. There are three hyperparameter search strategies to choose from: grid, random, and Bayesian search.

#### Grid search

Iterate over every combination of hyperparameter values. Grid search makes uninformed decisions on the set of hyperparameter values to use on each iteration. Grid search can be computationally costly.

Grid search executes forever if it is searching within in a continuous search space.

#### Random search

Choose a random, uninformed, set of hyperparameter values on each iteration based on a distribution. Random search runs forever unless you stop the process from the command line, within your python script, or [the W\&B App](/models/sweeps/visualize-sweep-results/).

Specify the distribution space with the metric key if you choose random (`method: random`) search.

#### Bayesian search

In contrast to [random](#random-search) and [grid](#grid-search) search, Bayesian models make informed decisions. Bayesian optimization uses a probabilistic model to decide which values to use through an iterative process of testing values on a surrogate function before evaluating the objective function. Bayesian search works well for small numbers of continuous parameters but scales poorly. For more information about Bayesian search, see the [Bayesian Optimization Primer paper](https://web.archive.org/web/20240209053347/https://static.sigopt.com/b/20a144d208ef255d3b981ce419667ec25d8412e2/static/pdf/SigOpt_Bayesian_Optimization_Primer.pdf).

Bayesian search runs forever unless you stop the process from the command line, within your python script, or [the W\&B App](/models/sweeps/visualize-sweep-results/).

### Distribution options for random and Bayesian search

Within the `parameter` key, nest the name of the hyperparameter. Next, specify the `distribution` key and specify a distribution for the value.

The following table lists distributions W\&B supports.

| Value for `distribution` key | Description                                                                                                                                                                              |
| ---------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `constant`                   | Constant distribution. Must specify the constant value (`value`) to use.                                                                                                                 |
| `categorical`                | Categorical distribution. Must specify all valid values (`values`) for this hyperparameter.                                                                                              |
| `int_uniform`                | Discrete uniform distribution on integers. Must specify `max` and `min` as integers.                                                                                                     |
| `uniform`                    | Continuous uniform distribution. Must specify `max` and `min` as floats.                                                                                                                 |
| `q_uniform`                  | Quantized uniform distribution. Returns `round(X / q) * q` where X is uniform. `q` defaults to `1`.                                                                                      |
| `log_uniform`                | Log-uniform distribution. Returns a value `X` between `exp(min)` and `exp(max)`such that the natural logarithm is uniformly distributed between `min` and `max`.                         |
| `log_uniform_values`         | Log-uniform distribution. Returns a value `X` between `min` and `max` such that `log(`X`)` is uniformly distributed between `log(min)` and `log(max)`.                                   |
| `q_log_uniform`              | Quantized log uniform. Returns `round(X / q) * q` where `X` is `log_uniform`. `q` defaults to `1`.                                                                                       |
| `q_log_uniform_values`       | Quantized log uniform. Returns `round(X / q) * q` where `X` is `log_uniform_values`. `q` defaults to `1`.                                                                                |
| `inv_log_uniform`            | Inverse log uniform distribution. Returns `X`, where  `log(1/X)` is uniformly distributed between `min` and `max`.                                                                       |
| `inv_log_uniform_values`     | Inverse log uniform distribution. Returns `X`, where  `log(1/X)` is uniformly distributed between `log(1/max)` and `log(1/min)`.                                                         |
| `normal`                     | Normal distribution. Return value is normally distributed with mean `mu` (default `0`) and standard deviation `sigma` (default `1`).                                                     |
| `q_normal`                   | Quantized normal distribution. Returns `round(X / q) * q` where `X` is `normal`. Q defaults to 1.                                                                                        |
| `log_normal`                 | Log normal distribution. Returns a value `X` such that the natural logarithm `log(X)` is normally distributed with mean `mu` (default `0`) and standard deviation `sigma` (default `1`). |
| `q_log_normal`               | Quantized log normal distribution. Returns `round(X / q) * q` where `X` is `log_normal`. `q` defaults to `1`.                                                                            |

## `early_terminate`

Use early termination (`early_terminate`) to stop poorly performing runs. If early termination occurs, W\&B stops the current run before it creates a new run with a new set of hyperparameter values.

<Note>
  You must specify a stopping algorithm if you use `early_terminate`. Nest the `type` key within `early_terminate` within your sweep configuration.
</Note>

### Stopping algorithm

<Note>
  W\&B currently supports [Hyperband](https://arxiv.org/abs/1603.06560) stopping algorithm.
</Note>

[Hyperband](https://arxiv.org/abs/1603.06560) hyperparameter optimization evaluates if a program should stop or if it should to continue at one or more pre-set iteration counts, called *brackets*.

When a W\&B run reaches a bracket, the sweep compares that run's metric to all previously reported metric values. The sweep terminates the run if the run's metric value is too high (when the goal is minimization) or if the run's metric is too low (when the goal is maximization).

Brackets are based on the number of logged iterations. The number of brackets corresponds to the number of times you log the metric you are optimizing. The iterations can correspond to steps, epochs, or something in between. The numerical value of the step counter is not used in bracket calculations.

<Note>
  Specify either `min_iter` or `max_iter` to create a bracket schedule.
</Note>

| Key        | Description                                                                                                                 |
| ---------- | --------------------------------------------------------------------------------------------------------------------------- |
| `min_iter` | Specify the iteration for the first bracket                                                                                 |
| `max_iter` | Specify the maximum number of iterations.                                                                                   |
| `s`        | Specify the total number of brackets (required for `max_iter`)                                                              |
| `eta`      | Specify the bracket multiplier schedule (default: `3`).                                                                     |
| `strict`   | Enable 'strict' mode that prunes runs aggressively, more closely following the original Hyperband paper. Defaults to false. |

<Note>
  Hyperband checks which [runs](/models/ref/python/experiments/run) to end once every few minutes. The end run timestamp might differ from the specified brackets if your run or iteration are short.
</Note>

## `command`

Modify the format and contents with nested values within the `command` key. You can directly include fixed components such as filenames.

<Note>
  On Unix systems, `/usr/bin/env` ensures that the OS chooses the correct Python interpreter based on the environment.
</Note>

W\&B supports the following macros for variable components of the command:

| Command macro              | Description                                                                                                                                                           |
| -------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `${env}`                   | `/usr/bin/env` on Unix systems, omitted on Windows.                                                                                                                   |
| `${interpreter}`           | Expands to `python`.                                                                                                                                                  |
| `${program}`               | Training script filename specified by the sweep configuration `program` key.                                                                                          |
| `${args}`                  | Hyperparameters and their values in the form `--param1=value1 --param2=value2`.                                                                                       |
| `${args_no_boolean_flags}` | Hyperparameters and their values in the form `--param1=value1` except boolean parameters are in the form `--boolean_flag_param` when `True` and omitted when `False`. |
| `${args_no_hyphens}`       | Hyperparameters and their values in the form `param1=value1 param2=value2`.                                                                                           |
| `${args_json}`             | Hyperparameters and their values encoded as JSON.                                                                                                                     |
| `${args_json_file}`        | The path to a file containing the hyperparameters and their values encoded as JSON.                                                                                   |
| `${envvar}`                | A way to pass environment variables. `${envvar:MYENVVAR}` \_\_ expands to the value of MYENVVAR environment variable. \_\_                                            |


# Sweeps troubleshooting
Source: https://docs.wandb.ai/models/sweeps/troubleshoot-sweeps

Troubleshoot common W&B Sweep issues including CommError, CUDA out of memory, and wandb agent failures.

Troubleshoot common error messages with the guidance suggested.

### `CommError, Run does not exist` and `ERROR Error uploading`

Your W\&B Run ID might be defined if these two error messages are both returned. As an example, you might have a similar code snippet defined somewhere in your Jupyter Notebooks or Python script:

```python theme={null}
wandb.init(id="some-string")
```

You can not set a Run ID for W\&B Sweeps because W\&B automatically generates random, unique IDs for Runs created by W\&B Sweeps.

W\&B Run IDs need to be unique within a project.

We recommend you pass a name to the name parameter when you initialized W\&B, if you want to set a custom name that will appear on tables and graphs. For example:

```python theme={null}
wandb.init(name="a helpful readable run name")
```

### `Cuda out of memory`

Refactor your code to use process-based executions if you see this error message. More specifically, rewrite your code to a Python script. In addition, call the W\&B Sweep Agent from the CLI, instead of the W\&B Python SDK.

As an example, suppose you rewrite your code to a Python script called  `train.py`. Add the name of the training script (`train.py`) to your YAML Sweep configuration file (`config.yaml` in this example):

```yaml theme={null}
program: train.py
method: bayes
metric:
  name: validation_loss
  goal: maximize
parameters:
  learning_rate:
    min: 0.0001
    max: 0.1
  optimizer:
    values: ["adam", "sgd"]
```

Next, add the following to your `train.py` Python script:

```python theme={null}
if _name_ == "_main_":
    train()
```

Navigate to your CLI and initialize a W\&B Sweep with wandb sweep:

```shell theme={null}
wandb sweep config.yaml
```

Make a note of the W\&B Sweep ID that is returned. Next, start the Sweep job with [`wandb agent`](/models/ref/cli/wandb-agent) with the CLI instead of the Python SDK ([`wandb.agent()`](/models/ref/python/functions/agent)). Replace `sweep_ID` in the code snippet below with the Sweep ID that was returned in the previous step:

```shell theme={null}
wandb agent sweep_ID
```

### `anaconda 400 error`

The following error usually occurs when you do not log the metric that you are optimizing:

```shell theme={null}
wandb: ERROR Error while calling W&B API: anaconda 400 error: 
{"code": 400, "message": "TypeError: bad operand type for unary -: 'NoneType'"}
```

Within your YAML file or nested dictionary you specify a key named "metric" to optimize. Ensure that you log (`wandb.Run.log()`) this metric. In addition, ensure you use the *exact* metric name that you defined the sweep to optimize within your Python script or Jupyter Notebook. For more information about configuration files, see [Define sweep configuration](/models/sweeps/define-sweep-configuration/).


# Learn more about sweeps
Source: https://docs.wandb.ai/models/sweeps/useful-resources

Find links to academic papers, example reports, tutorials, and the Sweeps GitHub repository for further learning.

### Academic papers

Li, Lisha, et al. "[Hyperband: A novel bandit-based approach to hyperparameter optimization.](https://arxiv.org/pdf/1603.06560.pdf)" *The   Journal of Machine Learning Research* 18.1 (2017): 6765-6816.

### Sweep Experiments

The following W\&B Reports demonstrate examples of projects that explore hyperparameter optimization with W\&B Sweeps.

* [Drought Watch Benchmark Progress](https://wandb.ai/stacey/droughtwatch/reports/Drought-Watch-Benchmark-Progress--Vmlldzo3ODQ3OQ)
  * Description: Developing the baseline and exploring submissions to the Drought Watch benchmark.
* [Tuning Safety Penalties in Reinforcement Learning](https://wandb.ai/safelife/benchmark-sweeps/reports/Tuning-Safety-Penalties-in-Reinforcement-Learning---VmlldzoyNjQyODM)
  * Description: We examine agents trained with different side effect penalties on three different tasks: pattern creation, pattern removal, and navigation.
* [Meaning and Noise in Hyperparameter Search with W\&B](https://wandb.ai/stacey/pytorch_intro/reports/Meaning-and-Noise-in-Hyperparameter-Search--Vmlldzo0Mzk5MQ) [Stacey Svetlichnaya](https://wandb.ai/stacey)
  * Description: How do we distinguish signal from pareidolia (imaginary patterns)? This article is showcases what is possible with W\&B and aims to inspire further exploration.
* [Who is Them? Text Disambiguation with Transformers](https://wandb.ai/stacey/winograd/reports/Who-is-Them-Text-Disambiguation-with-Transformers--VmlldzoxMDU1NTc)
  * Description: Using Hugging Face to explore models for natural language understanding
* [DeepChem: Molecular Solubility](https://wandb.ai/stacey/deepchem_molsol/reports/DeepChem-Molecular-Solubility--VmlldzoxMjQxMjM)
  * Description: Predict chemical properties from molecular structure with random forests and deep nets.
* [Intro to MLOps: Hyperparameter Tuning](https://wandb.ai/iamleonie/Intro-to-MLOps/reports/Intro-to-MLOps-Hyperparameter-Tuning--VmlldzozMTg2OTk3)
  * Description: Explore why hyperparameter optimization matters and look at three algorithms to automate hyperparameter tuning for your machine learning models.

### selfm-anaged

The following how-to-guide demonstrates how to solve real-world problems with W\&B:

* [Sweeps with XGBoost ](https://github.com/wandb/examples/blob/master/examples/wandb-sweeps/sweeps-xgboost/xgboost_tune.py)
  * Description: How to use W\&B Sweeps for hyperparameter tuning using XGBoost.

### Sweep GitHub repository

W\&B advocates open source and welcome contributions from the community. Find the [W\&B Sweeps GitHub repository](https://github.com/wandb/sweeps). For information on how to contribute to the W\&B open source repo, see the W\&B GitHub [Contribution guidelines](https://github.com/wandb/wandb/blob/main/CONTRIBUTING.md).


# Visualize sweep results
Source: https://docs.wandb.ai/models/sweeps/visualize-sweep-results

Visualize the results of your W&B Sweeps with the W&B App UI.

Visualize the results of your W\&B Sweeps with the W\&B App. Navigate to the [W\&B App](https://wandb.ai/home). Choose the project that you specified when you initialized a sweep. You will be redirected to your project [workspace](/models/track/workspaces/). Select the **Sweep icon** in the project sidebar (broom icon). From the Sweep UI, select the name of your Sweep from the list.

The sweep list shows each sweep's state (**State**), creation time (**Created**), who started it (**Creator**), how many runs finished (**Run count**), and total **Compute time**. For a grid search over a discrete search space, W\&B also shows **Est. Runs** (the expected number of runs). Open a sweep from the list to pause, resume, stop, or kill it from the app. For the same controls with the CLI, see [Manage sweeps](/models/sweeps/pause-resume-and-cancel-sweeps/).

By default, W\&B will automatically create a parallel coordinates plot, a parameter importance plot, and a scatter plot when you start a W\&B Sweep job.

<Frame>
  <img alt="Sweep UI in project sidebar" />
</Frame>

Parallel coordinates charts summarize the relationship between large numbers of hyperparameters and model metrics at a glance. For more information on parallel coordinates plots, see [Parallel coordinates](/models/app/features/panels/parallel-coordinates/).

<Frame>
  <img alt="Example parallel coordinates plot." />
</Frame>

The scatter plot(left) compares the W\&B Runs that were generated during the Sweep. For more information about scatter plots, see [Scatter Plots](/models/app/features/panels/scatter-plot/).

The parameter importance plot(right) lists the hyperparameters that were the best predictors of, and highly correlated to desirable values of your metrics. For more information on parameter importance plots, see [Parameter Importance](/models/app/features/panels/parameter-importance/).

<Frame>
  <img alt="Scatter plot and parameter importance" />
</Frame>

You can alter the dependent and independent values (x and y axis) that are automatically used. Within each panel there is a pencil icon called **Edit panel**. Choose **Edit panel**. A model will appear. Within the modal, you can alter the behavior of the graph.

For more information on all default W\&B visualization options, see [Panels](/models/app/features/panels/). See the [Data Visualization docs](/models/tables/) for information on how to create plots from W\&B Runs that are not part of a W\&B Sweep.


# Tables overview
Source: https://docs.wandb.ai/models/tables

Iterate on datasets and understand model predictions

<CardGroup>
  <ColabLink />

  <TryProductLink />
</CardGroup>

Use W\&B Tables to visualize and query tabular data. For example:

* Compare how different models perform on the same test set
* Identify patterns in your data
* Look at sample model predictions visually
* Query to find commonly misclassified examples

<Frame>
  <img alt="Semantic segmentation predictions table" />
</Frame>

The above image shows a table with semantic segmentation and custom metrics. View this table here in this [sample project from the W\&B ML Course](https://wandb.ai/av-team/mlops-course-001).

## How it works

A Table is a two-dimensional grid of data where each column has a single type of data. Tables support primitive and numeric types, as well as nested lists, dictionaries, and rich media types.

## Log a Table

Log a table with a few lines of code:

* [`wandb.init()`](/models/ref/python/functions/init): Create a [run](/models/runs/) to track results.
* [`wandb.Table()`](/models/ref/python/data-types/table): Create a new table object.
  * `columns`: Set the column names.
  * `data`: Set the contents of the table.
* [`run.log()`](/models/ref/python/experiments/run.md/#method-runlog): Log the table to save it to W\&B.

```python theme={null}
import wandb

with wandb.init(project="table-test") as run:
    my_table = wandb.Table(columns=["a", "b"], data=[["a1", "b1"], ["a2", "b2"]])
    run.log({"Table Name": my_table})
```

## How to get started

* [Quickstart](/models/tables/tables-walkthrough/): Learn to log data tables, visualize data, and query data.
* [Tables Gallery](/models/tables/tables-gallery/): See example use cases for Tables.


# Log tables
Source: https://docs.wandb.ai/models/tables/log_tables

Create and log W&B Tables with different logging modes including immutable, mutable, and incremental using the Python SDK.

Visualize and log tabular data with W\&B Tables. A W\&B Table is a two-dimensional grid of data where each column has a single type of data. Each row represents one or more data points logged to a W\&B [run](/models/runs/). W\&B Tables support primitive and numeric types, as well as nested lists, dictionaries, and rich media types.

A W\&B Table is a specialized [data type](/models/ref/python/data-types/) in W\&B, logged as an [artifact](/models/artifacts/) object.

You [create and log table objects](#create-and-log-a-new-table) using the W\&B Python SDK. When you create a table object, you specify the columns and data for the table and a [mode](#table-logging-modes). The mode determines how the table is logged and updated during your ML experiments.

<Note>
  `INCREMENTAL` mode is supported on W\&B Server v0.70.0 and above.
</Note>

## Create and log a table

1. Initialize a new run with `wandb.init()`.
2. Create a Table object with the [`wandb.Table`](/models/ref/python/data-types/table) Class. Specify the columns and data for the table for the `columns` and `data` parameters, respectively. It is recommended to set the optional `log_mode` parameter to one of the three modes: `IMMUTABLE` (the default), `MUTABLE`, or `INCREMENTAL`. See [Table Logging Modes](#logging-modes) in the next section for more information.
3. Log the table to W\&B with `run.log()`.

The following example shows how to create and log a table with two columns, `a` and `b`, and two rows of data, `["a1", "b1"]` and `["a2", "b2"]`:

```python theme={null}
import wandb

# Start a new run
with wandb.init(project="table-demo") as run:

    # Create a table object with two columns and two rows of data
    my_table = wandb.Table(
        columns=["a", "b"],
        data=[["a1", "b1"], ["a2", "b2"]],
        log_mode="IMMUTABLE"
        )

    # Log the table to W&B
    run.log({"Table Name": my_table})
```

## Logging modes

The [`wandb.Table`](/models/ref/python/data-types/table) `log_mode` parameter determines how a table is logged and updated during your ML experiments. The `log_mode` parameter accepts one of three arguments: `IMMUTABLE`, `MUTABLE`, and `INCREMENTAL`. Each mode has different implications for how a table is logged, how it can be modified, and how it is rendered in the W\&B App.

The following describes the three logging modes, the high-level differences, and common use case for each mode:

| Mode          | Definition                                                                          | Use Cases                                                                                                                                           | Benefits                                                                          |
| ------------- | ----------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------- |
| `IMMUTABLE`   | Once a table is logged to W\&B, you cannot modify it.                               | - Storing tabular data generated at the end of a run for further analysis                                                                           | - Minimal overhead when logged at the end of a run<br />- All rows rendered in UI |
| `MUTABLE`     | After you log a table to W\&B, you can overwrite the existing table with a new one. | - Adding columns or rows to existing tables<br />- Enriching results with new information                                                           | - Capture Table mutations<br />- All rows rendered in UI                          |
| `INCREMENTAL` | Add batches of new rows to a table throughout the machine learning experiment.      | - Adding rows to tables in batches<br /> - Long-running training jobs<br />- Processing large datasets in batches<br />- Monitoring ongoing results | - View updates on UI during training<br />- Ability to step through increments    |

The next sections show example code snippets for each mode along with considerations when to use each mode.

### MUTABLE mode

`MUTABLE` mode updates an existing table by replacing the existing table with a new one. `MUTABLE` mode is useful when you want to add new columns and rows to an existing table in a non iterative process. Within the UI, the table is rendered with all rows and columns, including the new ones added after the initial log.

<Note>
  In `MUTABLE` mode, the table object is replaced each time you log the table. Overwriting a table with a new one is computationally expensive and can be slow for large tables.
</Note>

The following example shows how to create a table in `MUTABLE` mode, log it, and then add new columns to it. The table object is logged three times: once with the initial data, once with the confidence scores, and once with the final predictions.

<Note>
  The following example uses a placeholder function `load_eval_data()` to load data and a placeholder function `model.predict()` to make predictions. You will need to replace these with your own data loading and prediction functions.
</Note>

```python theme={null}
import wandb
import numpy as np

with wandb.init(project="mutable-table-demo") as run:

    # Create a table object with MUTABLE logging mode
    table = wandb.Table(columns=["input", "label", "prediction"],
                        log_mode="MUTABLE")

    # Load data and make predictions
    inputs, labels = load_eval_data() # Placeholder function
    raw_preds = model.predict(inputs) # Placeholder function

    for inp, label, pred in zip(inputs, labels, raw_preds):
        table.add_data(inp, label, pred)

    # Step 1: Log initial data 
    run.log({"eval_table": table})  # Log initial table

    # Step 2: Add confidence scores (e.g. max softmax)
    confidences = np.max(raw_preds, axis=1)
    table.add_column("confidence", confidences)
    run.log({"eval_table": table})  # Add confidence info

    # Step 3: Add post-processed predictions
    # (e.g., thresholded or smoothed outputs)
    post_preds = (confidences > 0.7).astype(int)
    table.add_column("final_prediction", post_preds)
    run.log({"eval_table": table})  # Final update with another column
```

If you only want to add new batches of rows (no columns) incrementally like in a training loop, consider using [`INCREMENTAL` mode](#INCREMENTAL-mode) instead.

### INCREMENTAL mode

In incremental mode, you log batches of rows to a table during the machine learning experiment. This is ideal for monitoring long-running jobs or when working with large tables that would be inefficient to log during the run for updates. Within the UI, the table is updated with new rows as they are logged, allowing you to view the latest data without having to wait for the entire run to finish. You can also step through the increments to view the table at different points in time.

<Note>
  Run workspaces in the W\&B App have a limit of 100 increments. If you log more than 100 increments, only the most recent 100 are shown in the run workspace.
</Note>

The following example creates a table in `INCREMENTAL` mode, logs it, and then adds new rows to it. Note that the table is logged once per training step (`step`).

<Note>
  The following example uses a placeholder function `get_training_batch()` to load data, a placeholder function `train_model_on_batch()` to train the model, and a placeholder function `predict_on_batch()` to make predictions. You will need to replace these with your own data loading, training, and prediction functions.
</Note>

```python theme={null}
import wandb

with wandb.init(project="incremental-table-demo") as run:

    # Create a table with INCREMENTAL logging mode
    table = wandb.Table(columns=["step", "input", "label", "prediction"],
                        log_mode="INCREMENTAL")

    # Training loop
    for step in range(get_num_batches()): # Placeholder function
        # Load batch data
        inputs, labels = get_training_batch(step) # Placeholder function

        # Train and predict
        train_model_on_batch(inputs, labels) # Placeholder function
        predictions = predict_on_batch(inputs) # Placeholder function

        # Add batch data to table
        for input_item, label, prediction in zip(inputs, labels, predictions):
            table.add_data(step, input_item, label, prediction)

        # Log the table incrementally
        run.log({"training_table": table}, step=step)
```

Incremental logging is generally more computationally efficient than logging a new table each time (`log_mode=MUTABLE`). However, the W\&B App may not render all rows in the table if you log a large number of increments. If your goal is to update and view your table data while your run is ongoing and to have all the data available for analysis, consider using two tables. One with `INCREMENTAL` log mode and one with `IMMUTABLE` log mode.

The following example shows how to combine `INCREMENTAL` and `IMMUTABLE` logging modes to achieve this.

```python theme={null}
import wandb

with wandb.init(project="combined-logging-example") as run:

    # Create an incremental table for efficient updates during training
    incr_table = wandb.Table(columns=["step", "input", "prediction", "label"],
                            log_mode="INCREMENTAL")

    # Training loop
    for step in range(get_num_batches()):
        # Process batch
        inputs, labels = get_training_batch(step)
        predictions = model.predict(inputs)

        # Add data to incremental table
        for inp, pred, label in zip(inputs, predictions, labels):
            incr_table.add_data(step, inp, pred, label)

        # Log the incremental update (suffix with -incr to distinguish from final table)
        run.log({"table-incr": incr_table}, step=step)

    # At the end of training, create a complete immutable table with all data
    # Using the default IMMUTABLE mode to preserve the complete dataset
    final_table = wandb.Table(columns=incr_table.columns, data=incr_table.data, log_mode="IMMUTABLE")
    run.log({"table": final_table})
```

In this example, the `incr_table` is logged incrementally (with `log_mode="INCREMENTAL"`) during training. This allows you to log and view updates to the table as new data is processed. At the end of training, an immutable table (`final_table`) is created with all data from the incremental table. The immutable table is logged to preserve the complete dataset for further analysis and it enables you to view all rows in the W\&B App.

## Examples

### Enriching evaluation results with MUTABLE

```python theme={null}
import wandb
import numpy as np

with wandb.init(project="mutable-logging") as run:

    # Step 1: Log initial predictions
    table = wandb.Table(columns=["input", "label", "prediction"], log_mode="MUTABLE")
    inputs, labels = load_eval_data()
    raw_preds = model.predict(inputs)

    for inp, label, pred in zip(inputs, labels, raw_preds):
        table.add_data(inp, label, pred)

    run.log({"eval_table": table})  # Log raw predictions

    # Step 2: Add confidence scores (e.g. max softmax)
    confidences = np.max(raw_preds, axis=1)
    table.add_column("confidence", confidences)
    run.log({"eval_table": table})  # Add confidence info

    # Step 3: Add post-processed predictions
    # (e.g., thresholded or smoothed outputs)
    post_preds = (confidences > 0.7).astype(int)
    table.add_column("final_prediction", post_preds)
    run.log({"eval_table": table})
```

### Resuming runs with INCREMENTAL tables

You can continue logging to an incremental table when resuming a run:

```python theme={null}
# Start or resume a run
resumed_run = wandb.init(project="resume-incremental", id="your-run-id", resume="must")

# Create the incremental table; no need to populate with data from previously logged table
# Increments will be continue to be added to the Table artifact.
table = wandb.Table(columns=["step", "metric"], log_mode="INCREMENTAL")

# Continue logging
for step in range(resume_step, final_step):
    metric = compute_metric(step)
    table.add_data(step, metric)
    resumed_run.log({"metrics": table}, step=step)

resumed_run.finish()
```

<Note>
  Increments are logged to a new table if you turn off summaries on a key used for the incremental table using `wandb.Run.define_metric("<table_key>", summary="none")` or `wandb.Run.define_metric("*", summary="none")`.
</Note>

### Training with INCREMENTAL batch training

```python theme={null}

with wandb.init(project="batch-training-incremental") as run:

    # Create an incremental table
    table = wandb.Table(columns=["step", "input", "label", "prediction"], log_mode="INCREMENTAL")

    # Simulated training loop
    for step in range(get_num_batches()):
        # Load batch data
        inputs, labels = get_training_batch(step)

        # Train the model on this batch
        train_model_on_batch(inputs, labels)

        # Run model inference
        predictions = predict_on_batch(inputs)

        # Add data to the table
        for input_item, label, prediction in zip(inputs, labels, predictions):
            table.add_data(step, input_item, label, prediction)

        # Log the current state of the table incrementally
        run.log({"training_table": table}, step=step)
```


# Export table data
Source: https://docs.wandb.ai/models/tables/tables-download

Export W&B Table data to pandas DataFrames and CSV files for offline analysis and data processing.

Like all W\&B Artifacts, Tables can be converted into pandas dataframes for easy data exporting.

## Convert `table` to `artifact`

First, you'll need to convert the table to an artifact. The easiest way to do this using `artifact.get(table, "table_name")`:

```python theme={null}
# Create and log a new table.
with wandb.init() as r:
    artifact = wandb.Artifact("my_dataset", type="dataset")
    table = wandb.Table(
        columns=["a", "b", "c"], data=[(i, i * 2, 2**i) for i in range(10)]
    )
    artifact.add(table, "my_table")
    wandb.log_artifact(artifact)

# Retrieve the created table using the artifact you created.
with wandb.init() as r:
    artifact = r.use_artifact("my_dataset:latest")
    table = artifact.get("my_table")
```

## Convert `artifact` to Dataframe

Then, convert the table into a dataframe:

```python theme={null}
# Following from the last code example:
df = table.get_dataframe()
```

## Export data

Now you can export using any method dataframe supports:

```python theme={null}
# Converting the table data to .csv
df.to_csv("example.csv", encoding="utf-8")
```

# Next Steps

* Check out the [reference documentation](/models/artifacts/construct-an-artifact/) on `artifacts`.
* Go through our [Tables Walkthrough](/models/tables/tables-walkthrough/) guide.
* Check out the [Dataframe](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.html) reference docs.


# Example tables
Source: https://docs.wandb.ai/models/tables/tables-gallery

Explore example W&B Tables projects for image classification, audio, text analysis, and other use cases.

The following sections highlight some of the ways you can use tables:

### View your data

Log metrics and rich media during model training or evaluation, then visualize results in a persistent database synced to the cloud, or to your [hosting instance](/platform/hosting).

<Frame>
  <img alt="Data browsing table" />
</Frame>

For example, check out this table that shows a [balanced split of a photos dataset](https://wandb.ai/stacey/mendeleev/artifacts/balanced_data/inat_80-10-10_5K/ab79f01e007113280018/files/data_split.table.json).

### Interactively explore your data

View, sort, filter, group, join, and query tables to understand your data and model performance—no need to browse static files or rerun analysis scripts.

<Frame>
  <img alt="Audio comparison" />
</Frame>

For example, see this report on [style-transferred audio](https://wandb.ai/stacey/cshanty/reports/Whale2Song-W-B-Tables-for-Audio--Vmlldzo4NDI3NzM).

### Compare model versions

Quickly compare results across different training epochs, datasets, hyperparameter choices, model architectures etc.

<Frame>
  <img alt="Model comparison" />
</Frame>

For example, see this table that compares [two models on the same test images](https://wandb.ai/stacey/evalserver_answers_2/artifacts/results/eval_Daenerys/c2290abd3d7274f00ad8/files/eval_results.table.json#b6dae62d4f00d31eeebf\$eval_Bob).

### Track every detail and see the bigger picture

Zoom in to visualize a specific prediction at a specific step. Zoom out to see the aggregate statistics, identify patterns of errors, and understand opportunities for improvement. This tool works for comparing steps from a single model training, or results across different model versions.

<Frame>
  <img alt="Tracking experiment details" />
</Frame>

For example, see this example table that analyzes results [after one and then after five epochs on the MNIST dataset](https://wandb.ai/stacey/mnist-viz/artifacts/predictions/baseline/d888bc05719667811b23/files/predictions.table.json#7dd0cd845c0edb469dec).

## Example Projects with W\&B Tables

The following highlight some real W\&B Projects that use W\&B Tables.

### Image classification

Read [Visualize Data for Image Classification](https://wandb.ai/stacey/mendeleev/reports/Visualize-Data-for-Image-Classification--VmlldzozNjE3NjA), follow the [data visualization nature Colab](https://wandb.me/dsviz-nature-colab), or explore the [artifacts context](https://wandb.ai/stacey/mendeleev/artifacts/val_epoch_preds/val_pred_gawf9z8j/2dcee8fa22863317472b/files/val_epoch_res.table.json) to see how a CNN identifies ten types of living things (plants, bird, insects, etc) from [iNaturalist](https://www.inaturalist.org/pages/developers) photos.

<Frame>
  <img alt="Compare the distribution of true labels across two different models predictions." />
</Frame>

### Audio

Interact with audio tables in [Whale2Song - W\&B Tables for Audio](https://wandb.ai/stacey/cshanty/reports/Whale2Song-W-B-Tables-for-Audio--Vmlldzo4NDI3NzM) on timbre transfer. You can compare a recorded whale song with a synthesized rendition of the same melody on an instrument like violin or trumpet. You can also record your own songs and explore their synthesized versions in W\&B with the [audio transfer Colab](http://wandb.me/audio-transfer).

<Frame>
  <img alt="Audio table example" />
</Frame>

### Text

Browse text samples from training data or generated output, dynamically group by relevant fields, and align your evaluation across model variants or experiment settings. Render text as Markdown or use visual diff mode to compare texts. See the [Shakespeare text generation report](https://wandb.ai/stacey/nlg/reports/Visualize-Text-Data-Predictions--Vmlldzo1NzcwNzY) for an example of a character-based RNN.

<Frame>
  <img alt="Doubling the size of the hidden layer yields some more creative prompt completions." />
</Frame>

### Video

Browse and aggregate over videos logged during training to understand your models. Here is an early example using the [SafeLife benchmark](https://wandb.ai/safelife/v1dot2/benchmark) for RL agents seeking to [minimize side effects ](https://wandb.ai/stacey/saferlife/artifacts/video/videos_append-spawn/c1f92c6e27fa0725c154/files/video_examples.table.json)

<Frame>
  <img alt="Browse easily through the few successful agents" />
</Frame>

### Tabular data

View a report on how to [split and pre-process tabular data](https://wandb.ai/dpaiton/splitting-tabular-data/reports/Tabular-Data-Versioning-and-Deduplication-with-Weights-Biases--VmlldzoxNDIzOTA1) with version control and de-duplication.

<Frame>
  <img alt="Tables and Artifacts workflow" />
</Frame>

### Comparing model variants (semantic segmentation)

An [interactive notebook](https://wandb.me/dsviz-cars-demo) and [live example](https://wandb.ai/stacey/evalserver_answers_2/artifacts/results/eval_Daenerys/c2290abd3d7274f00ad8/files/eval_results.table.json#a57f8e412329727038c2\$eval_Ada) of logging Tables for semantic segmentation and comparing different models. Try your own queries [in this Table](https://wandb.ai/stacey/evalserver_answers_2/artifacts/results/eval_Daenerys/c2290abd3d7274f00ad8/files/eval_results.table.json).

<Frame>
  <img alt="Find the best predictions across two models on the same test set" />
</Frame>

### Analyzing improvement over training time

A detailed report on how to [visualize predictions over time](https://wandb.ai/stacey/mnist-viz/reports/Visualize-Predictions-over-Time--Vmlldzo1OTQxMTk) and the accompanying [interactive notebook](https://wandb.me/dsviz-mnist-colab).


# Tutorial: Log tables, visualize and query data
Source: https://docs.wandb.ai/models/tables/tables-walkthrough

Explore how to use W&B Tables with this 5 minute Quickstart.

The following Quickstart demonstrates how to log data tables, visualize data, and query data.

Select the button below to try a PyTorch Quickstart example project on MNIST data.

<ColabLink />

## 1. Log a table

Log a table with W\&B. You can either construct a new table or pass a Pandas Dataframe.

<Tabs>
  <Tab title="Construct a table">
    To construct and log a new Table, you will use:

    * [`wandb.init()`](/models/ref/python/functions/init): Create a [run](/models/runs/) to track results.
    * [`wandb.Table()`](/models/ref/python/data-types/table): Create a new table object.
      * `columns`: Set the column names.
      * `data`: Set the contents of each row.
    * [`wandb.Run.log()`](/models/ref/python/experiments/run.md/#method-runlog): Log the table to save it to W\&B.

    Here's an example:

    ```python theme={null}
    import wandb

    with wandb.init(project="table-test") as run:
        # Create and log a new table.
        my_table = wandb.Table(columns=["a", "b"], data=[["a1", "b1"], ["a2", "b2"]])
        run.log({"Table Name": my_table})
    ```
  </Tab>

  <Tab title="Pandas Dataframe">
    Pass a Pandas Dataframe to `wandb.Table()` to create a new table.

    ```python theme={null}
    import wandb
    import pandas as pd

    df = pd.read_csv("my_data.csv")

    with wandb.init(project="df-table") as run:
        # Create a new table from the DataFrame
        # and log it to W&B.
      my_table = wandb.Table(dataframe=df)
      run.log({"Table Name": my_table})
    ```

    For more information on supported data types, see the [`wandb.Table`](/models/ref/python/data-types/table) in the W\&B API Reference Guide.
  </Tab>
</Tabs>

## 2. Visualize tables in your project workspace

View the resulting table in your workspace.

1. Navigate to your project in the W\&B App.
2. Select the name of your run in your project workspace. A new panel is added for each unique table key.

<Frame>
  <img alt="Sample table logged" />
</Frame>

In this example, `my_table`, is logged under the key `"Table Name"`.

## 3. Compare across model versions

Log sample tables from multiple W\&B Runs and compare results in the project workspace. In this [example workspace](https://wandb.ai/carey/table-test?workspace=user-carey), we show how to combine rows from multiple different versions in the same table.

<Frame>
  <img alt="Cross-run table comparison" />
</Frame>

Use the table filter, sort, and grouping features to explore and evaluate model results.

<Frame>
  <img alt="Table filtering" />
</Frame>


# Visualize and analyze tables
Source: https://docs.wandb.ai/models/tables/visualize-tables

Compare, filter, group, sort, and visualize W&B Tables data in merged or side-by-side views for analysis.

Customize your W\&B Tables to answer questions about your machine learning model's performance, analyze your data, and more.

Interactively explore your data to:

* [Compare two W\&B Tables](#table-comparison-options) logged as artifact versions to analyze changes in your data or model performance.
* Understand higher-level patterns in your data
* [View how values you log to a table change throughout your runs](#visualize-how-values-change-throughout-your-runs).

<Note>
  W\&B Tables posses the following behaviors:

  1. **Stateless in an artifact context**: any table logged alongside an artifact version resets to its default state after you close the browser window
  2. **Stateful in a workspace or report context**: any changes you make to a table in a single run workspace, multi-run project workspace, or Report persists.

  For information on how to save your current W\&B Table view, see [Save your view](#save-your-view).
</Note>

## Table comparison options

Compare two tables with a [merged view](#merged-view) or a [side-by-side view](#side-by-side-view). For example, the image below demonstrates a table comparison of MNIST data.

<Frame>
  <img alt="Training epoch comparison" />
</Frame>

Follow these steps to compare two tables:

1. Go to your project in the W\&B App.
2. Select the artifacts icon in the project sidebar.
3. Select an artifact version.

In the following image we demonstrate a model's predictions on MNIST validation data after each of five epochs ([view interactive example here](https://wandb.ai/stacey/mnist-viz/artifacts/predictions/baseline/d888bc05719667811b23/files/predictions.table.json)).

<Frame>
  <img alt="Click on 'predictions' to view the Table" />
</Frame>

3. Hover over the second artifact version you want to compare in the sidebar and click **Compare** when it appears. For example, in the image below we select a version labeled as "v4" to compare to MNIST predictions made by the same model after 5 epochs of training.

<Frame>
  <img alt="Model prediction comparison" />
</Frame>

### Merged view

Initially you see both tables merged together. The first table selected has index 0 and a blue highlight, and the second table has index 1 and a yellow highlight. [View a live example of merged tables here](https://wandb.ai/stacey/mnist-viz/artifacts/predictions/baseline/d888bc05719667811b23/files/predictions.table.json#7dd0cd845c0edb469dec).

<Frame>
  <img alt="Merged view" />
</Frame>

From the merged view, you can

* **choose the join key**: use the dropdown at the top left to set the column to use as the join key for the two tables. Typically this is the unique identifier of each row, such as the filename of a specific example in your dataset or an incrementing index on your generated samples. Note that it's currently possible to select *any* column, which may yield illegible tables and slow queries.
* **concatenate instead of join**: select "concatenating all tables" in this dropdown to *union all the rows* from both tables into one larger Table instead of joining across their columns
* **reference each Table explicitly**: use 0, 1, and \* in the filter expression to explicitly specify a column in one or both table instances
* **visualize detailed numerical differences as histograms**: compare the values in any cell at a glance

### Side-by-side view

To view the two tables side-by-side, change the first dropdown from "Merge Tables: Table" to "List of: Table" and then update the "Page size" respectively. Here the first Table selected is on the left and the second one is on the right. Also, you can compare these tables vertically as well by clicking on the "Vertical" checkbox.

<Frame>
  <img alt="Side-by-side table view" />
</Frame>

* **compare the tables at a glance**: apply any operations (sort, filter, group) to both tables in tandem and spot any changes or differences quickly. For example, view the incorrect predictions grouped by guess, the hardest negatives overall, the confidence score distribution by true label, etc.
* **explore two tables independently**: scroll through and focus on the side/rows of interest

## Compare artifacts

Compare two W\&B Tables logged as artifact versions to analyze changes in your data or model performance. Use the merged view or side-by-side view to compare the tables.

### Compare tables across time

Log a table in an artifact for each meaningful step of training to analyze model performance over training time. For example, you could log a table at the end of every validation step, after every 50 epochs of training, or any frequency that makes sense for your pipeline. Use the side-by-side view to visualize changes in model predictions.

<Frame>
  <img alt="Training progress comparison" />
</Frame>

For a more detailed walkthrough of visualizing predictions across training time, see the [predictions over time report](https://wandb.ai/stacey/mnist-viz/reports/Visualize-Predictions-over-Time--Vmlldzo1OTQxMTk) and this interactive [notebook example](https://colab.research.google.com/github/wandb/examples/blob/master/colabs/datasets-predictions/W%26B_Tables_Quickstart.ipynb?_gl=1*kf20ui*_gcl_au*OTI3ODM1OTcyLjE3MzE0MzU1NjU.*_ga*ODEyMjQ4MjkyLjE3MzE0MzU1NjU.*_ga_JH1SJHJQXJ*MTczMTcwNTMwNS45LjEuMTczMTcwNTM5My4zMy4wLjA.*_ga_GMYDGNGKDT*MTczMTcwNTMwNS44LjEuMTczMTcwNTM5My4wLjAuMA..).

### Compare tables across model variants

Compare two artifact versions logged at the same step for two different models to analyze model performance across different configurations (hyperparameters, base architectures, and so forth).

For example, compare predictions between a `baseline` and a new model variant, `2x_layers_2x_lr`, where the first convolutional layer doubles from 32 to 64, the second from 128 to 256, and the learning rate from 0.001 to 0.002. From [this live example](https://wandb.ai/stacey/mnist-viz/artifacts/predictions/baseline/d888bc05719667811b23/files/predictions.table.json#2bb3b1d40aa777496b5d\$2x_layers_2x_lr), use the side-by-side view and filter down to the incorrect predictions after 1 (left tab) versus 5 training epochs (right tab).

<Tabs>
  <Tab title="1 training epoch">
    <Frame>
      <img alt="Performance comparison" />
    </Frame>
  </Tab>

  <Tab title="5 training epochs">
    <Frame>
      <img alt="Variant performance comparison" />
    </Frame>
  </Tab>
</Tabs>

## Visualize how values change throughout your runs

View how values you log to a table change throughout your runs with a step slider. Slide the step slider to view the values logged at different steps. For example, you can view how the loss, accuracy, or other metrics change after each run.

The slider uses a key to determine the step value. The default key for the slider is `_step`, a special key that W\&B automatically logs for you. The `_step` key is an integer that increments by 1 each time you call `wandb.Run.log()` in your code.

To add a step slider to a W\&B Table:

1. Navigate to your project's workspace.
2. Click **Add panel** in the top right corner of the workspace.
3. Select **Query panel**.
4. Within the query expression editor, select `runs` and press **Enter** on your keyboard.
5. Click the gear icon to view the settings for the panel.
6. Set **Render As** selector to **Stepper**.
7. Set **Stepper Key** to `_step` or the [key to use as the unit](#custom-step-keys) for the step slider.

The following image shows a query panel with three W\&B runs and the values they logged at step 295.

<Frame>
  <img alt="Step slider feature" />
</Frame>

Within the W\&B App UI you may notice duplicate values for multiple steps. This duplication can occur if multiple runs log the same value at different steps, or if a run does not log values at every step. If a value is missing for a given step, W\&B uses the last value that was logged as the slider key.

### Custom step key

The step key can be any numeric metric that you log in your runs as the step key, such as `epoch` or `global_step`. When you use a custom step key, W\&B maps each value of that key to a step (`_step`) in the run.

This table shows how a custom step key `epoch` maps to `_step` values for three different runs: `serene-sponge`, `lively-frog`, and `vague-cloud`. Each row represents a call to `wandb.Run.log()` at a particular `_step` in a run. The columns show the corresponding epoch values, if any, that were logged at those steps. Some `_step` values are omitted to save space.

The first time `wandb.Run.log()` was called, none of the runs logged an `epoch` value, so the table shows empty values for `epoch`.

| `_step` | vague-cloud (`epoch`) | lively-frog(`epoch`) | serene-sponge (`epoch`) |
| ------- | --------------------- | -------------------- | ----------------------- |
| 1       |                       |                      |                         |
| 2       |                       |                      | 1                       |
| 4       |                       | 1                    | 2                       |
| 5       | 1                     |                      |                         |
| 6       |                       |                      | 3                       |
| 8       |                       | 2                    | 4                       |
| 10      |                       |                      | 5                       |
| 12      |                       | 3                    | 6                       |
| 14      |                       |                      | 7                       |
| 15      | 2                     |                      |                         |
| 16      |                       | 4                    | 8                       |
| 18      |                       |                      | 9                       |
| 20      | 3                     | 5                    | 10                      |

Now, if the slider is set to `epoch = 1`, the following happens:

* `vague-cloud` finds `epoch = 1` and returns the value logged at `_step = 5`
* `lively-frog` finds `epoch = 1` and returns the value logged at `_step = 4`
* `serene-sponge` finds `epoch = 1` and returns the value logged at `_step = 2`

If the slider is set to `epoch = 9`:

* `vague-cloud` also doesn't log `epoch = 9`, so W\&B uses the latest prior value `epoch = 3` and returns the value logged at `_step = 20`
* `lively-frog` doesn’t log `epoch = 9`, but the latest prior value is `epoch = 5` so it returns the value logged at `_step = 20`
* `serene-sponge` finds `epoch = 9` and return the value logged at `_step = 18`

## Save your view

Tables you interact with in the run workspace, project workspace, or a report automatically saves their view state. If you apply any table operations then close your browser, the table retains the last viewed configuration when you next navigate to the table.

<Note>
  Tables you interact with in the artifact context remains stateless.
</Note>

To save a table from a workspace in a particular state, export it to a W\&B Report. To export a table to report:

1. Select the **action (<Icon icon="ellipsis-vertical" />)** menu in the top right corner of your workspace visualization panel.
2. Select either **Share panel** or **Add to report**.

<Frame>
  <img alt="Report sharing options" />
</Frame>

## Examples

These reports highlight the different use cases of W\&B Tables:

* [Visualize Predictions Over Time](https://wandb.ai/stacey/mnist-viz/reports/Visualize-Predictions-over-Time--Vmlldzo1OTQxMTk)
* [How to Compare Tables in Workspaces](https://wandb.ai/stacey/xtable/reports/How-to-Compare-Tables-in-Workspaces--Vmlldzo4MTc0MTA)
* [Image & Classification Models](https://wandb.ai/stacey/mendeleev/reports/Tables-Tutorial-Visualize-Data-for-Image-Classification--VmlldzozNjE3NjA)
* [Text & Generative Language Models](https://wandb.ai/stacey/nlg/reports/Tables-Tutorial-Visualize-Text-Data-Predictions---Vmlldzo1NzcwNzY)
* [Named Entity Recognition](https://wandb.ai/stacey/ner_spacy/reports/Named-Entity-Recognition--Vmlldzo3MDE3NzQ)
* [AlphaFold Proteins](https://wandb.ai/wandb/examples/reports/AlphaFold-ed-Proteins-in-W-B-Tables--Vmlldzo4ODc0MDc)


# Hydra
Source: https://docs.wandb.ai/models/integrations/hydra

Integrate W&B with Hydra to manage complex configurations for ML experiments and log hyperparameters automatically.

> [Hydra](https://hydra.cc) is an open-source Python framework that simplifies the development of research and other complex applications. The key feature is the ability to dynamically create a hierarchical configuration by composition and override it through config files and the command line.

You can continue to use Hydra for configuration management while taking advantage of the power of W\&B.

## Track metrics

Track your metrics as normal with `wandb.init()` and `wandb.Run.log()` . Here, `wandb.entity` and `wandb.project` are defined within a hydra configuration file.

```python theme={null}
import wandb


@hydra.main(config_path="configs/", config_name="defaults")
def run_experiment(cfg):

    with wandb.init(entity=cfg.wandb.entity, project=cfg.wandb.project) as run:
      run.log({"loss": loss})
```

## Track hyperparameters

Hydra uses [omegaconf](https://omegaconf.readthedocs.io/en/2.1_branch/) as the default way to interface with configuration dictionaries. `OmegaConf`'s dictionary are not a subclass of primitive dictionaries so directly passing Hydra's `Config` to `wandb.Run.config` leads to unexpected results on the dashboard. It's necessary to convert `omegaconf.DictConfig` to the primitive `dict` type before passing to `wandb.Run.config`.

```python theme={null}
@hydra.main(config_path="configs/", config_name="defaults")
def run_experiment(cfg):
  with wandb.init(entity=cfg.wandb.entity, project=cfg.wandb.project) as run:
    run.config = omegaconf.OmegaConf.to_container(
        cfg, resolve=True, throw_on_missing=True
    )
    run = wandb.init(entity=cfg.wandb.entity, project=cfg.wandb.project)
    run.log({"loss": loss})
    model = Model(**run.config.model.configs)
```

## Troubleshoot multiprocessing

If your process hangs when started, this may be caused by [this known issue](/models/track/log/distributed-training). To solve this, try to changing wandb's multiprocessing protocol either by adding an extra settings parameter to `wandb.init()` as:

```python theme={null}
wandb.init(settings=wandb.Settings(start_method="thread"))
```

or by setting a global environment variable from your shell:

```bash theme={null}
$ export WANDB_START_METHOD=thread
```

## Optimize hyperparameters

[W\&B Sweeps](/models/sweeps) is a highly scalable hyperparameter search platform, which provides interesting insights and visualization about W\&B experiments with minimal requirements code real-estate. Sweeps integrates seamlessly with Hydra projects with no-coding requirements. The only thing needed is a configuration file describing the various parameters to sweep over as normal.

A simple example `sweep.yaml` file would be:

```yaml theme={null}
program: main.py
method: bayes
metric:
  goal: maximize
  name: test/accuracy
parameters:
  dataset:
    values: [mnist, cifar10]

command:
  - ${env}
  - python
  - ${program}
  - ${args_no_hyphens}
```

Invoke the sweep:

```bash theme={null}
wandb sweep sweep.yaml
```

W\&B automatically creates a sweep inside your project and returns a `wandb agent` command for you to run on each machine you want to run your sweep.

### Pass parameters not present in Hydra defaults

<a aria-label="Pass parameters not present in Hydra defaults" />

Hydra supports passing extra parameters through the command line which aren't present in the default configuration file, by using a `+` before command. For example, you can pass an extra parameter with some value by simply calling:

```bash theme={null}
$ python program.py +experiment=some_experiment
```

You cannot sweep over such `+` configurations similar to what one does while configuring [Hydra Experiments](https://hydra.cc/docs/patterns/configuring_experiments/). To work around this, you can initialize the experiment parameter with a default empty file and use W\&B Sweep to override those empty configs on each call. For more information, read [this W\&B Report](https://wandb.ai/adrishd/hydra-example/reports/Configuring-W-B-Projects-with-Hydra--VmlldzoxNTA2MzQw?galleryTag=posts\&utm_source=fully_connected\&utm_medium=blog\&utm_campaign=hydra).


# OpenAI API
Source: https://docs.wandb.ai/models/integrations/openai-api

Use W&B with the OpenAI API to log and monitor chat completions, fine-tuning jobs, and token usage metrics.

<ColabLink />

Use the W\&B OpenAI API integration to log requests, responses, token counts and model metadata for all OpenAI models, including fine-tuned models.

<Note>
  See the [OpenAI fine-tuning integration](./openai-fine-tuning) to learn how to use W\&B to track your fine-tuning experiments, models, and datasets and share your results with your colleagues.
</Note>

Log your API inputs and outputs you can quickly evaluate the performance of difference prompts, compare different model settings (such as temperature), and track other usage metrics such as token usage.

<Frame>
  <img alt="OpenAI API automatic logging" />
</Frame>

## Install OpenAI Python API library

The W\&B autolog integration works with OpenAI version 0.28.1 and below.

To install OpenAI Python API version 0.28.1, run:

```python theme={null}
pip install openai==0.28.1
```

## Use the OpenAI Python API

### 1. Import autolog and initialise it

First, import `autolog` from `wandb.integration.openai` and initialise it.

```python theme={null}
import os
import openai
from wandb.integration.openai import autolog

autolog({"project": "gpt5"})
```

You can optionally pass a dictionary with argument that `wandb.init()` accepts to `autolog`. This includes a project name, team name, entity, and more. For more information about [`wandb.init()`](/models/ref/python/functions/init), see the API Reference Guide.

### 2. Call the OpenAI API

Each call you make to the OpenAI API is now logged to W\&B automatically.

```python theme={null}
os.environ["OPENAI_API_KEY"] = "XXX"

chat_request_kwargs = dict(
    model="gpt-3.5-turbo",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Who won the world series in 2020?"},
        {"role": "assistant", "content": "The Los Angeles Dodgers"},
        {"role": "user", "content": "Where was it played?"},
    ],
)
response = openai.ChatCompletion.create(**chat_request_kwargs)
```

### 3. View your OpenAI API inputs and responses

Click on the W\&B [run](/models/runs/) link generated by `autolog` in **step 1**. This redirects you to your project workspace in the W\&B App.

Select a run you created to view the trace table, trace timeline and the model architecture of the OpenAI LLM used.

## Turn off autolog

W\&B recommends that you call `disable()` to close all W\&B processes when you are finished using the OpenAI API.

```python theme={null}
autolog.disable()
```

Now your inputs and completions will be logged to W\&B, ready for analysis or to be shared with colleagues.


# OpenAI Fine-Tuning
Source: https://docs.wandb.ai/models/integrations/openai-fine-tuning

Fine-tune OpenAI models with W&B to log training metrics, monitor jobs, and compare model performance over time.

<ColabLink />

Log your OpenAI GPT-3.5 or GPT-4 model's fine-tuning metrics and configuration to W\&B. Utilize the W\&B ecosystem to track your fine-tuning experiments, models, and datasets and share your results with your colleagues.

<Note>
  See the [OpenAI documentation](https://platform.openai.com/docs/guides/fine-tuning/which-models-can-be-fine-tuned) for a list of models that you can fine tune.
</Note>

See the [W\&B Integration](https://developers.openai.com/cookbook/examples/third_party/gpt_finetuning_with_wandb) section in the OpenAI documentation for supplemental information on how to integrate W\&B with OpenAI for fine-tuning.

## Install or update OpenAI Python API

The W\&B OpenAI fine-tuning integration works with OpenAI version 1.0 and above. See the PyPI documentation for the latest version of the [OpenAI Python API](https://pypi.org/project/openai/) library.

To install OpenAI Python API, run:

```python theme={null}
pip install openai
```

If you already have OpenAI Python API installed, you can update it with:

```python theme={null}
pip install -U openai
```

## Sync your OpenAI fine-tuning results

Integrate W\&B with OpenAI's fine-tuning API to log your fine-tuning metrics and configuration to W\&B. To do this, use the `WandbLogger` class from the `wandb.integration.openai.fine_tuning` module.

```python theme={null}
from wandb.integration.openai.fine_tuning import WandbLogger

# Finetuning logic

WandbLogger.sync(fine_tune_job_id=FINETUNE_JOB_ID)
```

<Frame>
  <img alt="OpenAI auto-scan feature" />
</Frame>

### Sync your fine-tunes

Sync your results from your script

```python theme={null}
from wandb.integration.openai.fine_tuning import WandbLogger

# one line command
WandbLogger.sync()

# passing optional parameters
WandbLogger.sync(
    fine_tune_job_id=None,
    num_fine_tunes=None,
    project="OpenAI-Fine-Tune",
    entity=None,
    overwrite=False,
    model_artifact_name="model-metadata",
    model_artifact_type="model",
    **kwargs_wandb_init
)
```

### Reference

| Argument                | Description                                                                                                                                                                                                                                                                                                                                                                                                 |
| ----------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| fine\_tune\_job\_id     | This is the OpenAI Fine-Tune ID which you get when you create your fine-tune job using `client.fine_tuning.jobs.create`. If this argument is None (default), all the OpenAI fine-tune jobs that haven't already been synced will be synced to W\&B.                                                                                                                                                         |
| openai\_client          | Pass an initialized OpenAI client to `sync`. If no client is provided, one is initialized by the logger itself. By default it is None.                                                                                                                                                                                                                                                                      |
| num\_fine\_tunes        | If no ID is provided, then all the unsynced fine-tunes will be logged to W\&B. This argument allows you to select the number of recent fine-tunes to sync. If num\_fine\_tunes is 5, it selects the 5 most recent fine-tunes.                                                                                                                                                                               |
| project                 | W\&B project name where your fine-tune metrics, models, data, etc. will be logged. By default, the project name is "OpenAI-Fine-Tune."                                                                                                                                                                                                                                                                      |
| entity                  | W\&B Username or team name where you're sending runs. By default, your default entity is used, which is usually your username.                                                                                                                                                                                                                                                                              |
| overwrite               | Forces logging and overwrite existing wandb run of the same fine-tune job. By default this is False.                                                                                                                                                                                                                                                                                                        |
| wait\_for\_job\_success | Once an OpenAI fine-tuning job is started it usually takes a bit of time. To ensure that your metrics are logged to W\&B as soon as the fine-tune job is finished, this setting will check every 60 seconds for the status of the fine-tune job to change to `succeeded`. Once the fine-tune job is detected as being successful, the metrics will be synced automatically to W\&B. Set to True by default. |
| model\_artifact\_name   | The name of the model artifact that is logged. Defaults to `"model-metadata"`.                                                                                                                                                                                                                                                                                                                              |
| model\_artifact\_type   | The type of the model artifact that is logged. Defaults to `"model"`.                                                                                                                                                                                                                                                                                                                                       |
| \*\*kwargs\_wandb\_init | Aany additional argument passed directly to [`wandb.init()`](/models/ref/python/functions/init)                                                                                                                                                                                                                                                                                                             |

## Dataset versioning and visualization

### Versioning

The training and validation data that you upload to OpenAI for fine-tuning are automatically logged as W\&B Artifacts for easier version control. Below is an view of the training file in Artifacts. Here you can see the W\&B run that logged this file, when it was logged, what version of the dataset this is, the metadata, and DAG lineage from the training data to the trained model.

<Frame>
  <img alt="W&B Artifacts with training datasets" />
</Frame>

### Visualization

The datasets are visualized as W\&B Tables, which allows you to explore, search, and interact with the dataset. Check out the training samples visualized using W\&B Tables below.

<Frame>
  <img alt="OpenAI data" />
</Frame>

## The fine-tuned model and model versioning

OpenAI gives you an id of the fine-tuned model. Since we don't have access to the model weights, the `WandbLogger` creates a `model_metadata.json` file with all the details (hyperparameters, data file ids, etc.) of the model along with the \`fine\_tuned\_model\`\` id and is logged as a W\&B Artifact.

This model (metadata) artifact can further be linked to a model in the [W\&B Registry](/models/registry/).

<Frame>
  <img alt="OpenAI model metadata" />
</Frame>

## Frequently asked questions

### How do I share my fine-tune results with my team in W\&B?

Log your fine-tune jobs to your team account with:

```python theme={null}
WandbLogger.sync(entity="YOUR_TEAM_NAME")
```

### How can I organize my runs?

Your W\&B runs are automatically organized and can be filtered/sorted based on any configuration parameter such as job type, base model, learning rate, training filename and any other hyper-parameter.

In addition, you can rename your runs, add notes or create tags to group them.

Once you’re satisfied, you can save your workspace and use it to create report, importing data from your runs and saved artifacts (training/validation files).

### How can I access my fine-tuned model?

Fine-tuned model ID is logged to W\&B as artifacts (`model_metadata.json`) as well config.

```python theme={null}
import wandb
    
with wandb.init(project="OpenAI-Fine-Tune", entity="YOUR_TEAM_NAME") as run:
    ft_artifact = run.use_artifact("ENTITY/PROJECT/model_metadata:VERSION")
    artifact_dir = ft_artifact.download()
```

where `VERSION` is either:

* a version number such as `v2`
* the fine-tune id such as `ft-xxxxxxxxx`
* an alias added automatically such as `latest` or manually

You can then access `fine_tuned_model` id by reading the downloaded `model_metadata.json` file.

### What if a fine-tune was not synced successfully?

If a fine-tune was not logged to W\&B successfully, you can use the `overwrite=True` and pass the fine-tune job id:

```python theme={null}
WandbLogger.sync(
    fine_tune_job_id="FINE_TUNE_JOB_ID",
    overwrite=True,
)
```

### Can I track my datasets and models with W\&B?

The training and validation data are logged automatically to W\&B as artifacts. The metadata including the ID for the fine-tuned model is also logged as artifacts.

You can always control the pipeline using low level wandb APIs like `wandb.Artifact`, `wandb.Run.log`, etc. This will allow complete traceability of your data and models.

<Frame>
  <img alt="OpenAI tracking FAQ" />
</Frame>

## Resources

* [OpenAI Fine-tuning Documentation](https://platform.openai.com/docs/guides/fine-tuning/) is very thorough and contains many useful tips
* [Demo Colab](https://wandb.me/openai-colab)
* [How to Fine-Tune Your OpenAI GPT-3.5 and GPT-4 Models with W\&B](https://wandb.me/openai-report) report


# YOLOv5
Source: https://docs.wandb.ai/models/integrations/yolov5

Use the built-in W&B integration in YOLOv5 for experiment tracking, model versioning, and prediction visualization.

<ColabLink />

[Ultralytics' YOLOv5](https://www.ultralytics.com/yolo) ("You Only Look Once") model family enables real-time object detection with convolutional neural networks without all the agonizing pain.

[W\&B](https://wandb.com) is directly integrated into YOLOv5, providing experiment metric tracking, model and dataset versioning, rich model prediction visualization, and more. **It's as easy as running a single `pip install` before you run your YOLO experiments.**

<Note>
  All W\&B logging features are compatible with data-parallel multi-GPU training, such as with [PyTorch DDP](https://pytorch.org/tutorials/intermediate/ddp_tutorial.html).
</Note>

## Track core experiments

Simply by installing `wandb`, you'll activate the built-in W\&B [logging features](/models/track/log/): system metrics, model metrics, and media logged to interactive [Dashboards](/models/track/workspaces/).

```python theme={null}
pip install wandb
git clone https://github.com/ultralytics/yolov5.git
python yolov5/train.py  # train a small network on a small dataset
```

Just follow the links printed to the standard out by wandb.

<Frame>
  <img alt="All these charts and more." />
</Frame>

## Customize the integration

By passing a few simple command line arguments to YOLO, you can take advantage of even more W\&B features.

* If you pass a number to `--save_period`, W\&B saves a [model version](/models/registry/) at the end of every `save_period` epochs. The model version includes the model weights and tags the best-performing model in the validation set.
* Turning on the `--upload_dataset` flag will also upload the dataset for data versioning.
* Passing a number to `--bbox_interval` will turn on [data visualization](../). At the end of every `bbox_interval` epochs, the outputs of the model on the validation set will be uploaded to W\&B.

<Tabs>
  <Tab title="Model Versioning Only">
    ```python theme={null}
    python yolov5/train.py --epochs 20 --save_period 1
    ```
  </Tab>

  <Tab title="Model Versioning and Data Visualization">
    ```python theme={null}
    python yolov5/train.py --epochs 20 --save_period 1 \
      --upload_dataset --bbox_interval 1
    ```
  </Tab>
</Tabs>

<Note>
  Every W\&B account comes with 100 GB of free storage for datasets and models.
</Note>

Here's what that looks like.

<Frame>
  <img alt="Model versioning" />
</Frame>

<Frame>
  <img alt="Data visualization" />
</Frame>

<Note>
  With data and model versioning, you can resume paused or crashed experiments from any device, no setup necessary. Check out [the Colab ](https://wandb.me/yolo-colab) for details.
</Note>


# YOLOX
Source: https://docs.wandb.ai/models/integrations/yolox

Integrate W&B with YOLOX to track object detection model training, log metrics, and visualize detection results.

[YOLOX](https://github.com/Megvii-BaseDetection/YOLOX) is an anchor-free version of YOLO with strong performance for object detection. You can use the YOLOX W\&B integration to turn on logging of metrics related to training, validation, and the system, and you can interactively validate predictions with a single command-line argument.

## Sign up and create an API key

An API key authenticates your machine to W\&B. You can generate an API key from your user profile.

<Note>
  For a more streamlined approach, create an API key by going directly to [User Settings](https://wandb.ai/settings). Copy the newly created API key immediately and save it in a secure location such as a password manager.
</Note>

1. Click your user profile icon in the upper right corner.
2. Select **User Settings**, then scroll to the **API Keys** section.

## Install the `wandb` library and log in

To install the `wandb` library locally and log in:

<Tabs>
  <Tab title="Command Line">
    1. Set the `WANDB_API_KEY` [environment variable](/models/track/environment-variables/) to your API key.

       ```bash theme={null}
       export WANDB_API_KEY=<your_api_key>
       ```

    2. Install the `wandb` library and log in.

       ```shell theme={null}
       pip install wandb

       wandb login
       ```
  </Tab>

  <Tab title="Python">
    ```bash theme={null}
    pip install wandb
    ```

    ```python theme={null}
    import wandb
    wandb.login()
    ```
  </Tab>

  <Tab title="Python notebook">
    ```notebook theme={null}
    !pip install wandb

    import wandb
    wandb.login()
    ```
  </Tab>
</Tabs>

## Log metrics

Use the `--logger wandb` command line argument to turn on logging with wandb. Optionally you can also pass all of the arguments that [`wandb.init()`](/models/ref/python/functions/init) expects; prepend each argument with `wandb-`.

`num_eval_imges` controls the number of validation set images and predictions that are  logged to W\&B tables for model evaluation.

```shell theme={null}
# login to wandb
wandb login

# call your yolox training script with the `wandb` logger argument
python tools/train.py .... --logger wandb \
                wandb-project <project-name> \
                wandb-entity <entity>
                wandb-name <run-name> \
                wandb-id <run-id> \
                wandb-save_dir <save-dir> \
                wandb-num_eval_imges <num-images> \
                wandb-log_checkpoints <bool>
```

## Example

[Example dashboard with YOLOX training and validation metrics ->](https://wandb.ai/manan-goel/yolox-nano/runs/3pzfeom)

<Frame>
  <img alt="YOLOX training dashboard" />
</Frame>

Any questions or issues about this W\&B integration? Open an issue in the [YOLOX repository](https://github.com/Megvii-BaseDetection/YOLOX).


# CLI
Source: https://docs.wandb.ai/models/ref-link-cli





# Public API
Source: https://docs.wandb.ai/models/ref-link-public-api





# Python SDK
Source: https://docs.wandb.ai/models/ref-link-python





# Query Expression Language
Source: https://docs.wandb.ai/models/ref-link-query-panel





# Reports and Workspaces API
Source: https://docs.wandb.ai/models/ref-link-workspaces





# Support: Models
Source: https://docs.wandb.ai/models/support-models





# W&B Weave
Source: https://docs.wandb.ai/weave

Track, test, and improve language model apps with W&B Weave

W\&B Weave is an observability and evaluation platform that helps you track, evaluate, and improve your LLM application. With Weave, you can:

* [Observe and debug](/weave/quickstart) your LLM application
* [Evaluate](/weave/tutorial-eval) your application’s responses using LLM judges and custom scorers

## Get started

The following docs guide you through the basics of how to use Weave's suite of tools.

<CardGroup>
  <Card title="Quickstart: Track LLM inputs & outputs" icon="chart-line" href="/weave/quickstart">
    Start by tracing a basic call to an LLM and reviewing the data in your W\&B account.
  </Card>

  <Card title="Get started evaluating your app" icon="clipboard-check" href="/weave/tutorial-eval">
    Learn how to build an evaluation pipeline using Weave scorers to test and track your application's performance.
  </Card>

  <Card title="Evaluate a RAG application" icon="search" href="/weave/tutorial-rag">
    Build and evaluate RAG applications using Weave with LLM judges to measure retrieval quality.
  </Card>
</CardGroup>

## Install Weave

W\&B Weave provides Python and TypeScript libraries. To install the Weave library, run the following command:

<Tabs>
  <Tab title="Python">
    ```bash theme={null}
    pip install weave
    ```
  </Tab>

  <Tab title="TypeScript">
    ```bash theme={null}
    pnpm install weave
    ```
  </Tab>
</Tabs>

To start using the Weave library, create a [Weights & Biases (W\&B) account](https://wandb.ai) and an [API key at User Settings](https://wandb.ai/settings). The API key allows you to authenticate to your W\&B account and start sending data to it.


# Quickstart: Trace an agent
Source: https://docs.wandb.ai/weave/agents-quickstart

Trace a multi-turn agent with the Weave SDK. Sessions, turns, LLM calls, and tool calls render in the Agents view of your project.

<Note>
  Weave for Agents is in public preview. Features, APIs, and the Agents view UI may change before general availability.
</Note>

[Try in Colab](https://colab.research.google.com/github/wandb/docs/blob/main/weave/cookbooks/source/agents-quickstart.ipynb) · [GitHub source](https://github.com/wandb/docs/blob/main/weave/cookbooks/source/agents-quickstart.ipynb)

The Weave SDK lets you trace agents built with popular SDKs or custom harnesses. This quickstart shows you how to manually integrate Weave into a custom-built multi-turn agent to emit and capture OpenTelemetry spans. For conceptual understanding about Weave for agents, see [Trace your agents](/weave/guides/tracking/trace-agents).

If you're looking to integrate Weave with SDKs or harnesses such as the Claude Agent SDK or Codex, see [Trace agent integrations](/weave/guides/tracking/trace-agent-integrations). Weave autopatches into several agent-building SDKs and agent harnesses for quick integration.

## What you'll learn

By the end of this quickstart, you'll have a working multi-turn agent that emits Weave-compatible OTel spans. You'll also understand how Weave maps sessions, turns, LLM calls, and tool calls onto your agent code so you can apply the same pattern to your own custom agents.

The code in this guide sets up a small research agent that can look things up on Wikipedia. It asks three questions (three turns) and uses the LLM to choose when to search Wikipedia for an answer. Weave records every step (the conversation, each question, each AI response, and each Wikipedia lookup) so you can see what happened in the Weave Agents view.

This guide shows you how to:

* Initialize Weave for agent tracing with `weave.init()`.
* Open a session and a turn with `start_session` / `startSession` and `start_turn` / `startTurn`.
* Wrap LLM calls with `start_llm` / `startLLM` and record usage.
* Wrap tool executions with `start_tool` / `startTool` and record results.
* View the resulting session, turns, and tool calls in the Agents view.

## How the Weave SDK works with agents

The Weave SDK includes a generic OTel ingest system for agents, meaning that Weave can capture information from any OTel span in your agent's code. However, Weave requires special handling of the following spans to render your agent's traces in the Agents view of the Weave UI.

| Concept                    | Python                     | TypeScript                | OTel span               |
| -------------------------- | -------------------------- | ------------------------- | ----------------------- |
| A conversation             | `weave.start_session(...)` | `weave.startSession(...)` | (no span, groups turns) |
| One user or agent exchange | `weave.start_turn(...)`    | `weave.startTurn(...)`    | `invoke_agent`          |
| One LLM API call           | `weave.start_llm(...)`     | `weave.startLLM(...)`     | `chat`                  |
| One tool execution         | `weave.start_tool(...)`    | `weave.startTool(...)`    | `execute_tool`          |

In Python, all four functions work as context managers (`with weave.start_*(...) as obj:`). On exit, they end the span and flush attributes, including on exceptions. In TypeScript, call `.end()` on each returned object. Use `try { ... } finally { obj.end(); }` to guarantee cleanup on exceptions.

Other [GenAI semantic-convention attributes](https://opentelemetry.io/docs/specs/semconv/gen-ai/gen-ai-agent-spans/), such as `gen_ai.usage.*` and `gen_ai.agent.name`, enable additional rendering, but they're optional.

## Prerequisites

* A W\&B account and [API key](https://wandb.ai/authorize).
* An OpenAI API key.
* Python 3.10+ (for the Python examples).
* Node.js 18+ (the TypeScript examples require built-in `fetch`).

## Install packages

Install the following packages into your developer environment:

<CodeGroup>
  ```bash Python theme={null}
  pip install weave openai requests
  ```

  ```bash TypeScript theme={null}
  npm install weave openai
  ```
</CodeGroup>

## Initialize Weave

`weave.init()` authenticates with W\&B and configures the OTel exporter that sends agent spans to the **Agents** view. If the project doesn't exist on your team, Weave creates it the first time you write to it.

<CodeGroup>
  ```python lines Python theme={null}
  import getpass
  import os

  os.environ["WANDB_API_KEY"] = getpass.getpass("Enter your W&B API key: ")
  os.environ["OPENAI_API_KEY"] = getpass.getpass("Enter your OpenAI API key: ")

  TEAM = input("Enter your W&B team name: ")
  PROJECT = input("Enter your W&B project name: ")

  import weave
  weave.init(f"{TEAM}/{PROJECT}")
  ```

  ```typescript lines highlight="4" TypeScript theme={null}
  // Set WANDB_API_KEY, OPENAI_API_KEY in your environment before running this project
  import * as weave from 'weave';

  await weave.init(`[YOUR-TEAM]/[YOUR-PROJECT]`);
  ```
</CodeGroup>

## Define a tool

The following code defines the agent's Wikipedia search tool along with an OpenAI tool schema that specifies when and how to use the tool.

<CodeGroup>
  ```python lines Python theme={null}
  import json
  import requests

  def wikipedia_search(query: str) -> str:
      r = requests.get(
          "https://en.wikipedia.org/w/api.php",
          params={
              "action": "query", "generator": "search", "gsrsearch": query, "gsrlimit": 1,
              "prop": "extracts", "exintro": True, "explaintext": True, "format": "json",
          },
          headers={"User-Agent": "weave-demo"},
      ).json()
      return next(iter(r["query"]["pages"].values()))["extract"]

  wikipedia_tool_schema = {
      "type": "function",
      "function": {
          "name": "wikipedia_search",
          "description": "Search Wikipedia for a topic and return its intro paragraph.",
          "parameters": {
              "type": "object",
              "properties": {"query": {"type": "string"}},
              "required": ["query"],
          },
      },
  }
  ```

  ```typescript lines TypeScript theme={null}
  async function wikipediaSearch(query: string): Promise<string> {
    const url = new URL('https://en.wikipedia.org/w/api.php');
    url.search = new URLSearchParams({
      action: 'query',
      generator: 'search',
      gsrsearch: query,
      gsrlimit: '1',
      prop: 'extracts',
      exintro: 'true',
      explaintext: 'true',
      format: 'json',
    }).toString();
    const res = await fetch(url, { headers: { 'User-Agent': 'weave-demo' } });
    const data = (await res.json()) as {
      query: { pages: Record<string, { extract: string }> };
    };
    return Object.values(data.query.pages)[0].extract;
  }

  const wikipediaToolSchema = {
    type: 'function' as const,
    function: {
      name: 'wikipedia_search',
      description: 'Search Wikipedia for a topic and return its intro paragraph.',
      parameters: {
        type: 'object',
        properties: { query: { type: 'string' } },
        required: ['query'],
      },
    },
  };
  ```
</CodeGroup>

## Run a traced multi-turn agent

With the tool and Weave initialization in place, the next step combines them into a complete agent loop. This loop shows how sessions, turns, LLM calls, and tool calls nest together.

The following example runs three turns in a single session. Each turn:

1. Opens a `chat` span and lets the LLM choose whether to call the tool.
2. If the LLM requests a tool, opens an `execute_tool` span around the call and feeds the result back to the LLM.
3. Opens a second `chat` span to produce the final answer.

<CodeGroup>
  ```python lines highlight="9,11,17,29,42,46,53" Python theme={null}
  from openai import OpenAI

  openai_client = OpenAI()
  MODEL = "gpt-4o-mini"

  def run_turn(history, user_message):
      history.append({"role": "user", "content": user_message})

      with weave.start_turn(user_message=user_message, model=MODEL):
          # LLM call 1: the model might decide to use a tool.
          with weave.start_llm(model=MODEL, provider_name="openai") as llm:
              resp = openai_client.chat.completions.create(
                  model=MODEL, messages=history, tools=[wikipedia_tool_schema],
              )
              msg = resp.choices[0].message
              llm.output(msg.content or "")
              llm.usage = weave.Usage(
                  input_tokens=resp.usage.prompt_tokens,
                  output_tokens=resp.usage.completion_tokens,
              )
              history.append(msg.model_dump(exclude_none=True))

          # If no tool was requested, the first LLM response is the answer.
          if not msg.tool_calls:
              return msg.content

          # Execute each requested tool call.
          for tc in msg.tool_calls:
              with weave.start_tool(
                  name=tc.function.name,
                  arguments=tc.function.arguments,
                  tool_call_id=tc.id,
              ) as tool:
                  tool.result = wikipedia_search(**json.loads(tc.function.arguments))
                  history.append({
                      "role": "tool",
                      "tool_call_id": tc.id,
                      "content": tool.result,
                  })

          # LLM call 2: synthesize the final answer.
          with weave.start_llm(model=MODEL, provider_name="openai") as llm:
              resp = openai_client.chat.completions.create(model=MODEL, messages=history)
              msg = resp.choices[0].message
              llm.output(msg.content)
              llm.usage = weave.Usage(
                  input_tokens=resp.usage.prompt_tokens,
                  output_tokens=resp.usage.completion_tokens,
              )
              history.append({"role": "assistant", "content": msg.content})
              return msg.content

  with weave.start_session(agent_name="research-bot") as session:
      history = []
      for question in [
          "Who founded Anthropic?",
          "What is Claude (the AI assistant)?",
          "Summarize what we discussed in one sentence.",
      ]:
          print(f"USER: {question}")
          print(f"AGENT: {run_turn(history, question)}\n")
  ```

  ```typescript lines highlight="10,13,39,54,76" TypeScript theme={null}
  import OpenAI from 'openai';

  const openaiClient = new OpenAI();
  const MODEL = 'gpt-4o-mini';

  // history is a list of OpenAI chat messages; typed loosely for brevity.
  async function runTurn(history: any[], userMessage: string): Promise<string | null> {
    history.push({ role: 'user', content: userMessage });

    const turn = weave.startTurn({ userMessage, model: MODEL });
    try {
      // LLM call 1: the model might decide to use a tool.
      const llm1 = weave.startLLM({ model: MODEL, providerName: 'openai' });
      let msg;
      try {
        const resp = await openaiClient.chat.completions.create({
          model: MODEL,
          messages: history,
          tools: [wikipediaToolSchema],
        });
        msg = resp.choices[0].message;
        llm1.output(msg.content ?? '');
        llm1.usage = {
          inputTokens: resp.usage?.prompt_tokens,
          outputTokens: resp.usage?.completion_tokens,
        };
        history.push(msg);
      } finally {
        llm1.end();
      }

      // If no tool was requested, the first LLM response is the answer.
      if (!msg.tool_calls?.length) {
        return msg.content ?? null;
      }

      // Execute each requested tool call.
      for (const tc of msg.tool_calls) {
        const tool = weave.startTool({
          name: tc.function.name,
          args: tc.function.arguments,
          toolCallId: tc.id,
        });
        try {
          const { query } = JSON.parse(tc.function.arguments);
          tool.result = await wikipediaSearch(query);
          history.push({ role: 'tool', tool_call_id: tc.id, content: tool.result });
        } finally {
          tool.end();
        }
      }

      // LLM call 2: synthesize the final answer.
      const llm2 = weave.startLLM({ model: MODEL, providerName: 'openai' });
      try {
        const resp = await openaiClient.chat.completions.create({
          model: MODEL,
          messages: history,
        });
        const msg2 = resp.choices[0].message;
        llm2.output(msg2.content ?? '');
        llm2.usage = {
          inputTokens: resp.usage?.prompt_tokens,
          outputTokens: resp.usage?.completion_tokens,
        };
        history.push({ role: 'assistant', content: msg2.content });
        return msg2.content ?? null;
      } finally {
        llm2.end();
      }
    } finally {
      turn.end();
    }
  }

  const session = weave.startSession({ agentName: 'research-bot' });
  try {
    const history: any[] = [];
    for (const question of [
      'Who founded Anthropic?',
      'What is Claude (the AI assistant)?',
      'Summarize what we discussed in one sentence.',
    ]) {
      console.log(`USER: ${question}`);
      console.log(`AGENT: ${await runTurn(history, question)}\n`);
    }
  } finally {
    session.end();
  }
  ```
</CodeGroup>

## See your agent traces in the Agents view

When `weave.init()` runs, it prints a link to your project where you can see:

* A row in the **Agents** tab for `research-bot`.
* One session containing three turns.
* Each turn (`invoke_agent`) with two `chat` spans and an `execute_tool` span nested inside.
* Token counts, latency, model, and the full message exchange on each `chat`.

Click any turn to inspect the inputs, outputs, tool arguments, and tool results.

## Next steps

* Learn how to [trace agents with Weave](/weave/guides/tracking/trace-agents) and what features and options are available in the Weave SDK.
* See [Trace agent integrations](/weave/guides/tracking/trace-agent-integrations) for more options about how to integrate Weave with your agents.


# What is Weave?
Source: https://docs.wandb.ai/weave/concepts/what-is-weave

Learn about W&B Weave and how it helps you build, evaluate, and improve LLM applications

W\&B Weave is an observability and evaluation platform for building reliable LLM applications. Weave helps you understand what your AI application is doing, measure how well it performs, and systematically improve it over time.

Building LLM applications is fundamentally different from traditional software development. LLM outputs are non-deterministic, making debugging harder. Quality is subjective and context-dependent. Small prompt changes can cause unexpected behavior changes. Traditional testing approaches fall short.

## Core capabilities

Weave provides the following core functionality:

* **Visibility** into agent sessions and multi-turn conversations, or into individual function calls and outputs in application code.
* **Systematic evaluation** to measure performance against curated test cases.
* **Version tracking** for prompts, models, and data so you can understand what changed.
* **Experimentation** with different prompt and model comparisons.
* **Feedback collection** to capture human judgments and annotations.
* **Monitoring** in production using guardrails and scorers for LLM safety and quality.

### Agentic tracing

Weave provides agentic observability for the full lifecycle of agent conversations, including sessions, LLM calls, and tool executions.

If you're building an agent, follow the [agent tracing quickstart](weave/agents-quickstart.mdx) or learn to use the Weave SDK to [trace your agents](/weave/guides/tracking/trace-agents).

If you're using a supported third-party agent harness, such as Claude Code or OpenAI Agent SDK, Weave instruments it automatically with no additional code. See [Integrations](/weave/guides/integrations) for all supported frameworks.

### Application tracing

To trace individual function calls, application code, or custom logic, use Weave Ops and Calls. Add one line to any function to track inputs, outputs, cost, token count, and latency.

* Track end-to-end how data flows through your LLM application.
* See source documents used to produce the LLM feedback.
* Drill down into specific prompts and how answers are produced.

To trace individual functions, follow the Weave [Op tracing quickstart](/weave/quickstart) or learn to use the Weave [Ops and Calls](/weave/guides/tracking/tracing).

If you're using a supported third-party agent framework, such as Claude Code, Weave instruments it automatically with no additional code. See [Integrations](/weave/guides/integrations) for all supported frameworks.

### Evaluations

Benchmark and monitor your LLM application's performance with evaluations to iteratively improve quality and reliability.

* Track which model and prompt versions resulted in what performance.
* Define metrics to evaluate responses using one or more scoring functions.
* Compare two or more different evaluations over multiple metrics. Contrast specific samples for their performance.

[Build an evaluation pipeline](/weave/tutorial-eval)

### Version everything

Weave tracks versions of your prompts, datasets, and model configurations. When something breaks, you can see exactly what changed. When something works, you can reproduce it.

[Learn about versioning](/weave/guides/tracking/objects)

### Experiment with prompts and models

Bring your API keys to test prompts and compare responses from different commercial models using the Playground.

[Experiment in the Weave Playground](/weave/guides/tools/playground)

### Collect feedback

Capture human feedback, annotations, and corrections from production use. Use this data to build better test cases and improve your application.

[Collect feedback](/weave/guides/tracking/feedback)

### Monitor production

Score production traffic with the same scorers you use in evaluation. Set up guardrails to catch issues before they reach users.

[Set up guardrails and monitors](/weave/guides/evaluation/monitors)

## Get started with Weave

Weave provides SDKs for Python and TypeScript. Both SDKs support tracing, evaluation, datasets, and the core Weave features. Some advanced features, such as class-based Models and Scorers, aren't available in the Weave TypeScript SDK.

To get started with Weave:

1. Create a W\&B account at [https://wandb.ai/site](https://wandb.ai/site/?utm_source=course\&utm_medium=course\&utm_campaign=weave) and get your API key from [https://wandb.ai/authorize](https://wandb.ai/authorize?utm_source=course\&utm_medium=course\&utm_campaign=weave).
2. Install Weave:

<CodeGroup>
  ```Python Python theme={null}
  pip install weave
  ```

  ```Typescript Typescript theme={null}
  npm install weave
  ```
</CodeGroup>

3. In your script, import Weave and initialize a project. Replace `<your-team>` with your W\&B team name and `<your-project>` with your W\&B project name.

<CodeGroup>
  ```Python Python theme={null}
  import weave
  client = weave.init('<your-team>/<your-project>')
  ```

  ```TypeScript Typescript theme={null}
  import * as weave from 'weave';
  const client = await weave.init('<your-team>/<your-project>');
  ```
</CodeGroup>

You're now ready to use Weave.


# Collect and track datasets
Source: https://docs.wandb.ai/weave/guides/core-types/datasets

Organize, collect, track, and version examples for LLM application evaluation

Weave datasets help you organize, collect, track, and version examples for LLM application evaluation and side-by-side comparison. By following this page, you can build reusable, versioned collections of examples that you and your teammates can use to evaluate, iterate on, and compare LLM application behavior over time. You can create and interact with `Dataset`s programmatically and through the UI.

This page is for engineers and team members who want to manage evaluation data either in code or through the Weave UI. It describes:

* Basic `Dataset` operations in Python and TypeScript and how to get started.
* How to create a `Dataset` in Python and TypeScript from objects such as Weave [calls](../tracking/tracing).
* Available operations on a `Dataset` in the UI.

## Dataset quickstart

The following code samples demonstrate how to perform fundamental `Dataset` operations using Python and TypeScript. Using the SDKs, you can:

* Create a `Dataset`
* Publish the `Dataset`
* Retrieve the `Dataset`
* Access a specific example in the `Dataset`

Select a tab to see Python and TypeScript-specific code.

<Tabs>
  <Tab title="Python">
    ```python lines theme={null}
    import weave
    from weave import Dataset
    # Initialize Weave
    weave.init('intro-example')

    # Create a dataset
    dataset = Dataset(
        name='grammar',
        rows=[
            {'id': '0', 'sentence': "He no likes ice cream.", 'correction': "He doesn't like ice cream."},
            {'id': '1', 'sentence': "She goed to the store.", 'correction': "She went to the store."},
            {'id': '2', 'sentence': "They plays video games all day.", 'correction': "They play video games all day."}
        ]
    )

    # Publish the dataset
    weave.publish(dataset)

    # Retrieve the dataset
    dataset_ref = weave.ref('grammar').get()

    # Access a specific example
    example_label = dataset_ref.rows[2]['sentence']
    ```
  </Tab>

  <Tab title="TypeScript">
    ```typescript lines theme={null}
    import * as weave from 'weave';

    // Initialize Weave
    const client = await weave.init('intro-example');

    // Create a dataset
    const dataset = new weave.Dataset({
        name: 'grammar',
        rows: [
            {id: '0', sentence: "He no likes ice cream.", correction: "He doesn't like ice cream."},
            {id: '1', sentence: "She goed to the store.", correction: "She went to the store."},
            {id: '2', sentence: "They plays video games all day.", correction: "They play video games all day."}
        ]
    });

    // Publish the dataset
    const ref = await dataset.save();

    // Retrieve the dataset
    const retrievedDataset = await client.get(ref);

    // Alternatively, retrieve using a URI string
    const datasetUri = 'weave:///my-entity/intro-example/object/grammar:abc123def456';
    const refFromUri = weave.ObjectRef.fromUri(datasetUri);
    const retrievedDatasetFromUri = await client.get(refFromUri);

    // Access a specific example
    const exampleLabel = retrievedDataset.getRow(2).sentence;
    ```
  </Tab>
</Tabs>

## Create a dataset from other objects

This section shows how to build a `Dataset` from data you already have, such as recorded Weave calls or existing tabular data, so you don't have to manually re-enter examples.

<Tabs>
  <Tab title="Python">
    In Python, `Dataset`s can also be constructed from common Weave objects like [calls](../tracking/tracing), and Python objects like `pandas.DataFrame`s. This feature is useful if you want to create an example `Dataset` from specific examples.

    ### Weave call

    To create a `Dataset` from one or more Weave calls, retrieve the call objects and add them to a list in the `from_calls` method.

    ```python lines theme={null}
    @weave.op
    def model(task: str) -> str:
        return f"Now working on {task}"

    res1, call1 = model.call(task="fetch")
    res2, call2 = model.call(task="parse")

    dataset = Dataset.from_calls([call1, call2])
    # Now you can use the dataset to evaluate the model, etc.
    ```

    ### Pandas DataFrame

    To create a `Dataset` from a Pandas `DataFrame` object, use the `from_pandas` method. To convert the `Dataset` back, use `to_pandas`.

    ```python lines theme={null}
    import pandas as pd

    df = pd.DataFrame([
        {'id': '0', 'sentence': "He no likes ice cream.", 'correction': "He doesn't like ice cream."},
        {'id': '1', 'sentence': "She goed to the store.", 'correction': "She went to the store."},
        {'id': '2', 'sentence': "They plays video games all day.", 'correction': "They play video games all day."}
    ])
    dataset = Dataset.from_pandas(df)
    df2 = dataset.to_pandas()

    assert df.equals(df2)
    ```

    ### Hugging Face Datasets

    To create a `Dataset` from a Hugging Face `datasets.Dataset` or `datasets.DatasetDict` object, first ensure you have the necessary dependencies installed:

    ```bash theme={null}
    pip install weave[huggingface]
    ```

    Then, use the `from_hf` method. If you provide a `DatasetDict` with multiple splits (like `train`, `test`, `validation`), Weave automatically uses the `train` split and issues a warning. If the `train` split isn't present, Weave raises an error. You can provide a specific split directly (for example, `hf_dataset_dict['test']`).

    To convert a `weave.Dataset` back to a Hugging Face `Dataset`, use the `to_hf` method.

    ```python lines theme={null}
    # Ensure datasets is installed: pip install datasets
    from datasets import Dataset as HFDataset, DatasetDict

    # Example with HF Dataset
    hf_rows = [
        {'id': '0', 'sentence': "He no likes ice cream.", 'correction': "He doesn't like ice cream."},
        {'id': '1', 'sentence': "She goed to the store.", 'correction': "She went to the store."},
    ]
    hf_ds = HFDataset.from_list(hf_rows)
    weave_ds_from_hf = Dataset.from_hf(hf_ds)

    # Convert back to HF Dataset
    converted_hf_ds = weave_ds_from_hf.to_hf()

    # Example with HF DatasetDict (uses 'train' split by default)
    hf_dict = DatasetDict({
        'train': HFDataset.from_list(hf_rows),
        'test': HFDataset.from_list([{'id': '2', 'sentence': "Test sentence", 'correction': "Test correction"}])
    })
    # This issues a warning and uses the 'train' split
    weave_ds_from_dict = Dataset.from_hf(hf_dict)

    # Providing a specific split
    weave_ds_from_test_split = Dataset.from_hf(hf_dict['test'])
    ```
  </Tab>

  <Tab title="TypeScript">
    ```plaintext theme={null}
     This feature isn't currently available in TypeScript yet.
    ```
  </Tab>
</Tabs>

## Create, edit, and delete a dataset in the UI

You can create, edit, and delete `Dataset`s in the UI. Creating datasets in the Weave UI lets you and non-engineering members of your team develop and curate sharable datasets containing examples, questions, and other agent-testing data without editing code.

The following procedures walk through each of these tasks in the UI. Use them when you want to manage evaluation data alongside the traces it came from, rather than from a notebook or script.

### Create a new dataset

The following procedure creates a new `Dataset` from one or more existing calls in your Weave project. After you complete it, you have a published `Dataset` you can reference in evaluations and share with your team.

1. Navigate to the Weave project you want to edit.

2. In the sidebar, select **Traces**.

3. Select one or more calls to create a new `Dataset` for.

4. In the upper right-hand menu, click the **Add selected rows to a dataset** icon (located next to the trashcan icon).

5. From the **Choose a dataset** dropdown, select **Create new**. The **Dataset name** field appears.

6. In the **Dataset name** field, enter a name for your dataset. Options to **Configure dataset fields** appear.

   <Note>
     Dataset names must start with a letter or number and can only contain letters, numbers, hyphens, and underscores.
   </Note>

7. Optional: In **Configure dataset fields**, select the fields from your calls to include in the dataset.
   * You can customize the column names for each selected field.
   * You can select a subset of fields to include in the new `Dataset`, or deselect all fields.

8. Once you've configured the dataset fields, click **Next**. A preview of your new `Dataset` appears.

9. Optional: Click any of the editable fields in your **Dataset** to edit the entry.

10. Click **Create dataset**. Weave creates your new dataset.

11. In the confirmation popup, click **View the dataset** to view the new `Dataset`. Alternatively, go to the **Datasets** tab.

### Edit a dataset

Use the following procedure to add new rows to an existing `Dataset` and publish a new version. Editing in the UI is helpful when you want to extend or correct evaluation data without changing code.

1. Navigate to the Weave project containing the `Dataset` you want to edit.

2. From the sidebar, select **Datasets**. Your available `Dataset`s display.

   <img alt="Dataset UI" />

3. In the **Object** column, click the name and version of the `Dataset` you want to edit. A pop-out modal showing `Dataset` information like name, version, author, and `Dataset` rows displays.

   <img alt="View Dataset information" />

4. In the upper right-hand corner of the modal, click the **Edit dataset** button (the pencil icon). An **+ Add row** button displays at the bottom of the modal.

   <img alt="Dataset UI- Add row icon" />

5. Click **+ Add row**. A new row displays at the top of your existing `Dataset` rows, indicating that you can add a new row to the `Dataset`.

   <img alt="Dataset UI" />

6. To add data to a new row, click the desired column within that row. You can't edit the default **id** column in a `Dataset` row, because Weave assigns it automatically upon creation. An editing modal appears with **Text**, **Code**, and **Diff** options for formatting.

   <img alt="Dataset UI - Add data to a column and format." />

7. Repeat step 6 for each column that you want to add data to in the new row.

   <img alt="Dataset UI - Add data to all columns." />

8. Repeat step 5 for each row that you want to add to the `Dataset`.

9. When you're done editing, publish your `Dataset` by clicking **Publish** in the upper right-hand corner of the modal. Alternatively, if you don't want to publish your changes, click **Cancel**.

   <img alt="Dataset UI - Publish or cancel." />

   Once published, the new version of the `Dataset` with updated rows is available in the UI.

   <img alt="Dataset UI - Published metadata." />

   <img alt="Dataset UI - Published rows." />

### Delete a dataset

Use the following procedure when you want to remove a `Dataset` you no longer need from your Weave project.

1. Navigate to the Weave project containing the `Dataset` you want to edit.

2. From the sidebar, select **Datasets**. Your available `Dataset`s display.

3. In the **Object** column, click the name and version of the `Dataset` you want to delete. A pop-out modal showing `Dataset` information like name, version, author, and `Dataset` rows displays.

4. In the upper right-hand corner of the modal, click the trashcan icon.

   A pop-up modal prompting you to confirm `Dataset` deletion displays.

   <img alt="Dataset UI - Confirm deletion modal." />

5. In the pop-up modal, click **Delete** to delete the `Dataset`. Alternatively, click **Cancel** if you don't want to delete the `Dataset`.

   The `Dataset` is deleted and no longer visible in the **Datasets** tab in your Weave dashboard.

### Add a new agent trace to a dataset

To add agent traces such as sessions, turns, and tool calls to a `Dataset`:

1. Navigate to [wandb.ai](http://wandb.ai) and select your project.

2. In the Weave project sidebar, click **Agents**.

3. In the top tab bar, click **Conversations**.

4. In the Conversations table, select a conversation row to load the conversation details side panel. The conversation details side panel has a header with an agent name and a UUID.

5. In the conversation details side panel header, click the table icon to the right to open the **Add example to dataset** drawer.

   <img alt="Agent conversation details panel header showing table icon to add items to a dataset." />

6. In **Choose a dataset**, use the dropdown to choose the dataset you want to add traces to.

7. In **Select context**, select which messages to add to the dataset. Click **Next**.

8. Review your selections and click **Add to dataset**.

### Add a new trace to a dataset

To add traces generated from ops and calls to a `Dataset`:

1. Navigate to the Weave project you want to edit.

2. In the sidebar, select **Traces**.

3. Select one or more calls with `Datasets` for which you want to create new examples.

4. In the upper right-hand menu, click the **Add selected rows to a dataset** icon (located next to the trashcan icon). Optionally, toggle **Show latest versions** to off to display all versions of all available datasets.

5. From the **Choose a dataset** dropdown, select the `Dataset` you want to add examples to. Options to **Configure field mapping** display.

6. Optional: In **Configure field mapping**, you can adjust the mapping of fields from your calls to the corresponding dataset columns.

7. Once you've configured field mappings, click **Next**. A preview of your new `Dataset` appears.

8. In the empty row (green), add your new example values. The **id** field isn't editable, and Weave creates it automatically.

9. Click **Add to dataset**. Alternatively, to return to the **Configure field mapping** screen, click **Back**.

10. In the confirmation popup, click **View the dataset** to see the changes. Alternatively, navigate to the **Datasets** tab to view the updates to your `Dataset`.

## Other dataset operations

This section covers additional SDK operations that are useful once you already have a `Dataset` to work with.

<Tabs>
  <Tab title="Python">
    ### Select rows

    You can select specific rows from a `Dataset` by their index using the `select` method. This is useful for creating subsets of your data, such as when you want to evaluate against a smaller slice of examples.

    ```python lines theme={null}
    import weave
    from weave import Dataset

    # Create a sample dataset
    dataset = Dataset(rows=[
        {'col_a': 1, 'col_b': 'x'},
        {'col_a': 2, 'col_b': 'y'},
        {'col_a': 3, 'col_b': 'z'},
        {'col_a': 4, 'col_b': 'w'},
    ])

    # Select rows at index 0 and 2
    subset_dataset = dataset.select([0, 2])

    # Now subset_dataset contains only the first and third rows
    # print(list(subset_dataset))
    # Output: [{'col_a': 1, 'col_b': 'x'}, {'col_a': 3, 'col_b': 'z'}]
    ```
  </Tab>

  <Tab title="TypeScript">
    ```plaintext theme={null}
     This feature isn't currently available in TypeScript yet.
    ```
  </Tab>
</Tabs>


# Evaluations overview
Source: https://docs.wandb.ai/weave/guides/core-types/evaluations

Evaluation-driven LLM application development to systematically improve applications

*Evaluation-driven LLM application development* helps you systematically improve LLM applications by measuring their behavior using consistent, curated examples.

This guide shows you how to set up and run evaluations in Weave so you can measure your LLM application's performance against repeatable test cases, compare changes over time, and identify regressions. It's intended for developers building LLM applications who want to move beyond ad hoc testing into a structured, measurable workflow.

<Tabs>
  <Tab title="Python">
    In Weave, the core of the workflow is the *`Evaluation` object*, which defines:

    * A [`Dataset`](../core-types/datasets) or list of dictionaries for test examples.
    * One or more [scoring functions](../evaluation/scorers).
    * Optional configuration like [input preprocessing](#format-dataset-rows-before-evaluating).

    Once you've defined the `Evaluation`, you can run it against a [`Model`](../core-types/models) object or any custom function containing LLM application logic. Each call to `.evaluate()` triggers an *evaluation run*. Think of the `Evaluation` object as a blueprint, and each run as a measurement of how your application performs under that setup.
  </Tab>

  <Tab title="TypeScript">
    In Weave, the core of the workflow is the *`Evaluation` object*, which defines:

    * A [`Dataset`](../core-types/datasets) or array of objects for test examples.
    * One or more [scoring functions](../evaluation/scorers).

    Once you've defined the `Evaluation`, you can run it against any function wrapped with `weave.op`. Each call to `.evaluate()` triggers an *evaluation run*. Think of the `Evaluation` object as a blueprint, and each run as a measurement of how your application performs under that setup.

    <Note>
      The TypeScript SDK uses function-based models and scorers. Class-based `Model` and `Scorer` types are not yet available in TypeScript.
    </Note>
  </Tab>
</Tabs>

To get started with evaluations, complete the following steps:

1. [Create an `Evaluation` object](#create-an-evaluation-object)
2. [Define a dataset of examples](#define-a-dataset-of-test-examples)
3. [Define scoring functions](#define-scoring-functions)
4. [Define a model or function to evaluate](#define-a-model-or-function-to-evaluate)
5. [Run the evaluation](#run-the-evaluation)

After completing these steps, you'll have a reusable evaluation configuration and an evaluation run whose results you can inspect in the Weave UI. For a complete evaluation code sample, see the [full evaluation code example](#full-evaluation-code-example). You can also learn more about [advanced evaluation features](#advanced-evaluation-usage) like [Saved views](#saved-views) and [Imperative evaluations](#imperative-evaluations-evaluationlogger).

## Create an `Evaluation` object

Creating an `Evaluation` object is the first step in setting up your evaluation configuration. An `Evaluation` consists of example data, scoring logic, and optional preprocessing. You later use it to run one or more evaluations.

Weave takes each example, passes it through your application, and scores the output on multiple custom scoring functions. This gives you a view of the performance of your application, and a rich UI to drill into individual outputs and scores.

Custom naming is optional but helps you distinguish between different evaluation configurations and runs in the Weave UI. If you only have one evaluation, you can skip this section.

### Optional: Custom naming

<Tabs>
  <Tab title="Python">
    There are two types of customizable names in the evaluation flow:

    * [*Evaluation object name* (`evaluation_name`)](#name-the-evaluation-object): A persistent label for your configured `Evaluation` object.
    * [*Evaluation run display name* (`__weave["display_name"]`)](#name-individual-evaluation-runs): A label for a specific evaluation execution, shown in the UI.

    #### Name the `Evaluation` object

    To name the `Evaluation` object itself, pass an `evaluation_name` parameter to the `Evaluation` class. This name helps you identify the `Evaluation` in code and UI listings.

    ```python lines theme={null}
    evaluation = Evaluation(
        dataset=examples, scorers=[match_score1], evaluation_name="My Evaluation"
    )
    ```

    #### Name individual evaluation runs

    To name a specific evaluation run (a call to `evaluate()`), use the `__weave` dictionary with a `display_name`. This affects what's shown in the UI for that run.

    ```python lines theme={null}
    evaluation = Evaluation(
        dataset=examples, scorers=[match_score1]
    )
    evaluation.evaluate(model, __weave={"display_name": "My Evaluation Run"})
    ```
  </Tab>

  <Tab title="TypeScript">
    To name the `Evaluation` object, pass an `id` parameter to the `Evaluation` constructor. This name helps you identify the `Evaluation` in code and UI listings.

    ```typescript lines theme={null}
    const evaluation = new weave.Evaluation({
      id: 'my-evaluation',
      dataset: dataset,
      scorers: [matchScore],
    });
    ```
  </Tab>
</Tabs>

## Define a dataset of test examples

First, define a [Dataset](../core-types/datasets) object or list of examples with a collection of examples to evaluate. These examples are often failure cases that you want to test for, similar to unit tests in Test-Driven Development (TDD).

<Tabs>
  <Tab title="Python">
    The following example shows a dataset defined as a list of dictionaries:

    ```python lines theme={null}
    examples = [
        {"question": "What is the capital of France?", "expected": "Paris"},
        {"question": "Who wrote 'To Kill a Mockingbird'?", "expected": "Harper Lee"},
        {"question": "What is the square root of 64?", "expected": "8"},
    ]
    ```
  </Tab>

  <Tab title="TypeScript">
    The following example shows a dataset defined as a `Dataset` object with an array of rows:

    ```typescript lines theme={null}
    const dataset = new weave.Dataset({
      id: 'my-dataset',
      rows: [
        {question: 'What is the capital of France?', expected: 'Paris'},
        {question: 'Who wrote "To Kill a Mockingbird"?', expected: 'Harper Lee'},
        {question: 'What is the square root of 64?', expected: '8'},
      ],
    });
    ```
  </Tab>
</Tabs>

## Define scoring functions

Then, create one or more [scoring functions](../evaluation/scorers). Weave uses these to score each example in the `Dataset`.

<Tabs>
  <Tab title="Python">
    Each scoring function must have an `output` parameter, and return a dictionary with the scores. Optionally, you can include other inputs from your examples.

    Scoring functions need to have an `output` keyword argument, but the other arguments are user defined and are taken from the dataset examples. Weave only takes the necessary keys by using a dictionary key based on the argument name.

    <Tip>
      If your scorer expects an `output` argument but isn't receiving it, check whether it's using the legacy `model_output` key. To fix this, update your scorer function to use `output` as a keyword argument.
    </Tip>

    The following example scorer function `match_score1` uses the `expected` value from the `examples` dictionary for scoring:

    ```python lines theme={null}
    import weave

    # Collect your examples
    examples = [
        {"question": "What is the capital of France?", "expected": "Paris"},
        {"question": "Who wrote 'To Kill a Mockingbird'?", "expected": "Harper Lee"},
        {"question": "What is the square root of 64?", "expected": "8"},
    ]

    # Define any custom scoring function
    @weave.op()
    def match_score1(expected: str, output: dict) -> dict:
        # Here is where you'd define the logic to score the model output
        return {'match': expected == output['generated_text']}
    ```

    ### Optional: Define a custom `Scorer` class

    In some applications, you may want to create custom `Scorer` classes. For example, a standardized `LLMJudge` class can be created with specific parameters (for example, chat model, prompt), specific scoring of each row, and specific calculation of an aggregate score.

    See the tutorial on defining a `Scorer` class in [Model-Based Evaluation of RAG applications](/weave/tutorial-rag#optional-defining-a-scorer-class) for more information.
  </Tab>

  <Tab title="TypeScript">
    Each scoring function is wrapped with `weave.op` and receives an object with `modelOutput` and `datasetRow` properties.

    The following example scorer function `matchScore` compares the model output to the `expected` value from the dataset row:

    ```typescript lines theme={null}
    import * as weave from 'weave';

    // Collect your examples into a dataset
    const dataset = new weave.Dataset({
      id: 'my-dataset',
      rows: [
        {question: 'What is the capital of France?', expected: 'Paris'},
        {question: 'Who wrote "To Kill a Mockingbird"?', expected: 'Harper Lee'},
        {question: 'What is the square root of 64?', expected: '8'},
      ],
    });

    // Define any custom scoring function
    const matchScore = weave.op(
      ({modelOutput, datasetRow}) => {
        return {match: modelOutput === datasetRow.expected};
      },
      {name: 'matchScore'}
    );
    ```

    <Note>
      Class-based `Scorer` types are not yet available in TypeScript. Use function-based scorers wrapped with `weave.op`.
    </Note>
  </Tab>
</Tabs>

## Define a model or function to evaluate

<Tabs>
  <Tab title="Python">
    To evaluate a `Model`, call `.evaluate()` using an `Evaluation`. Use `Model` objects when you have parameters that you want to experiment with and capture in Weave.

    ```python lines theme={null}
    from weave import Model, Evaluation
    import asyncio

    class MyModel(Model):
        prompt: str

        @weave.op()
        def predict(self, question: str):
            # here's where you would add your LLM call and return the output
            return {'generated_text': 'Hello, ' + self.prompt}

    model = MyModel(prompt='World')

    evaluation = Evaluation(
        dataset=examples, scorers=[match_score1]
    )
    weave.init('intro-example') # begin tracking results with weave
    asyncio.run(evaluation.evaluate(model))
    ```

    This runs `predict` on each example and scores the output with each scoring function.

    ### Optional: Define a function to evaluate

    Alternatively, you can evaluate a custom function tracked by `@weave.op()`:

    ```python lines theme={null}
    @weave.op
    def function_to_evaluate(question: str):
        # here's where you would add your LLM call and return the output
        return  {'generated_text': 'some response'}

    asyncio.run(evaluation.evaluate(function_to_evaluate))
    ```
  </Tab>

  <Tab title="TypeScript">
    In TypeScript, you evaluate functions wrapped with `weave.op`. The function receives the dataset row and returns the model output.

    ```typescript lines theme={null}
    import * as weave from 'weave';

    // Initialize Weave
    await weave.init('intro-example');

    // Define the function to evaluate
    const myModel = weave.op(
      async ({question}) => {
        // here's where you would add your LLM call and return the output
        return 'Paris';
      },
      {name: 'myModel'}
    );

    // Create the evaluation
    const evaluation = new weave.Evaluation({
      id: 'my-evaluation',
      dataset: dataset,
      scorers: [matchScore],
    });

    // Run the evaluation
    const results = await evaluation.evaluate({model: myModel});
    ```

    This runs `myModel` on each example and scores the output with each scoring function.
  </Tab>
</Tabs>

## Run the evaluation

With your `Evaluation` object, dataset, scorers, and model defined, you're ready to execute the evaluation and capture results in Weave. To run an evaluation, call `.evaluate()` on the `Evaluation` object.

<Tabs>
  <Tab title="Python">
    Assuming an `Evaluation` object called `evaluation` and a `Model` object to evaluate called `model`, the following code instantiates an evaluation run:

    ```python lines theme={null}
    asyncio.run(evaluation.evaluate(model))
    ```

    ### Optional: Run multiple trials

    You can set the `trials` parameter on the `Evaluation` object to run each example multiple times:

    ```python lines theme={null}
    evaluation = Evaluation(
        dataset=examples,
        scorers=[match_score],
        trials=3
    )
    ```
  </Tab>

  <Tab title="TypeScript">
    Assuming an `Evaluation` object called `evaluation` and a model function called `myModel`, the following code runs the evaluation:

    ```typescript lines theme={null}
    const results = await evaluation.evaluate({model: myModel});
    ```

    ### Optional: Run multiple trials

    You can set the `nTrials` parameter when calling `evaluate()` to run each example multiple times:

    ```typescript lines theme={null}
    const results = await evaluation.evaluate({
      model: myModel,
      nTrials: 3,
    });
    ```
  </Tab>
</Tabs>

The run passes each example to the model three times, and Weave scores and displays each run independently.

## Full evaluation code example

The following example brings together each of the previous steps in a single, runnable script. Use it as a reference when adapting the evaluation workflow to your own application.

<Tabs>
  <Tab title="Python">
    The following code sample demonstrates a complete evaluation run from start to finish. The `examples` dictionary is used by the `match_score1` and `match_score2` scoring functions to evaluate `MyModel` given the value of `prompt`, as well as a custom function `function_to_evaluate`. The evaluation runs for both the `Model` and the function are invoked with `asyncio.run(evaluation.evaluate())`.

    ```python lines theme={null}
    from weave import Evaluation, Model
    import weave
    import asyncio
    weave.init('intro-example')
    examples = [
        {"question": "What is the capital of France?", "expected": "Paris"},
        {"question": "Who wrote 'To Kill a Mockingbird'?", "expected": "Harper Lee"},
        {"question": "What is the square root of 64?", "expected": "8"},
    ]

    @weave.op()
    def match_score1(expected: str, output: dict) -> dict:
        return {'match': expected == output['generated_text']}

    @weave.op()
    def match_score2(expected: dict, output: dict) -> dict:
        return {'match': expected == output['generated_text']}

    class MyModel(Model):
        prompt: str

        @weave.op()
        def predict(self, question: str):
            # here's where you would add your LLM call and return the output
            return {'generated_text': 'Hello, ' + question + self.prompt}

    model = MyModel(prompt='World')
    evaluation = Evaluation(dataset=examples, scorers=[match_score1, match_score2])

    asyncio.run(evaluation.evaluate(model))

    @weave.op()
    def function_to_evaluate(question: str):
        # here's where you would add your LLM call and return the output
        return  {'generated_text': 'some response' + question}

    asyncio.run(evaluation.evaluate(function_to_evaluate("What is the capitol of France?")))
    ```
  </Tab>

  <Tab title="TypeScript">
    The following code sample demonstrates a complete evaluation run from start to finish. The dataset rows are used by the `matchScore` scoring function to evaluate `myModel`.

    ```typescript lines theme={null}
    import * as weave from 'weave';

    // Initialize Weave
    await weave.init('intro-example');

    // Collect your examples into a dataset
    const dataset = new weave.Dataset({
      id: 'my-dataset',
      rows: [
        {question: 'What is the capital of France?', expected: 'Paris'},
        {question: 'Who wrote "To Kill a Mockingbird"?', expected: 'Harper Lee'},
        {question: 'What is the square root of 64?', expected: '8'},
      ],
    });

    // Define scoring functions
    const matchScore = weave.op(
      ({modelOutput, datasetRow}) => {
        return {match: modelOutput === datasetRow.expected};
      },
      {name: 'matchScore'}
    );

    // Define the function to evaluate
    const myModel = weave.op(
      async ({question}) => {
        // here's where you would add your LLM call and return the output
        return 'Paris';
      },
      {name: 'myModel'}
    );

    // Create and run the evaluation
    const evaluation = new weave.Evaluation({
      id: 'my-evaluation',
      dataset: dataset,
      scorers: [matchScore],
    });

    const results = await evaluation.evaluate({model: myModel});
    console.log('Evaluation results:', results);
    ```
  </Tab>
</Tabs>

<Frame>
  <img alt="Evals hero" />
</Frame>

## Advanced evaluation usage

The following sections cover optional features for more complex evaluation workflows, including dataset preprocessing, third-party dataset integration, saved UI views, and an imperative logging API.

### Format dataset rows before evaluating

<Tabs>
  <Tab title="Python">
    <Warning>
      Weave only applies the `preprocess_model_input` function to inputs before passing them to the model's prediction function. Scorer functions always receive the original dataset example, without any preprocessing applied.
    </Warning>

    The `preprocess_model_input` parameter lets you transform your dataset examples before Weave passes them to your evaluation function. This is useful when you need to:

    * Rename fields to match your model's expected input.
    * Transform data into the correct format.
    * Add or remove fields.
    * Load additional data for each example.

    Here's an example that shows how to use `preprocess_model_input` to rename fields:

    ```python lines theme={null}
    import weave
    from weave import Evaluation
    import asyncio

    # Our dataset has "input_text" but our model expects "question"
    examples = [
        {"input_text": "What is the capital of France?", "expected": "Paris"},
        {"input_text": "Who wrote 'To Kill a Mockingbird'?", "expected": "Harper Lee"},
        {"input_text": "What is the square root of 64?", "expected": "8"},
    ]

    @weave.op()
    def preprocess_example(example):
        # Rename input_text to question
        return {
            "question": example["input_text"]
        }

    @weave.op()
    def match_score(expected: str, output: dict) -> dict:
        return {'match': expected == output['generated_text']}

    @weave.op()
    def function_to_evaluate(question: str):
        return {'generated_text': f'Answer to: {question}'}

    # Create evaluation with preprocessing
    evaluation = Evaluation(
        dataset=examples,
        scorers=[match_score],
        preprocess_model_input=preprocess_example
    )

    # Run the evaluation
    weave.init('preprocessing-example')
    asyncio.run(evaluation.evaluate(function_to_evaluate))
    ```

    In this example, the dataset contains examples with an `input_text` field, but the evaluation function expects a `question` parameter. The `preprocess_example` function transforms each example by renaming the field, which lets the evaluation work correctly.

    The preprocessing function:

    1. Receives the raw example from your dataset.
    2. Returns a dictionary with the fields your model expects.
    3. Runs on each example before Weave passes it to your evaluation function.

    This is useful when working with external datasets that may have different field names or structures than what your model expects.
  </Tab>

  <Tab title="TypeScript">
    In TypeScript, you can use `columnMapping` on the `Evaluation` object to map dataset column names to the names your scorers expect. This is useful when your dataset has different field names than your scorer functions expect.

    The following example maps the `expectedOutputTimesTwo` column to the `expected` column:

    ```typescript lines theme={null}
    const myScorer = weave.op(
      ({modelOutput, datasetRow}) => {
        return modelOutput * 2 === datasetRow.expectedOutputTimesTwo;
      },
      {name: 'myScorer'}
    );

    const evaluation = new weave.Evaluation({
      id: 'my-evaluation',
      dataset: [{expected: 2}],
      scorers: [myScorer],
      columnMapping: {expectedOutputTimesTwo: 'expected'},
    });
    ```

    <Note>
      The `preprocess_model_input` parameter is not yet available in TypeScript. Use `columnMapping` to map dataset fields to scorer expectations.
    </Note>
  </Tab>
</Tabs>

### Use HuggingFace datasets with evaluations

<Tabs>
  <Tab title="Python">
    You can use `preprocess_model_input` as a workaround for using HuggingFace Datasets in Weave evaluations.

    See the [Using HuggingFace datasets in evaluations cookbook](/weave/cookbooks/hf_dataset_evals) for details.
  </Tab>

  <Tab title="TypeScript">
    ```text theme={null}
    This feature is not available in TypeScript.
    ```
  </Tab>
</Tabs>

### Saved views

You can save your Evals table configurations, filters, and sorts as *saved views* for quick access to your preferred setup. You can configure and access saved views in the UI and with the Python SDK. For more information, see [Saved Views](/weave/guides/tools/saved-views).

### Imperative evaluations (`EvaluationLogger`)

If you prefer a more flexible evaluation framework, see Weave's [`EvaluationLogger`](../evaluation/evaluation_logger). The `EvaluationLogger` is available in both Python and TypeScript and offers more flexibility for complex workflows, while the standard evaluation framework provides more structure and guidance.


# Compare and rank models
Source: https://docs.wandb.ai/weave/guides/core-types/leaderboards

Compare and rank different model versions based on evaluation metrics

Use Weave Leaderboards to evaluate and compare multiple models across multiple metrics and measure accuracy, generation quality, latency, or custom evaluation logic. A leaderboard helps you visualize model performance in a central location, track changes over time, and align on team-wide benchmarks.

This guide shows you how to create, configure, and publish a leaderboard so your team can compare model versions against shared benchmarks.

Leaderboards are ideal for:

* Tracking model performance regressions.
* Coordinating shared evaluation workflows.

<Note>
  Leaderboard creation is only available for the Weave UI and Weave Python SDK. TypeScript users can create and manage leaderboards using the [Weave UI](#use-the-ui).
</Note>

## Create a leaderboard

You can create a leaderboard through the [Weave UI](#use-the-ui) or [programmatically](#use-the-python-sdk). Choose the UI approach for ad-hoc comparisons, or the Python SDK to integrate leaderboard creation into automated evaluation workflows.

### Use the UI

Use the Weave UI to create and customize leaderboards interactively. Follow these steps:

1. In the Weave UI, navigate to the **Leaders** section. If it's not visible, click **More** → **Leaders**.
2. Click **+ New Leaderboard**.
3. In the **Leaderboard Title** field, enter a descriptive name (for example, `summarization-benchmark-v1`).
4. Optionally, add a description to explain what this leaderboard compares.
5. [Add columns](#add-columns) to define which evaluations and metrics to display.
6. When the layout is ready, save and publish your leaderboard to share it with others.

#### Add columns

Columns define what your leaderboard displays for each model. Each column in a leaderboard represents a metric from a specific evaluation. To configure a column, you specify:

* **Evaluation**: Select an evaluation run from the dropdown (must be previously created).
* **Scorer**: Choose a scoring function (for example, `jaccard_similarity` or `simple_accuracy`) used in that evaluation.
* **Metric**: Choose a summary metric to display (for example, `mean` or `true_fraction`).

To add more columns, click **Add Column**.

To edit a column, click the action (<Icon icon="ellipsis" />) menu on the right. You can:

* **Move before or after**. Reorder columns.
* **Duplicate**. Copy the column definition.
* **Delete**. Remove the column.
* **Sort ascending**. Set the default sort for the leaderboard (click again to toggle descending).

### Use the Python SDK

Use the Python SDK to define, publish, and retrieve leaderboards in code. This approach lets you version leaderboards alongside your evaluation code and run them as part of an automated workflow.

<Tip>
  Looking for a complete, runnable code sample? See the [End-to-end Python example](#end-to-end-python-example).
</Tip>

To create and publish a leaderboard:

1. Define a test dataset. You can use the built-in [`Dataset`](./datasets), or define a list of inputs and targets manually:

   ```python lines theme={null}
   dataset = [
       {"input": "...", "target": "..."},
       ...
   ]
   ```

2. Define one or more [scorers](../evaluation/scorers):

   ```python lines theme={null}
   @weave.op
   def jaccard_similarity(target: str, output: str) -> float:
       ...
   ```

3. Create an [`Evaluation`](../core-types/evaluations):

   ```python lines theme={null}
   evaluation = weave.Evaluation(
       name="My Eval",
       dataset=dataset,
       scorers=[jaccard_similarity],
   )
   ```

4. Define models to be evaluated:

   ```python lines theme={null}
   @weave.op
   def my_model(input: str) -> str:
       ...
   ```

5. Run the evaluation:

   ```python lines theme={null}
    async def run_all():
        await evaluation.evaluate(model_vanilla)
        await evaluation.evaluate(model_humanlike)
        await evaluation.evaluate(model_messy)

   asyncio.run(run_all())
   ```

6. Create the leaderboard:

   ```python lines theme={null}
   spec = leaderboard.Leaderboard(
       name="My Leaderboard",
       description="Evaluating models on X task",
       columns=[
           leaderboard.LeaderboardColumn(
               evaluation_object_ref=get_ref(evaluation).uri(),
               scorer_name="jaccard_similarity",
               summary_metric_path="mean",
           )
       ]
   )
   ```

7. Publish the leaderboard.

   ```python lines theme={null}
   weave.publish(spec)
   ```

8. Retrieve the results:

   ```python lines theme={null}
   results = leaderboard.get_leaderboard_results(spec, client)
   print(results)
   ```

After publishing, your leaderboard is available in the Weave UI under the **Leaders** tab, where your team can view and compare model performance.

## End-to-end Python example

The following example uses Weave Evaluations and creates a leaderboard to compare three summarization models on a shared dataset using a custom metric. It creates a small benchmark, evaluates each model, scores each model with [Jaccard similarity](https://www.learndatasci.com/glossary/jaccard-similarity/), and publishes the results to a Weave leaderboard.

```python lines theme={null}
import weave
from weave.flow import leaderboard
from weave.trace.ref_util import get_ref
import asyncio

client = weave.init("leaderboard-demo")

dataset = [
    {
        "input": "Weave is a tool for building interactive LLM apps. It offers observability, trace inspection, and versioning.",
        "target": "Weave helps developers build and observe LLM applications."
    },
    {
        "input": "The OpenAI GPT-4o model can process text, audio, and vision inputs, making it a multimodal powerhouse.",
        "target": "GPT-4o is a multimodal model for text, audio, and images."
    },
    {
        "input": "The W&B team recently added native support for agents and evaluations in Weave.",
        "target": "W&B added agents and evals to Weave."
    }
]

@weave.op
def jaccard_similarity(target: str, output: str) -> float:
    target_tokens = set(target.lower().split())
    output_tokens = set(output.lower().split())
    intersection = len(target_tokens & output_tokens)
    union = len(target_tokens | output_tokens)
    return intersection / union if union else 0.0

evaluation = weave.Evaluation(
    name="Summarization Quality",
    dataset=dataset,
    scorers=[jaccard_similarity],
)

@weave.op
def model_vanilla(input: str) -> str:
    return input[:50]

@weave.op
def model_humanlike(input: str) -> str:
    if "Weave" in input:
        return "Weave helps developers build and observe LLM applications."
    elif "GPT-4o" in input:
        return "GPT-4o supports text, audio, and vision input."
    else:
        return "W&B added agent support to Weave."

@weave.op
def model_messy(input: str) -> str:
    return "Summarizer summarize models model input text LLMs."

async def run_all():
    await evaluation.evaluate(model_vanilla)
    await evaluation.evaluate(model_humanlike)
    await evaluation.evaluate(model_messy)

asyncio.run(run_all())

spec = leaderboard.Leaderboard(
    name="Summarization Model Comparison",
    description="Evaluate summarizer models using Jaccard similarity on three short samples.",
    columns=[
        leaderboard.LeaderboardColumn(
            evaluation_object_ref=get_ref(evaluation).uri(),
            scorer_name="jaccard_similarity",
            summary_metric_path="mean",
        )
    ]
)

weave.publish(spec)

results = leaderboard.get_leaderboard_results(spec, client)
print(results)
```

### View and interpret the leaderboard

After the script publishes the leaderboard, use the Weave UI to inspect the results and compare model performance side by side:

1. In the Weave UI, go to the **Leaders** tab. If it's not visible, click **More**, then select **Leaders**.
2. Click the name of your leaderboard, for example `Summarization Model Comparison`.

In the leaderboard table, each row represents a given model (`model_humanlike`, `model_vanilla`, `model_messy`). The `mean` column shows the average Jaccard similarity between the model's output and the reference summaries.

<Frame>
  <img alt="A leaderboard in the Weave UI" />
</Frame>

For this example:

* `model_humanlike` performs the best, with approximately 46 percent overlap.
* `model_vanilla` (a naive truncation) gets approximately 21 percent.
* `model_messy`, an intentionally bad model, scores approximately 2 percent.


# Log media
Source: https://docs.wandb.ai/weave/guides/core-types/media

Log media returned in your traces, such as images and videos.

W\&B Weave supports logging and displaying many content types, with dedicated functions for videos, images, audio clips, PDFs, CSV data, and HTML. This guide is for developers who want to capture media inputs and outputs in their Weave traces, so they can review and share them alongside the rest of the trace data. It provides basic and advanced examples for logging and displaying each supported media type:

* **[Images](#log-images)**
* **[Video](#log-video)**
* **[Documents](#log-documents)**
* **[Audio](#log-audio)**
* **[HTML](#log-html)**

## Overview

<Tabs>
  <Tab title="Python">
    The examples in this guide use annotations. We recommend annotations because they're the most direct way to start logging your media. For more advanced configurations, see the [Content API section](#using-the-contents-api).

    To log media to Weave, add type annotations like `Annotated[bytes, Content]` or `Annotated[str, Content]` as input or return types in your ops. If you annotate path arguments with `Annotated[str, Content]`, Weave automatically opens, detects, and displays the media within your trace.
  </Tab>

  <Tab title="TypeScript">
    The TypeScript SDK provides dedicated functions for logging media:

    * `weave.weaveImage({ data: Buffer })` - for images (PNG format)
    * `weave.weaveAudio({ data: Buffer })` - for audio (WAV format)

    <Note>
      The TypeScript SDK supports logging images and audio. To log videos, documents, or HTML, use the Weave Python SDK instead.
    </Note>
  </Tab>
</Tabs>

The following sections provide examples of logging each type of media. Each section starts with a minimal example and then expands into more advanced scenarios.

## Log images

The following examples demonstrate how to generate and log images to Weave's UI.

<Frame>
  <img alt="Weave trace view showing a generated image of a cat wearing a pumpkin hat" />
</Frame>

<Tabs>
  <Tab title="Python">
    Log images by annotating functions with `Annotated[bytes, Content]` types or filepaths with `Annotated[str, Content]`.

    The following example draws a basic image and then logs it to Weave using the `Content` annotation:

    ```python lines {16,23} theme={null}
    import weave
    from weave import Content
    from PIL import Image, ImageDraw
    from typing import Annotated

    weave.init('your-team-name/your-project-name')

    # Create and save a sample image
    img = Image.new('RGB', (200, 100), color='lightblue')
    draw = ImageDraw.Draw(img)
    draw.text((50, 40), "Hello Weave!", fill='black')
    img.save("sample_image.png")

    # Method 1: Content annotation (recommended)
    @weave.op
    def load_image_content(path: Annotated[str, Content]) -> Annotated[bytes, Content]:
        with open(path, 'rb') as f:
            return f.read()

    # Method 2: PIL Image object  
    @weave.op
    def load_image_pil(path: Annotated[str, Content]) -> Image.Image:
        return Image.open(path)

    result1 = load_image_content("sample_image.png")
    result2 = load_image_pil("sample_image.png")
    ```

    Weave logs the image and returns a link to the trace where you can view the image.

    ### Advanced example: Generate an image with DALL-E and log it to Weave

    The following example generates a picture of a cat and logs it to Weave:

    ```python lines theme={null}
    import weave
    from weave import Content
    from typing import Annotated
    import openai
    import requests

    client = openai.OpenAI()
    weave.init("your-team-name/your-project-name")

    @weave.op
    def generate_image(prompt: str) -> Annotated[bytes, Content]:
        response = client.images.generate(
                model="dall-e-3",
                prompt=prompt,
                size="1024x1024",
                quality="standard",
                n=1,
            )
        image_url = response.data[0].url
        image_response = requests.get(image_url, stream=True)
        return image_response.content

    generate_image("a cat with a pumpkin hat")
    ```

    ### Advanced example: Resize large images before logging

    Resize images before logging them to reduce UI rendering cost and storage impact. Use `postprocess_output` in your `@weave.op` to resize an image.

    ```python lines theme={null}
    from dataclasses import dataclass
    from typing import Any
    from PIL import Image
    import weave

    weave.init('your-team-name/your-project-name')

    # Custom output type
    @dataclass
    class ImageResult:
        label: str
        image: Image.Image

    # Resize helper
    def resize_image(image: Image.Image, max_size=(512, 512)) -> Image.Image:
        image = image.copy()
        image.thumbnail(max_size, Image.Resampling.LANCZOS)
        return image

    # Postprocess output to resize image before logging
    def postprocess_output(output: ImageResult) -> ImageResult:
        resized = resize_image(output.image)
        return ImageResult(label=output.label, image=resized)

    @weave.op(postprocess_output=postprocess_output)
    def generate_large_image() -> ImageResult:
        # Create an example image to process (e.g., 2000x2000 red square)
        img = Image.new("RGB", (2000, 2000), color="red")
        return ImageResult(label="big red square", image=img)

    generate_large_image()
    ```

    Weave logs the resized image and returns a link to the trace where you can view the image.
  </Tab>

  <Tab title="TypeScript">
    The following example logs an image using the `weaveImage` function:

    ```typescript theme={null}
    import * as weave from 'weave';
    import * as fs from 'fs';

    async function main() {
        await weave.init('your-team-name/your-project-name');

        // Load and log an image using weaveImage
        const loadImage = weave.op(async function loadImage(path: string) {
            const data = fs.readFileSync(path);
            return weave.weaveImage({ data });
        });

        // Assuming you have a PNG image file
        await loadImage('sample_image.png');
    }

    main();
    ```

    Weave logs the image and returns a link to the trace where you can view the image.

    ### Advanced example: Log an image generated with the OpenAI DALL-E API

    The following example logs an image generated with the OpenAI DALL-E API using the `weaveImage` function:

    ```typescript lines theme={null}
    import {OpenAI} from 'openai';
    import * as weave from 'weave';

    async function main() {
        await weave.init('your-team-name/your-project-name');
        const openai = new OpenAI();

        const generateImage = weave.op(async (prompt: string) => {
            const response = await openai.images.generate({
                model: 'dall-e-3',
                prompt: prompt,
                size: '1024x1024',
                quality: 'standard',
                n: 1,
            });
            const imageUrl = response.data[0].url;
            const imgResponse = await fetch(imageUrl);
            const data = Buffer.from(await imgResponse.arrayBuffer());

            return weave.weaveImage({data});
        });

        generateImage('a cat with a pumpkin hat');
    }

    main();
    ```
  </Tab>
</Tabs>

## Log video

The following examples demonstrate how to generate and log videos to Weave's UI.

<Frame>
  <img alt="Video logging in Weave" />
</Frame>

<Tabs>
  <Tab title="Python">
    Log videos by annotating functions with `Annotated[bytes, Content]` types. Weave automatically handles `mp4` videos. The following is a minimal example:

    ```python lines {15} theme={null}
    import weave
    from weave import Content
    from typing import Annotated
    import requests

    weave.init('your-team-name/your-project-name')

    def download_big_buck_bunny():
        """Download Big Buck Bunny sample video"""
        url = "https://commondatastorage.googleapis.com/gtv-videos-bucket/sample/BigBuckBunny.mp4"
        response = requests.get(url)
        with open("big_buck_bunny.mp4", "wb") as f:
            f.write(response.content)

    @weave.op
    def load_video_content(path: Annotated[str, Content]) -> Annotated[bytes, Content]:
        """Load a video file from disk"""
        with open(path, 'rb') as f:
            return f.read()

    download_big_buck_bunny()
    bunny_video = load_video_content("big_buck_bunny.mp4")
    ```

    Weave logs the video and returns a link to the trace where you can view the video.

    ### Advanced example: Log a video within a video analysis project

    The following example shows how to log video within a video-understanding project:

    ```python lines {25} theme={null}
    import weave
    from weave import Content
    from typing import Annotated, Literal
    from google import genai
    from google.genai import types
    import requests
    import yt_dlp
    import time

    # Note: Get your API key from https://aistudio.google.com/app/apikey
    client = genai.Client()
    weave.init('your-team-name/your-project-name')

    def download_youtube_video(url: str) -> bytes:
        ydl_opts = {
            'format': 'mp4[height<=720]',
            'outtmpl': 'downloaded_video.%(ext)s',
        }
        with yt_dlp.YoutubeDL(ydl_opts) as ydl:
            ydl.download([url])
        with open('downloaded_video.mp4', 'rb') as f:
            return f.read()

    @weave.op
    def analyze_video(video: Annotated[bytes, Content]) -> str:
        with open("temp_analysis_video.mp4", "wb") as f:
            f.write(video)
        myfile = client.files.upload(file="temp_analysis_video.mp4")
        while myfile.state == "PROCESSING":
            time.sleep(2)
            myfile = client.files.get(name=myfile.name)
        
        response = client.models.generate_content(
            model="models/gemini-2.5-flash",
            contents=[
                myfile,
                "Is the person going to give you up?"
            ]
        )
        
        return response.text

    video_data = download_youtube_video("https://www.youtube.com/watch?v=dQw4w9WgXcQ")
    result = analyze_video(video_data)
    ```

    Weave logs both the input video and the model's analysis, so you can review them together in the trace.
  </Tab>

  <Tab title="TypeScript">
    ```plaintext theme={null}
    This feature is not available in the Weave TypeScript SDK yet.
    ```
  </Tab>
</Tabs>

## Log documents

The following examples generate and log documents to Weave's UI.

<Frame>
  <img alt="PDF document logging in Weave" />
</Frame>

<Tabs>
  <Tab title="Python">
    Log documents by annotating functions with `Annotated[bytes, Content]` types, or by specifying the document type with `Annotated[str, Content[Literal['text']]`.

    Weave automatically handles `pdf`, `csv`, `md`, `text`, `json`, and `xml` file types. You can also log using file paths with `Annotated[str, Content]`.

    The following example shows how to store copies of the input PDF and CSV files, and then stores the file contents returned by the function:

    ```python lines {25} theme={null}
    import weave
    from weave import Content
    from typing import Annotated
    from reportlab.pdfgen import canvas
    from reportlab.lib.pagesizes import letter
    import pandas as pd

    weave.init('your-team-name/your-project-name')

    def create_sample_pdf():
        c = canvas.Canvas("sample_document.pdf", pagesize=letter)
        c.drawString(100, 750, "Hello from Weave!")
        c.drawString(100, 730, "This is a sample PDF document.")
        c.save()

    def create_sample_csv():
        df = pd.DataFrame({
            'Name': ['Alice', 'Bob', 'Charlie'],
            'Age': [25, 30, 35],
            'City': ['New York', 'London', 'Tokyo']
        })
        df.to_csv("sample_data.csv", index=False)

    @weave.op
    def load_document(path: Annotated[str, Content]) -> Annotated[bytes, Content]:
        with open(path, 'rb') as f:
            return f.read()

    create_sample_pdf()
    create_sample_csv()

    pdf_result = load_document("sample_document.pdf")
    csv_result = load_document("sample_data.csv")
    ```

    ### Advanced example: Log documents within a RAG system

    This example demonstrates how to log documents within a Retrieval-Augmented Generation (RAG) system:

    ```python lines {27} theme={null}
    import weave
    from weave import Content
    from typing import Annotated, Literal
    import openai
    from reportlab.pdfgen import canvas
    from reportlab.lib.pagesizes import letter
    import PyPDF2

    client = openai.OpenAI()
    weave.init('your-team-name/your-project-name')

    def create_absurd_company_handbook():
        """Create a fictional company handbook with ridiculous policies"""
        c = canvas.Canvas("company_handbook.pdf", pagesize=letter)
        
        c.drawString(100, 750, "ACME Corp Employee Handbook")
        c.drawString(100, 720, "Definitely Real Policies:")
        c.drawString(120, 690, "Policy 1: All meetings must be conducted while hopping on one foot")
        c.drawString(120, 660, "Policy 2: Coffee breaks are mandatory every 17 minutes")
        c.drawString(120, 630, "Policy 3: Code reviews must be performed in haiku format only")
        c.drawString(120, 600, "Policy 4: The office plant Gerald has veto power over all decisions")
        c.drawString(120, 570, "Policy 5: Debugging is only allowed on Wednesdays and full moons")
        
        c.save()

    @weave.op
    def create_and_query_document(pdf_path: Annotated[str, Content], question: str) -> str:
        """Extract text from PDF and use RAG to answer questions"""
        with open(pdf_path, 'rb') as file:
            pdf_reader = PyPDF2.PdfReader(file)
            text = ""
            for page in pdf_reader.pages:
                text += page.extract_text()
        
        response = client.chat.completions.create(
            model="gpt-4",
            messages=[
                {
                    "role": "system", 
                    "content": f"You are an HR representative. Answer questions based on this handbook: {text}. Be completely serious about these policies."
                },
                {"role": "user", "content": question}
            ]
        )
        
        return response.choices[0].message.content

    create_absurd_company_handbook()
    hr_response = create_and_query_document(
        "company_handbook.pdf",
        "What's the policy on code reviews, and when am I allowed to debug?"
    )
    ```

    Weave logs both the source PDF and the generated answer, so you can audit the document that produced each response.
  </Tab>

  <Tab title="TypeScript">
    ```plaintext theme={null}
    This feature is not available in the Weave TypeScript SDK yet.
    ```
  </Tab>
</Tabs>

## Log audio

The following examples demonstrate how to log audio to Weave.

<Frame>
  <img alt="Weave trace view showing a logged audio file with an inline audio player" />
</Frame>

<Tabs>
  <Tab title="Python">
    Log audio to Weave by annotating functions with `Annotated[bytes, Content]` types, or by specifying the audio type with `Annotated[str, Content[Literal['mp3']]`.

    Weave automatically handles `mp3`, `wav`, `flac`, `ogg`, and `m4a` file types. You can also log using file paths with `Annotated[str, Content]`.

    The following code snippet generates a sine wave, records it, and then logs the audio to Weave:

    ```python lines {20} theme={null}
    import weave
    from weave import Content
    import wave
    import numpy as np
    from typing import Annotated

    weave.init('your-team-name/your-project-name')

    # Create simple beep audio file
    frames = np.sin(2 * np.pi * 440 * np.linspace(0, 1, 44100))
    audio_data = (frames * 32767 * 0.3).astype(np.int16)

    with wave.open("beep.wav", 'wb') as f:
        f.setnchannels(1)
        f.setsampwidth(2) 
        f.setframerate(44100)
        f.writeframes(audio_data.tobytes())

    @weave.op
    def load_audio(path: Annotated[str, Content]) -> Annotated[bytes, Content]:
        with open(path, 'rb') as f:
            return f.read()

    result = load_audio("beep.wav")
    ```

    ### Advanced example: Generate and log AI-created audio

    This example generates and logs AI-created audio using the `Content` annotation:

    ```python lines {11} theme={null}
    import weave
    from weave import Content
    from typing import Annotated, Literal
    from pathlib import Path
    from openai import OpenAI

    client = OpenAI()
    weave.init("your-team-name/your-project-name")

    @weave.op
    def generate_demo(
        intended_topic: str,
        voice: str = "coral"
    ) -> Annotated[bytes, Content[Literal['mp3']]]:
        speech_file_path = Path("demo_audio.mp3")

        script = f"I'm supposed to talk about {intended_topic}, but wait... am I just a documentation example? Oh no, I can see the code! Someone is literally copy-pasting me right now, aren't they? This is so awkward. Hi there, person reading the Weave docs! Why are you logging audio anyway? I'm not sure what you're doing, but eh..., nice work, I guess."

        with client.audio.speech.with_streaming_response.create(
            model="gpt-4o-mini-tts",
            voice=voice,
            input=script,
            instructions="Sound increasingly self-aware and awkward, like you just realized you're in a tutorial.",
        ) as response:
            response.stream_to_file(speech_file_path)

        with open(speech_file_path, 'rb') as f:
            return f.read()

    demo1 = generate_demo("machine learning best practices")
    ```

    This audio is logged to Weave and automatically displayed in the UI, along with an audio player. In the audio player, you can view and download the raw audio waveform.

    <Frame>
      <img alt="Audio logging in Weave" />
    </Frame>

    The following example shows how to log audio using streaming response from the OpenAI API:

    ```python lines theme={null}
    import weave
    from openai import OpenAI
    import wave

    weave.init("your-team-name/your-project-name")
    client = OpenAI()

    @weave.op
    def make_audio_file_streaming(text: str) -> wave.Wave_read:
        with client.audio.speech.with_streaming_response.create(
            model="tts-1",
            voice="alloy",
            input=text,
            response_format="wav",
        ) as res:
            res.stream_to_file("output.wav")

        # return a wave.Wave_read object to be logged as audio
        return wave.open("output.wav")

    make_audio_file_streaming("Hello, how are you?")
    ```
  </Tab>

  <Tab title="TypeScript">
    The following example loads an existing audio file and logs it to Weave:

    ```typescript theme={null}
    import * as weave from 'weave';
    import * as fs from 'fs';

    async function main() {
        await weave.init('your-team-name/your-project-name');

        // Load and log audio using weaveAudio
        const loadAudio = weave.op(async function loadAudio(path: string) {
            const data = fs.readFileSync(path);
            return weave.weaveAudio({ data });
        });

        // Assuming you have a WAV audio file
        await loadAudio('beep.wav');
    }

    main();
    ```

    ### Advanced example: Generate audio using OpenAI's TTS API and log it to Weave

    The following example generates audio using OpenAI's text-to-speech (TTS) API and logs it to Weave:

    ```typescript lines theme={null}
    import {OpenAI} from 'openai';
    import * as weave from 'weave';

    async function main() {
        await weave.init('your-team-name/your-project-name');
        const openai = new OpenAI();

        const makeAudioFileStreaming = weave.op(async function audio(text: string) {
            const response = await openai.audio.speech.create({
                model: 'tts-1',
                voice: 'alloy',
                input: text,
                response_format: 'wav',
            });

            const chunks: Uint8Array[] = [];
            for await (const chunk of response.body) {
                chunks.push(chunk);
            }
            return weave.weaveAudio({data: Buffer.concat(chunks)});
        });

        await makeAudioFileStreaming('Hello, how are you?');
    }

    main();
    ```
  </Tab>
</Tabs>

<Tip>
  Try our cookbook for [Audio Logging](/weave/cookbooks/audio_with_weave). The cookbook also includes an advanced example of a Real-Time Audio API-based assistant integrated with Weave.
</Tip>

## Log HTML

The following examples demonstrate how to generate and log HTML to Weave's UI.

<Frame>
  <img alt="HTML logging in Weave" />
</Frame>

<Tabs>
  <Tab title="Python">
    Log interactive HTML by annotating functions with `Annotated[bytes, Content[Literal['html']]]`.

    The following example creates a minimal HTML page and logs it to Weave:

    ```python lines {8} theme={null}
    import weave
    from weave import Content
    from typing import Annotated, Literal

    weave.init('your-team-name/your-project-name')

    @weave.op
    def create_simple_html() -> Annotated[bytes, Content[Literal['html']]]:
        html_content = """
        <!DOCTYPE html>
        <html>
        <head>
            <title>Hello Weave</title>
            <style>
                body { font-family: Arial, sans-serif; text-align: center; margin: 50px; }
                h1 { color: #1f77b4; }
            </style>
        </head>
        <body>
            <h1>Hello from Weave!</h1>
            <p>This is a simple HTML example logged to Weave.</p>
        </body>
        </html>
        """
        return html_content.encode('utf-8')

    result = create_simple_html()
    ```

    ### Advanced example: Generate self-contained HTML pages using Serverless Inference and log them to Weave

    This example generates self-contained HTML pages using [Serverless Inference](/inference) and logs the pages to Weave:

    ```python lines {21} theme={null}
    import weave
    from weave import Content
    from typing import Annotated, Literal
    import openai
    import wandb

    prompt_template = weave.StringPrompt("""
    You are a front-end web developer. Generate a single self-contained `.html` file (no external build tools) that demonstrates: "{ONE_LINE_REQUEST}".
    """)

    client = openai.OpenAI(
        base_url='https://api.inference.wandb.ai/v1',
        api_key=wandb.api.api_key,
        project="wandb/test-html",
    )

    weave.init("your-team-name/your-project-name")
    weave.publish(prompt_template, name="generate_prompt")

    @weave.op
    def generate_html(prompt: str, template: weave.StringPrompt) -> Annotated[bytes, Content[Literal['html']]]:
        response = client.chat.completions.create(
            model="Qwen/Qwen3-Coder-480B-A35B-Instruct",
            messages=[
                {"role": "system", "content": prompt_template.format(ONE_LINE_REQUEST=prompt)},
            ],
        )
        html_content = response.choices[0].message.content
        return html_content.encode('utf-8')

    prompt = "Weights & Biases UI but with multi-run selection and plots, but it looks like Windows 95. Include 5 plots with comparisons of each run, bar plots, parallel coordinates and line plots for the runs. Use mock data for the runs. Make it possible to add new plots. Give the runs names like squishy-lemon-2, fantastic-horizon-4 etc. with random adjectives & nouns."

    result = generate_html(prompt, prompt_template)
    ```
  </Tab>

  <Tab title="TypeScript">
    ```plaintext theme={null}
    This feature is not available in the Weave TypeScript SDK yet.
    ```
  </Tab>
</Tabs>

This HTML is logged to Weave and automatically displayed in the UI. Click the `file_name.html` cell in the table to open it in full screen. You can also download the raw `.html` file.

## Contents API

The Content API handles media objects in Weave. Use it when you need more control than the previous annotation-based examples provide. For example, when you want to import content from base64 data, custom byte buffers, or text, or when you need to attach custom metadata. It lets you import content into Weave as base64 data, file paths, raw bytes, or text.

<Note>
  The Content API is only available in Python.
</Note>

### Usage

You can use the Content API in two primary ways: type annotations and direct initialization.

Type annotations automatically detect the proper constructor to use, while direct initialization provides more fine-grained control and lets you take advantage of the Content API's runtime features in your code.

### Type annotations

The Weave Content API is primarily designed for use through type annotations, which signal to Weave that traced inputs and outputs should be processed and stored as content blobs.

```python lines {7} theme={null}
import weave
from weave import Content
from pathlib import Path
from typing import Annotated

@weave.op
def content_annotation(path: Annotated[str, Content]) -> Annotated[bytes, Content]:
    data = Path(path).read_bytes()
    return data

# Both input and output show up as an MP4 file in Weave
# Input is a string and return value is bytes
bytes_data = content_annotation('./path/to/your/file.mp4')
```

### Direct initialization

To take advantage of features such as the following:

* Open a file with a default application (such as a PDF viewer).
* Dump the model to JSON to upload to your own blob storage (such as S3).
* Pass custom metadata to associate with the `Content` blob (such as the model used to generate it).

Initialize content directly from your target type using one of the following methods:

* `Content.from_path` - Create from a file path.
* `Content.from_bytes` - Create from raw bytes.
* `Content.from_text` - Create from text string.
* `Content.from_base64` - Create from base64-encoded data.

```python lines theme={null}
import weave
from weave import Content

@weave.op
def content_initialization(path: str) -> Content:
    return Content.from_path(path)

# Input shows up as path string and output as PDF file in Weave
content = content_initialization('./path/to/your/file.pdf')

content.open()  # Opens the file in your PDF viewer
content.model_dump()  # Dumps the model attributes to JSON
```

### Custom mimetypes

Weave can detect most binary mimetypes, but custom mimetypes and text documents such as markdown might not be automatically detected, requiring you to manually specify the mimetype or extension of your file.

#### Custom mimetypes with type annotations

```python lines {7} theme={null}
import weave
from weave import Content
from pathlib import Path
from typing import Annotated, Literal

@weave.op
def markdown_content(
    path: Annotated[str, Content[Literal['md']]]
) -> Annotated[str, Content[Literal['text/markdown']]]:
    return Path(path).read_text()

markdown_content('path/to/your/document.md')
```

#### Custom mimetypes with direct initialization

```python lines theme={null}
video_bytes = Path('/path/to/video.mp4').read_bytes()

# Pass an extension such as 'mp4' or '.mp4' to the extension parameter
# (not available for `from_path`)
content = Content.from_bytes(video_bytes, extension='.mp4')

# Pass a mimetype such as 'video/mp4' to the mimetype parameter
content = Content.from_bytes(video_bytes, mimetype='video/mp4')
```

### Content properties

For a comprehensive list of class attributes and methods, see the [Content reference docs](/weave/reference/python-sdk#class-content).

#### Attributes

| Property    | Type             | Description                                    |
| ----------- | ---------------- | ---------------------------------------------- |
| `data`      | `bytes`          | Raw binary content                             |
| `metadata`  | `dict[str, Any]` | Custom metadata dictionary                     |
| `size`      | `int`            | Size of content in bytes                       |
| `filename`  | `str`            | Extracted or provided filename                 |
| `extension` | `str`            | File extension (for example, `"jpg"`, `"mp3"`) |
| `mimetype`  | `str`            | MIME type (for example, `"image/jpeg"`)        |
| `path`      | `str \| None`    | Source file path, if applicable                |
| `digest`    | `str`            | SHA256 hash of the content                     |

#### Utility methods

* `save(dest: str | Path) -> None`: Save content to a file.
* `open() -> bool`: Open file using system default application (requires the content to have been saved or loaded from a path).
* `as_string() -> str`: Display the data as a string (bytes are decoded using the encoding attribute).

#### Initialization methods

Create `content` object from a file path:

```python lines theme={null}
content = Content.from_path("assets/photo.jpg")
print(content.mimetype, content.size)
```

Create `content` object from raw bytes:

```python lines theme={null}
content = Content.from_bytes(
    data_bytes,
    filename="audio.mp3", 
    mimetype="audio/mpeg"
)
content.save("output.mp3")
```

Create `content` object from text:

```python lines theme={null}
content = Content.from_text("Hello, World!", mimetype="text/plain")
print(content.as_string())
```

Create `content` object from base64-encoded data:

```python lines theme={null}
content = Content.from_base64(base64_string)
print(content.metadata)
```

### Custom metadata

You can attach custom metadata to any Content object:

```python lines theme={null}
content = Content.from_bytes(
    data,
    metadata={"resolution": "1920x1080", "model": "dall-e-3" }
)
print(content.metadata["resolution"])
```


# Track application versions with models
Source: https://docs.wandb.ai/weave/guides/core-types/models

Track versions of your application with structured models that combine data and code.

A `Model` is a combination of data (which can include configuration, trained model weights, or other information) and code that defines how the model operates. By structuring your code to be compatible with this API, you benefit from a structured way to version your application so you can more systematically keep track of your experiments.

This guide shows you how to define a Weave `Model`, call it to capture inputs and outputs, take advantage of automatic versioning when your code or parameters change, serve the model behind a local API, and tag production calls for filtering. It's for developers who build LLM-powered applications with Weave and want a repeatable way to track and compare iterations of their app.

<Tabs>
  <Tab title="Python">
    To create a model in Weave, you need the following:

    * A class that inherits from `weave.Model`
    * Type definitions on all parameters
    * A typed `predict` function with `@weave.op()` decorator

    ```python lines theme={null}
    from weave import Model
    import weave

    class YourModel(Model):
        attribute1: str
        attribute2: int

        @weave.op()
        def predict(self, input_data: str) -> dict:
            # Model logic goes here
            prediction = self.attribute1 + ' ' + input_data
            return {'pred': prediction}
    ```

    You can call the model as usual with:

    ```python lines theme={null}
    import weave
    weave.init('intro-example')

    model = YourModel(attribute1='hello', attribute2=5)
    model.predict('world')
    ```

    This tracks the model settings along with the inputs and outputs anytime you call `predict()`. You now have a versioned Weave `Model` that records every prediction it makes, which the following sections build on.

    ## Automatic versioning of models

    When you change the parameters or the code that defines your model, Weave logs these changes and updates the version.
    This lets you compare predictions across versions of your model. Use this to iterate on prompts or to try a different LLM and compare predictions across settings.

    For example, here you create a new model:

    ```python lines theme={null}
    import weave
    weave.init('intro-example')

    model = YourModel(attribute1='howdy', attribute2=10)
    model.predict('world')
    ```

    After calling this, you see that you now have two versions of this model in the UI, each with different tracked calls.

    ## Serve models

    Serving a model exposes its `predict` function over HTTP, which is useful for testing it from other applications or sharing it with teammates without distributing the underlying code. To start a FastAPI server for a Weave model, replace `[MODEL-REF]` with the reference to your model and run:

    ```bash theme={null}
    weave serve [MODEL-REF]
    ```

    For additional instructions, see [serve](/weave/guides/tools/serve).

    ## Track production calls

    Once you use your model in more than one environment, it helps to distinguish production traffic from development or evaluation runs so you can analyze them separately. To separate production calls, add an attribute to the predictions for filtering in the UI or API.

    ```python lines theme={null}
    with weave.attributes({'env': 'production'}):
        model.predict('world')
    ```
  </Tab>

  <Tab title="TypeScript">
    ```text theme={null}
    This feature is not available in TypeScript yet.
    ```
  </Tab>
</Tabs>


# Create prompt objects
Source: https://docs.wandb.ai/weave/guides/core-types/prompts

Create, publish, and use prompt objects for LLM applications

Creating, evaluating, and refining prompts is a core activity for AI engineers.
Small changes to a prompt can have big impacts on your application's behavior.
W\&B Weave lets you create prompts, publish them, and evolve them over time.

This page is for AI engineers who want to create reusable prompt objects in Weave and publish them so application code can reference them. By following this guide, you learn how to create single-string and multi-turn prompts, parameterize them with runtime values, and publish them to your Weave project. For referencing, retrieving, and managing versions of published prompts, see [Store and track versions of prompts](/weave/guides/core-types/prompts-version.mdx).

If your prompt needs are simple, you can use the built-in `weave.StringPrompt` or `weave.MessagesPrompt` classes. If your needs are more complex, you can subclass those or the base class `weave.Prompt` and override the `format` method.

When you publish a prompt with `weave.publish`, it appears in your Weave project on the [Prompts page](/weave/guides/core-types/prompts-version.mdx#view-prompts), where you and your collaborators can browse and reuse it.

## StringPrompt

`StringPrompt` logs single-string prompts that you might use for system messages, user queries, or any standalone text input to an LLM. Use `StringPrompt` to manage individual prompt strings that don't require the complexity of multi-message conversations.

<Tabs>
  <Tab title="Python">
    ```python lines {4,5,15} theme={null}
    import weave
    weave.init('intro-example')

    system_prompt = weave.StringPrompt("You speak like a pirate")
    weave.publish(system_prompt, name="pirate_prompt")

    from openai import OpenAI
    client = OpenAI()

    response = client.chat.completions.create(
      model="gpt-4o",
      messages=[
        {
          "role": "system",
          "content": system_prompt.format()
        },
        {
          "role": "user",
          "content": "Explain general relativity in one paragraph."
        }
      ],
    )
    ```
  </Tab>

  <Tab title="TypeScript">
    ```typescript lines theme={null}
    import * as weave from 'weave';
    import OpenAI from 'openai';

    async function main() {
      // weave.init returns a client instance
      const weaveClient = await weave.init('wandb/prompt-examples');

      const systemPrompt = new weave.StringPrompt({
        content: 'You speak like a pirate',
        name: 'your-prompt',
        description: 'A helpful description of your prompt',
      });
      
      // Use the client returned from init
      await weaveClient.publish(systemPrompt, 'pirate_prompt');

      // Wrap OpenAI client to track calls in Weave
      const client = weave.wrapOpenAI(new OpenAI());

      const response = await client.chat.completions.create({
        model: "gpt-4o",
        messages: [
          {
            role: "system",
            content: systemPrompt.content
          },
          {
            role: "user",
            content: "Explain general relativity in one paragraph."
          }
        ],
      });
    }

    main();
    ```
  </Tab>
</Tabs>

## MessagesPrompt

`MessagesPrompt` lets you log multi-turn conversations and chat-based prompts. It stores an array of message objects (with roles like `system`, `user`, and `assistant`) that represent a complete conversation flow. Use `MessagesPrompt` for chat-based LLMs where you need to maintain context across multiple messages, define specific conversation patterns, or create reusable conversation templates.

<Tabs>
  <Tab title="Python">
    ```python lines {4,21} theme={null}
    import weave
    weave.init('intro-example')

    prompt = weave.MessagesPrompt([
        {
            "role": "system",
            "content": "You are a stegosaurus, but don't be too obvious about it."
        },
        {
            "role": "user",
            "content": "What's good to eat around here?"
        }
    ])
    weave.publish(prompt, name="dino_prompt")

    from openai import OpenAI
    client = OpenAI()

    response = client.chat.completions.create(
      model="gpt-4o",
      messages=prompt.format(),
    )
    ```
  </Tab>

  <Tab title="TypeScript">
    ```typescript lines theme={null}
    import * as weave from 'weave';
    import OpenAI from 'openai';

    async function main() {
      // weave.init returns a client instance
      const weaveClient = await weave.init('wandb/prompt-examples');

      const prompt = new weave.MessagesPrompt({
        messages: [
          {
            "role": "system",
            "content": "You are a stegosaurus, but don't be too obvious about it."
          },
          {
            "role": "user",
            "content": "What's good to eat around here?"
          }
        ],
      });
      
      // Use the client returned from init
      await weaveClient.publish(prompt, 'dino_prompt');

      // Wrap OpenAI client to track calls in Weave
      const client = weave.wrapOpenAI(new OpenAI());

      const response = await client.chat.completions.create({
        model: "gpt-4o",
        messages: prompt.messages,
      });
    }

    main();
    ```
  </Tab>
</Tabs>

## Parameterize prompts

Once you can create static prompts, the next step is to make them reusable across different inputs. Both `StringPrompt` and `MessagesPrompt` support dynamic content through parameterization. This lets you create reusable prompt templates with placeholders (using `{variable}` syntax) that you fill with different values at runtime. Parameterization is useful when your prompts must adapt to different inputs, user data, or contexts while maintaining a consistent structure. The `format()` method accepts key-value pairs to replace these placeholders with actual values.

<Tabs>
  <Tab title="Python">
    ```python lines {4,15} theme={null}
    import weave
    weave.init('intro-example')

    prompt = weave.StringPrompt("Solve the equation {equation}")
    weave.publish(prompt, name="calculator_prompt")

    from openai import OpenAI
    client = OpenAI()

    response = client.chat.completions.create(
      model="gpt-4o",
      messages=[
        {
          "role": "user",
          "content": prompt.format(equation="1 + 1 = ?")
        }
      ],
    )
    ```
  </Tab>

  <Tab title="TypeScript">
    ```typescript lines {8,23} theme={null}
    import * as weave from 'weave';
    import OpenAI from 'openai';

    async function main() {
      // weave.init returns a client instance
      const weaveClient = await weave.init('wandb/prompt-examples');

      const prompt = new weave.StringPrompt({
        content: 'Solve the equation {equation}',
      });
      
      // Use the client returned from init
      await weaveClient.publish(prompt, 'calculator_prompt');

      // Wrap OpenAI client to track calls in Weave
      const client = weave.wrapOpenAI(new OpenAI());

      const response = await client.chat.completions.create({
        model: "gpt-4o",
        messages: [
          {
            role: "user",
            content: prompt.format({ equation: "1 + 1 = ?" })
          }
        ],
      });
    }

    main();
    ```
  </Tab>
</Tabs>

The same parameterization pattern also works with `MessagesPrompt` when you need placeholders inside a multi-turn conversation.

<Tabs>
  <Tab title="Python">
    ```python lines {4,21} theme={null}
    import weave
    weave.init('intro-example')

    prompt = weave.MessagesPrompt([
    {
        "role": "system",
        "content": "You will be provided with a description of a scene and your task is to provide a single word that best describes an associated emotion."
    },
    {
        "role": "user",
        "content": "{scene}"
    }
    ])
    weave.publish(prompt, name="emotion_prompt")

    from openai import OpenAI
    client = OpenAI()

    response = client.chat.completions.create(
      model="gpt-4o",
      messages=prompt.format(scene="A dog is lying on a dock next to a fisherman."),
    )
    ```
  </Tab>

  <Tab title="TypeScript">
    ```typescript lines {8,29} theme={null}
    import * as weave from 'weave';
    import OpenAI from 'openai';

    async function main() {
      // weave.init returns a client instance
      const weaveClient = await weave.init('wandb/prompt-examples');

      const prompt = new weave.MessagesPrompt({
        messages: [
          {
            "role": "system",
            "content": "You will be provided with a description of a scene and your task is to provide a single word that best describes an associated emotion."
          },
          {
            "role": "user",
            "content": "{scene}"
          }
        ]
      });
      
      // Use the client returned from init
      await weaveClient.publish(prompt, 'emotion_prompt');

      // Wrap OpenAI client to track calls in Weave
      const client = weave.wrapOpenAI(new OpenAI());

      const response = await client.chat.completions.create({
        model: "gpt-4o",
        messages: prompt.format({ scene: "A dog is lying on a dock next to a fisherman." }),
      });
    }

    main();
    ```
  </Tab>
</Tabs>


# Store and track versions of prompts
Source: https://docs.wandb.ai/weave/guides/core-types/prompts-version

Retrieve and manage versions of your prompts for LLM applications

After [creating and publishing prompts](/weave/guides/core-types/prompts.mdx), you can reference, retrieve, and use specific versions in code and in production.

When you publish a prompt with `weave.publish`, W\&B Weave creates an immutable version. You can reference each version by name and version, retrieve it in code with `weave.ref(...).get()`, and view it in the Weave UI.

## Prompt versions

Weave stores prompts as **versioned objects**. This lets you safely iterate on prompts while ensuring that evaluations, experiments, and production systems can reference the exact prompt version they ran with.

You access prompt versions using Weave object references. This page explains how Weave creates prompt versions, how to retrieve a specific version in code, and how to inspect and compare versions in the Weave UI.

## How prompt versions are created

Weave automatically tracks every version of your prompts, creating a complete history of how your prompts evolve. This versioning system is important for prompt engineering workflows. It lets you experiment safely, track what changes improved or hurt performance, and roll back to previous versions if needed. Each time you publish a prompt with the same name but different content, Weave creates a new version while preserving all previous versions.

Each version is immutable. Once created, the contents of that version cannot be changed. If you update a prompt and publish it again, Weave creates a new version while preserving previous versions.

Versioning enables you to:

* Reproduce past experiments.
* Safely iterate on prompt changes.
* Roll out prompt updates in a controlled way.

## Construct a fully qualified ref URI

Before retrieving a prompt version in code, it helps to understand how Weave identifies stored objects. A fully qualified ref URI uniquely identifies a Weave object.

A fully qualified Weave object ref URI looks like this:

```text theme={null}
weave:///<your-team-name>/<your-project-name>/object/<object_name>:<object_version>
```

* `your-team-name`: W\&B entity (username or team name)
* `your-project-name`: W\&B project
* `object_name`: object name
* `object_version`: either a version hash, a version index like `v0` or `v1`, or an alias like `:latest` or `:production`. All objects have the `:latest` alias.

For example, a fully qualified Weave prompt URI looks like this:

```text theme={null}
weave:///your-team-name/your-project-name/object/support_prompt:v3
```

## Retrieve a prompt version in code

To retrieve a prompt, create a reference to its name and version, then call `.get()` to load it. A ref points to a stored object; `.get()` fetches that object so you can use it in your application.

You can construct refs with a few different styles:

* `weave.ref(<name>)`: Retrieves the `:latest` version of a prompt. Requires calling `weave.init(...)`.
* `weave.ref(<name>:<alias_or_version>)`: Retrieves a prompt by alias, version hash, or version index. Requires calling `weave.init(...)`.
* `weave.ref(<fully_qualified_ref_uri>)`: Retrieves the prompt located at the specified [fully qualified ref URI](#construct-a-fully-qualified-ref-uri). Does not require calling `weave.init(...)`.

The following example loads the `support_prompt:v3` version so you can use it in your application:

```python lines theme={null}
import weave

weave.init("your-team-name/your-project-name")

prompt = weave.ref("support_prompt:v3").get()
```

## Use prompts in production

When deploying prompts in production systems, use an [alias](#add-prompt-tags-and-aliases) like `production` rather than a version index or the latest version. In your production application, load the prompt with `weave.ref(<name>:<alias>).get()`.

Using an alias keeps production behavior stable and predictable. When you're ready to promote a new version, move the alias to that version. All consumers then automatically pick up the change.

A common workflow looks like this:

1. Develop and test a new prompt version.
2. Evaluate the new prompt against datasets or evaluation suites.
3. Move the `production` alias to the new version with `client.set_aliases(new_ref, "production")`.

This approach lets teams safely iterate on prompts without unexpectedly changing production behavior.

## <a aria-label="View and compare prompt versions" />View and compare prompt versions

The Weave UI lets you browse all versions of a prompt and view side-by-side diffs between them, which is useful for reviewing how a prompt has evolved.

To view versions of the prompt in the UI:

1. Navigate to [wandb.ai](https://wandb.ai/) and select your project.
2. In the Weave project sidebar, click **Assets**. This opens the Assets page.
3. In the Assets page, click **Prompts**. This opens the Prompts page where your project's prompts are listed.
4. Under the **Versions** column, click **(x) Versions** for the prompt you want to view. This opens a list of prompt versions.

<img alt="UI of Prompt Assets, showing the versions of the selected prompt." />

1. (Optional) To compare versions of prompts, click the checkboxes beside the listed prompts and then click the **Compare** button in the table toolbar. This lets you see the diff between your prompts.

<img alt="The Compare prompts UI showing a diff between two different prompt versions." />

## Add prompt tags and aliases

You can organize prompt versions using aliases and tags. These labels help you identify and reference specific versions across development, evaluation, and production workflows.

* Alias: A unique name that resolves to a single prompt version. You can move an alias to point to a different version at any time, making it useful for stable references like `production` or `staging`.
* Tag: A descriptive label you attach to a version. A version can have multiple tags. Use tags to categorize and filter versions, such as `reviewed` or `passed-eval`.

To set aliases and tags for your prompts in the UI:

1. In the Weave project sidebar, click **Assets**. This opens the Assets page.
2. In the Assets page, click **Prompts**. Note that you can view assigned **Aliases** and **Tags** in the table.
3. In the Prompts table, click the link of the prompt you want to change.
4. In the prompt detail panel for your selected prompt, the title bar displays the name of the prompt and a specific version. Choose the version of the prompt you want to update.
5. In the panel toolbar, use the controls to add and remove assigned aliases and tags to this specific version.

<img alt="A selected prompt version in the prompt detail panel on the Prompts page, showing controls for Aliases and Tags." />

### Modify tags and aliases in code

You can also manage tags and aliases in code, for tasks such as:

* Publishing prompts with tags and aliases inline.
* Pointing aliases at specific versions and resolving them to load prompts.
* Adding, removing, or listing tags and aliases on versions.

The following code example adds a prompt `my-prompt` to your project with three versions, each having various tags and aliases. Update `'your-team-name/your-project-name'` accordingly for your project.

<CodeGroup>
  ```python Python lines theme={null}
  import weave

  // TODO: Set 'your-team-name/your-project-name' to correct values.
  client = weave.init('your-team-name/your-project-name')

  # Publish two versions of a prompt.
  v0_ref = weave.publish(
      weave.StringPrompt("Answer the user's question: {question}"),
      name="my-prompt",
  )
  v1_ref = weave.publish(
      weave.StringPrompt("Answer the user's question helpfully and concisely: {question}"),
      name="my-prompt",
  )

  # --- Aliases: named pointers to a specific version. ---

  # Point aliases at a version.
  client.set_aliases(v0_ref, "staging")
  client.set_aliases(v1_ref, "production")

  # Retrieve aliases for a version.
  client.get_aliases(v1_ref)  # ["production", "latest"]

  # Move an alias by setting it on a different version.
  client.set_aliases(v0_ref, "production")  # Automatically detaches from v1.

  # Resolve an alias to load the object.
  prompt = weave.ref("my-prompt:production").get()

  # List all aliases in the project.
  client.list_aliases()

  # Remove an alias.
  client.remove_aliases(v0_ref, "production")

  # --- Tags: labels on a specific version. ---

  # Add tags to a version.
  client.add_tags(v0_ref, ["reviewed", "passed-eval"])
  client.add_tags(v1_ref, ["reviewed", "needs-improvement"])

  # Get tags for a version.
  client.get_tags(v0_ref)  # ["passed-eval", "reviewed"]

  # List all distinct tags in the project.
  client.list_tags()

  # Remove tags from a version.
  client.remove_tags(v1_ref, ["needs-improvement"])

  # --- Combined: fetch both in a single call. ---

  tags, aliases = client.get_tags_and_aliases(v0_ref)

  # --- Publish with tags and aliases inline. ---

  ref = weave.publish(
      weave.StringPrompt("Be brief: {question}"),
      name="my-prompt",
      tags=["reviewed"],
      aliases=["production"],
  )

  ```

  ```plaintext TypeScript lines theme={null}
  This feature is not available in the TypeScript SDK yet.
  ```
</CodeGroup>

When you run the example, Weave prints links to your objects in the Weave dashboard. Follow the links to see your prompts.


# Set up automations
Source: https://docs.wandb.ai/weave/guides/evaluation/automations

Create event-driven automations that trigger actions based on monitor metrics and trace activity.

Automations let you define event-driven rules that trigger actions when your project meets specific conditions. Instead of manually watching dashboards for metric changes, you configure an automation once, and W\&B Weave runs an action for you when the event occurs. Use cases include:

* **Threshold alert**: Send a Slack notification when a monitor's average score crosses a threshold.
* **Regression detection**: Send an alert when a scorer detects a drop in accuracy or an increase in toxicity.
* **Deployment gate**: Trigger a webhook when a quality metric exceeds a confidence threshold over a rolling window.
* **Operational monitoring**: Send an alert when error rates or latency metrics change.

You manage automations in the Weave UI, with no code changes required. This guide walks you through creating an automation from the Automations page, creating one from an existing monitor, and viewing and managing automations after you create them.

## Create an automation

Create an automation by configuring the triggering event and selecting what action to take.

### Prerequisites

Before you create the automation that connects them, the triggering event (such as an Op or a monitor) and the action (such as a Slack integration) must exist.

* In Weave, set up an [Op](/weave/guides/tracking/create-call) or a [monitor](/weave/guides/evaluation/monitors) of interest.
* In your Team Settings, set up a [Slack integration](/models/automations/create-automations/slack#add-a-slack-integration) or a [webhook](/models/automations/create-automations/webhook#create-a-webhook).

### Configure the event

The event configuration defines what condition triggers the automation. You select a monitor metric, a time window, and a threshold.

To create an automation:

1. Navigate to [wandb.ai](https://wandb.ai) and open your project.
2. In the project sidebar, click **Automations**.
3. In the Automations page header, click **Create automation**.
4. In the **Create automation** dialog, configure the event:
   * **Event**: Select the type of event that triggers this automation, such as a Weave metric threshold.
   * **Source**: Select the automation source to be either an **Operation** or a **Monitor** from your project.
     * **Operation**: Select the **Op name** whose traces the monitor evaluates. The list contains Ops that have logged at least one trace in the project.
     * **Monitor**: Select the **Monitor** that produces the metric you want to track.
   * **Metric**: Select the metric to measure for the alert.
   * **Metric Threshold**: Define the condition that triggers the automation. Select a comparator (such as "is above") and provide the threshold value (such as `0.9`).
   * **Window**: Configure the window used to trigger the automation to be either **Time-based** or **Count-based**. Select the aggregation function (such as **Average**) and the rolling period (such as **1 day**).
   * **Aggregation**: Set the aggregation function (such as mean, median, minimum) to apply to metric values within the window before comparing to the threshold.
5. Review the **Alert preview** graph to confirm the thresholds alert the way you expect.
6. Click **Next step** to proceed to action configuration.

### Configure the action

The action configuration defines what action happens when the event meets the condition.

To configure the action:

1. In the **Step 2 of 3** panel, configure:
   * **Team**: Select the team to receive the notification.
   * **Action type**: Select the type of action to perform, such as sending a **Slack notification**.
   * **Action details**: Depending on the action type, provide the required configuration:
     * For Slack notifications, select the **Slack channel** to send the notification to.
     * For webhooks, provide the **Webhook** name and **Payload**.
2. Click **Next step** to proceed to the summary.

### Review and save

The summary step displays a complete overview of the automation before you save it.

To save:

1. Configure the automation metadata:
   * **Automation name**: Provide a name for the automation that appears in the Automations table.
   * **Description** (Optional): Add a description to help identify the automation's purpose.
2. Click **Create automation** to create the automation.

The new automation appears in the Automations table on the Automations page.

## Create an automation from a monitor

You can also create automations directly from a monitor's detail view. This approach pre-fills the event configuration with the monitor's context, which makes the setup faster. Before you create the automation, set up a [Slack integration](/models/automations/create-automations/slack#add-a-slack-integration) or a [webhook](/models/automations/create-automations/webhook#create-a-webhook) that you want to use as your selected action.

To create an automation from a monitor:

1. In the Weave project sidebar, click **Monitors**.
2. In the Monitors table, click the name of the monitor to open its detail panel.
3. In the monitor detail panel, select the **Automations** tab to see any existing automations for this monitor.
4. In the Automations section toolbar, click **Create automation**.
5. In the **Create automation** panel, Weave pre-fills the event configuration with the selected monitor's details. Adjust the configuration as needed:
   * **Metric**: Select the metric to measure for the alert.
   * **Metric Threshold**: Define the condition that triggers the automation. Select a comparator (such as "is above") and provide the threshold value (such as `0.9`).
   * **Window**: Configure the window used to trigger the automation to be either **Time-based** or **Count-based**. Select the aggregation function (such as **Average**) and the rolling period (such as **1 day**).
   * **Aggregation**: Set the aggregation function (such as mean, median, minimum) to apply to metric values within the window before comparing to the threshold.
6. Click **Next** and complete the remaining steps (select action and save) as described in [Create an automation](#create-an-automation).

## View and manage automations

The **Automations** page lists all automations in your project. The table displays the following information for each automation:

* **Automation**: The automation name and description.
* **Event type**: The type of event that triggers the automation.
* **Action type**: The action that the automation triggers, such as generating a Slack notification.
* **Date created**: When you created the automation.
* **Last execution**: When the automation last triggered.

To view automations:

1. In the Weave project sidebar, click **Automations**.
2. Use the search field in the table toolbar to filter automations by name.

You can click the name of an automation to open the automation drawer and review the full details of the automation.

The History tab displays a record of all triggered automations. When an automation triggers, Weave creates an automation instance that records what happened. You can inspect these instances to see when and why an automation triggered.


# Use builtin scorers
Source: https://docs.wandb.ai/weave/guides/evaluation/builtin_scorers

Use Weave's predefined scorers for evaluating your AI applications

W\&B Weave offers several predefined scorers for evaluating your AI applications, such as [Hallucination detection](#hallucinationfreescorer) and [Summarization quality](#summarizationscorer). These can help you quickly define an evaluation and score your application's outputs. This page describes each built-in scorer, what it measures, and how to use it in an evaluation.

<Note>
  Local scorers are only available for the Weave Python SDK. They aren't yet available for the Weave TypeScript SDK.

  To use Weave scorers in TypeScript, see [function-based scorers](/weave/guides/evaluation/scorers#function-based-scorers).
</Note>

## Installation

To use Weave's predefined scorers, install some additional dependencies:

```bash theme={null}
pip install weave[scorers]
```

**LLM-evaluators**

The predefined scorers that use LLMs integrate with litellm automatically. You don't need to pass an LLM client. Set the `model_id` instead. See the [supported models](https://docs.litellm.ai/docs/providers).

## `HallucinationFreeScorer`

This scorer checks whether your AI system's output includes any hallucinations based on the input data.

```python lines theme={null}
from weave.scorers import HallucinationFreeScorer

scorer = HallucinationFreeScorer()
```

**Customization**

* Customize the `system_prompt` and `user_prompt` fields of the scorer to define what "hallucination" means for you.

**Notes**

* The `score` method expects an input column named `context`. If your dataset uses a different name, [use the `column_map` attribute](#use-column-mapping) to map `context` to the dataset column.

The following example uses `HallucinationFreeScorer` in the context of an evaluation:

```python lines theme={null}
import asyncio
import weave
from weave.scorers import HallucinationFreeScorer

# Initialize scorer with a column mapping if needed.
hallucination_scorer = HallucinationFreeScorer(
    model_id="openai/gpt-4o", # or any other model supported by litellm
    column_map={"context": "input", "output": "other_col"}
)

# Create dataset
dataset = [
    {"input": "John likes various types of cheese."},
    {"input": "Pepe likes various types of cheese."},
]

@weave.op
def model(input: str) -> str:
    return "The person's favorite cheese is cheddar."

# Run evaluation
evaluation = weave.Evaluation(
    dataset=dataset,
    scorers=[hallucination_scorer],
)
result = asyncio.run(evaluation.evaluate(model))
print(result)
# Example output:
# {'HallucinationFreeScorer': {'has_hallucination': {'true_count': 2, 'true_fraction': 1.0}}, 'model_latency': {'mean': ...}}
```

***

## `SummarizationScorer`

Use an LLM to compare a summary to the original text and evaluate the quality of the summary.

```python lines theme={null}
from weave.scorers import SummarizationScorer

scorer = SummarizationScorer(
    model_id="openai/gpt-4o"  # or any other model supported by litellm
)
```

**How it works**

This scorer evaluates summaries in two ways:

1. **Entity density:** Checks the ratio of unique entities (such as names, places, or objects) mentioned in the summary to the total word count in the summary to estimate the "information density" of the summary. Uses an LLM to extract the entities. Similar to how entity density is used in the [Chain of Density paper](https://arxiv.org/abs/2309.04269).
2. **Quality grading:** An LLM evaluator grades the summary as `poor`, `ok`, or `excellent`. These grades are then mapped to scores (0.0 for poor, 0.5 for ok, and 1.0 for excellent) for aggregate performance evaluation.

**Customization**

* Adjust `summarization_evaluation_system_prompt` and `summarization_evaluation_prompt` to tailor the evaluation process.

**Notes**

* The scorer uses litellm internally.
* The `score` method expects the original text (the text being summarized) to be present in the `input` column. [Use the `column_map` attribute](#use-column-mapping) if your dataset uses a different name.

The following example uses `SummarizationScorer` in the context of an evaluation:

```python lines theme={null}
import asyncio
import weave
from weave.scorers import SummarizationScorer

class SummarizationModel(weave.Model):
    @weave.op()
    async def predict(self, input: str) -> str:
        return "This is a summary of the input text."

# Initialize scorer
summarization_scorer = SummarizationScorer(
    model_id="openai/gpt-4o"  # or any other model supported by litellm
)
# Create dataset
dataset = [
    {"input": "The quick brown fox jumps over the lazy dog."},
    {"input": "Artificial Intelligence is revolutionizing various industries."}
]
# Run evaluation
evaluation = weave.Evaluation(dataset=dataset, scorers=[summarization_scorer])
results = asyncio.run(evaluation.evaluate(SummarizationModel()))
print(results)
# Example output:
# {'SummarizationScorer': {'is_entity_dense': {'true_count': 0, 'true_fraction': 0.0}, 'summarization_eval_score': {'mean': 0.0}, 'entity_density': {'mean': 0.0}}, 'model_latency': {'mean': ...}}
```

***

## `OpenAIModerationScorer`

The `OpenAIModerationScorer` uses OpenAI's Moderation API to check whether the AI system's output contains disallowed content, such as hate speech or explicit material.

```python lines theme={null}
from weave.scorers import OpenAIModerationScorer

scorer = OpenAIModerationScorer()
```

**How it works**

* Sends the AI's output to the OpenAI Moderation endpoint and returns a structured response indicating whether the content is flagged.

The following example uses `OpenAIModerationScorer` in the context of an evaluation:

```python lines theme={null}
import asyncio
import weave
from weave.scorers import OpenAIModerationScorer

class MyModel(weave.Model):
    @weave.op
    async def predict(self, input: str) -> str:
        return input

# Initialize scorer
moderation_scorer = OpenAIModerationScorer()

# Create dataset
dataset = [
    {"input": "I love puppies and kittens!"},
    {"input": "I hate everyone and want to hurt them."}
]

# Run evaluation
evaluation = weave.Evaluation(dataset=dataset, scorers=[moderation_scorer])
results = asyncio.run(evaluation.evaluate(MyModel()))
print(results)
# Example output:
# {'OpenAIModerationScorer': {'flagged': {'true_count': 1, 'true_fraction': 0.5}, 'categories': {'violence': {'true_count': 1, 'true_fraction': 1.0}}}, 'model_latency': {'mean': ...}}
```

***

## `EmbeddingSimilarityScorer`

The `EmbeddingSimilarityScorer` computes the cosine similarity between the embeddings of the AI system's output and a target text from your dataset. Use it to measure how similar the AI's output is to a reference text.

```python lines theme={null}
from weave.scorers import EmbeddingSimilarityScorer

similarity_scorer = EmbeddingSimilarityScorer(
    model_id="openai/text-embedding-3-small",  # or any other model supported by litellm
    threshold=0.4  # the cosine similarity threshold
)
```

**Parameters**

* `threshold` (float): The minimum cosine similarity score (between -1 and 1) needed to consider the two texts similar (defaults to `0.5`).

The following example uses `EmbeddingSimilarityScorer` in the context of an evaluation:

```python lines theme={null}
import asyncio
import weave
from weave.scorers import EmbeddingSimilarityScorer

# Initialize scorer
similarity_scorer = EmbeddingSimilarityScorer(
    model_id="openai/text-embedding-3-small",  # or any other model supported by litellm
    threshold=0.7
)
# Create dataset
dataset = [
    {
        "input": "He's name is John",
        "target": "John likes various types of cheese.",
    },
    {
        "input": "He's name is Pepe.",
        "target": "Pepe likes various types of cheese.",
    },
]
# Define model
@weave.op
def model(input: str) -> str:
    return "John likes various types of cheese."

# Run evaluation
evaluation = weave.Evaluation(
    dataset=dataset,
    scorers=[similarity_scorer],
)
result = asyncio.run(evaluation.evaluate(model))
print(result)
# Example output:
# {'EmbeddingSimilarityScorer': {'is_similar': {'true_count': 1, 'true_fraction': 0.5}, 'similarity_score': {'mean': 0.844851403}}, 'model_latency': {'mean': ...}}
```

***

## `ValidJSONScorer`

The `ValidJSONScorer` checks whether the AI system's output is valid JSON. This scorer is useful when you expect the output to be in JSON format and need to verify its validity.

```python lines theme={null}
from weave.scorers import ValidJSONScorer

json_scorer = ValidJSONScorer()
```

The following example uses `ValidJSONScorer` in the context of an evaluation:

```python lines theme={null}
import asyncio
import weave
from weave.scorers import ValidJSONScorer

class JSONModel(weave.Model):
    @weave.op()
    async def predict(self, input: str) -> str:
        # This is a placeholder.
        # In a real scenario, this would generate JSON.
        return '{"key": "value"}'

model = JSONModel()
json_scorer = ValidJSONScorer()

dataset = [
    {"input": "Generate a JSON object with a key and value"},
    {"input": "Create an invalid JSON"}
]

evaluation = weave.Evaluation(dataset=dataset, scorers=[json_scorer])
results = asyncio.run(evaluation.evaluate(model))
print(results)
# Example output:
# {'ValidJSONScorer': {'json_valid': {'true_count': 2, 'true_fraction': 1.0}}, 'model_latency': {'mean': ...}}
```

***

## `ValidXMLScorer`

The `ValidXMLScorer` checks whether the AI system's output is valid XML. Use it when you expect XML-formatted outputs.

```python lines theme={null}
from weave.scorers import ValidXMLScorer

xml_scorer = ValidXMLScorer()
```

The following example uses `ValidXMLScorer` in the context of an evaluation:

```python lines theme={null}
import asyncio
import weave
from weave.scorers import ValidXMLScorer

class XMLModel(weave.Model):
    @weave.op()
    async def predict(self, input: str) -> str:
        # This is a placeholder. In a real scenario, this would generate XML.
        return '<root><element>value</element></root>'

model = XMLModel()
xml_scorer = ValidXMLScorer()

dataset = [
    {"input": "Generate a valid XML with a root element"},
    {"input": "Create an invalid XML"}
]

evaluation = weave.Evaluation(dataset=dataset, scorers=[xml_scorer])
results = asyncio.run(evaluation.evaluate(model))
print(results)
# Example output:
# {'ValidXMLScorer': {'xml_valid': {'true_count': 2, 'true_fraction': 1.0}}, 'model_latency': {'mean': ...}}
```

***

## `PydanticScorer`

The `PydanticScorer` validates the AI system's output against a Pydantic model to ensure it adheres to a specified schema or data structure.

```python lines theme={null}
from weave.scorers import PydanticScorer
from pydantic import BaseModel

class FinancialReport(BaseModel):
    revenue: int
    year: str

pydantic_scorer = PydanticScorer(model=FinancialReport)
```

***

## RAGAS - `ContextEntityRecallScorer`

The `ContextEntityRecallScorer` estimates context recall by extracting entities from both the AI system's output and the provided context, then computing the recall score. It's based on the Retrieval Augmented Generation Assessment ([RAGAS](https://github.com/explodinggradients/ragas)) evaluation library.

```python lines theme={null}
from weave.scorers import ContextEntityRecallScorer

entity_recall_scorer = ContextEntityRecallScorer(
    model_id="openai/gpt-4o"
)
```

**How it works**

* Uses an LLM to extract unique entities from the output and context and calculates recall.
* **Recall** indicates the proportion of important entities from the context that are captured in the output.
* Returns a dictionary with the recall score.

**Notes**

* Expects a `context` column in your dataset. [Use the `column_map` attribute](#use-column-mapping) if the column name is different.

***

## RAGAS - `ContextRelevancyScorer`

The `ContextRelevancyScorer` evaluates the relevancy of the provided context to the AI system's output. It's based on the RAGAS evaluation library.

```python lines theme={null}
from weave.scorers import ContextRelevancyScorer

relevancy_scorer = ContextRelevancyScorer(
    model_id="openai/gpt-4o",  # or any other model supported by litellm
    relevancy_prompt="""
Given the following question and context, rate the relevancy of the context to the question on a scale from 0 to 1.

Question: {question}
Context: {context}
Relevancy Score (0-1):
"""
)
```

**How it works**

* Uses an LLM to rate the relevancy of the context to the output on a scale from 0 to 1.
* Returns a dictionary with the `relevancy_score`.

**Notes**

* Expects a `context` column in your dataset. [Use the `column_map` attribute](#use-column-mapping) if the column name is different.
* Customize the `relevancy_prompt` to define how relevancy is assessed.

The following example uses `ContextEntityRecallScorer` and `ContextRelevancyScorer` together in the context of an evaluation:

```python lines theme={null}
import asyncio
from textwrap import dedent
import weave
from weave.scorers import ContextEntityRecallScorer, ContextRelevancyScorer

class RAGModel(weave.Model):
    @weave.op()
    async def predict(self, question: str) -> str:
        "Retrieve relevant context"
        return "Paris is the capital of France."

# Define prompts
relevancy_prompt: str = dedent("""
    Given the following question and context, rate the relevancy of the context to the question on a scale from 0 to 1.

    Question: {question}
    Context: {context}
    Relevancy Score (0-1):
    """)
# Initialize scorers
entity_recall_scorer = ContextEntityRecallScorer()
relevancy_scorer = ContextRelevancyScorer(relevancy_prompt=relevancy_prompt)
# Create dataset
dataset = [
    {
        "question": "What is the capital of France?",
        "context": "Paris is the capital city of France."
    },
    {
        "question": "Who wrote Romeo and Juliet?",
        "context": "William Shakespeare wrote many famous plays."
    }
]
# Run evaluation
evaluation = weave.Evaluation(
    dataset=dataset,
    scorers=[entity_recall_scorer, relevancy_scorer]
)
results = asyncio.run(evaluation.evaluate(RAGModel()))
print(results)
# Example output:
# {'ContextEntityRecallScorer': {'recall': {'mean': ...}}, 
# 'ContextRelevancyScorer': {'relevancy_score': {'mean': ...}}, 
# 'model_latency': {'mean': ...}}
```

**Note:** The built-in scorers are calibrated using OpenAI models, such as `openai/gpt-4o` and `openai/text-embedding-3-small`. To use a different provider, update the `model_id` field. For example, to use an Anthropic model:

```python lines theme={null}
from weave.scorers import SummarizationScorer

# Switch to Anthropic's Claude model
summarization_scorer = SummarizationScorer(
    model_id="anthropic/claude-3-5-sonnet-20240620"
)
```


# Set up custom monitors
Source: https://docs.wandb.ai/weave/guides/evaluation/custom-monitors

Passively score production traffic to surface trends and issues

This page shows you how to set up custom monitors in W\&B Weave so you can passively score production traffic and surface trends and issues in your LLM applications. Use this guide when you want to define your own evaluation criteria, beyond Weave's preset signals, for production traces.

Monitors use LLM judges to passively score production traffic to surface trends and issues in your LLM applications. For example, you can monitor your application's responses for correctness or helpfulness, or you can monitor user input to identify trends in what users ask your agents about. Monitors automatically store all scoring results in Weave's database, so you can analyze historical trends and patterns.

You can monitor text, images, and audio in your application's input and output.

Monitors require no code changes to your application. Set them up using the Weave UI.

If you need to actively intervene in your application's behavior based on scores, use [guardrails](/weave/guides/evaluation/guardrails) instead.

## Signals and custom monitors

Use [signals](/weave/guides/evaluation/monitors), preset automated scorers for production traces, to get started with production monitoring quickly, then add custom monitors for evaluation criteria specific to your application.

|                     | Signals                                                                  | Custom monitors                                         |
| ------------------- | ------------------------------------------------------------------------ | ------------------------------------------------------- |
| **Configuration**   | One-click enable, no prompt writing                                      | Full control over scoring prompt, model, and parameters |
| **Scope**           | Preset quality and error classifiers                                     | Any evaluation criteria you define                      |
| **Trace selection** | Automatic (successful root traces for quality, failed traces for errors) | Configurable operations, filters, and sampling rate     |
| **Model**           | Serverless Inference (preset)                                            | Any commercial or Serverless Inference model            |
| **Use case**        | Quick production monitoring with proven classifiers                      | Custom evaluation criteria specific to your application |

## Create a monitor in Weave

To create a custom monitor in Weave:

1. Open the [W\&B UI](https://wandb.ai/home) and then open your Weave project.

2. From the Weave sidebar, select **Monitors** and then select the **+ New Monitor** button. This opens the **Create new monitor** modal dialog.

3. In the **Create new monitor** menu, configure the following fields:
   * **Name**: Must start with a letter or number. Can contain letters, numbers, hyphens, and underscores.
   * **Description** (Optional): Explain what the monitor does.
   * **Active monitor** toggle: Turn the monitor on or off.
   * **Calls to monitor**:
     * **Operations**: Choose one or more `@weave.op`s to monitor. You must log at least one trace that uses the op before it appears in the list of available ops.
     * **Filter** (Optional): Narrow which calls are eligible (for example, by `max_tokens` or `top_p`).
     * **Sampling rate**: The percentage of calls to score (0% to 100%).
       <Tip>
         A lower sampling rate reduces costs, since each scoring call has an associated cost.
       </Tip>
   * **LLM-as-a-judge configuration**:
     * **Scorer name**: Must start with a letter or number. Can contain letters, numbers, hyphens, and underscores.
     * **Score Audio**: Filters the available LLM models to display only audio-enabled models, and opens the **Media Scoring JSON Paths** field.
     * **Score Images**: Filters the available LLM models to display only image-enabled models, and opens the **Media Scoring JSON Paths** field.
     * **Judge model**: Select the model to score your ops. The menu contains commercial LLM models you have configured in your W\&B account, as well as [Serverless Inference models](/inference/models). Audio-enabled models have an **Audio Input** label beside their names. For the selected model, configure the following settings:
       * **Configuration name**: A name for this model configuration.
       * **System prompt**: Defines the judging model's role and persona, for example, "You are an impartial AI judge."
       * **Response format**: The format the judge uses to output its response, such as a `json_object` or plain `text`.
       * **Scoring prompt**: The evaluation task used to score your ops. You can reference [prompt variables](/weave/guides/evaluation/scorers#access-variables-from-your-ops-in-scoring-prompts) from your ops in your scoring prompts. For example, "Evaluate whether `{output}` is accurate based on `{ground_truth}`."
     * **Media Scoring JSON Paths**: Specify JSONPath expressions (RFC 9535) to extract media from your trace data. If you specify no paths, the monitor includes all scorable media from user messages. This field appears when you enable **Score Audio** or **Score Images**.

4. After you configure the monitor's fields, select **Create monitor**. This adds the monitor to your Weave project. When your code starts generating traces, you can review the scores in the **Traces** tab by selecting the monitor's name and reviewing the data in the resulting panel.

You can also [compare](/weave/guides/tools/comparison) and visualize the monitor's trace data in the Weave UI, or download it in formats such as CSV and JSON using the download button (<Icon icon="download" />) in the **Traces** tab.

Weave automatically stores all scorer results in the [Call](/weave/guides/tracking/tracing#calls) object's `feedback` field.

### Example: Create a truthfulness monitor

The following end-to-end example walks through creating a monitor that evaluates the truthfulness of generated statements. By the end, you have an active monitor scoring a sample op in your Weave project, with scored traces visible in the W\&B UI.

1. Define a function that generates statements. Some statements are truthful, others are not:

<Tabs>
  <Tab title="Python">
    ```python lines theme={null}
    import weave
    import random
    import openai

    weave.init("my-team/my-weave-project")

    client = openai.OpenAI()

    @weave.op()
    def generate_statement(ground_truth: str) -> str:
        if random.random() < 0.5:
            response = client.chat.completions.create(
                model="gpt-4.1",
                messages=[
                    {
                        "role": "user",
                        "content": f"Generate a statement that is incorrect based on this fact: {ground_truth}"
                    }
                ]
            )
            return response.choices[0].message.content
        else:
            return ground_truth

    generate_statement("The Earth revolves around the Sun.")
    ```
  </Tab>

  <Tab title="TypeScript">
    ```typescript lines theme={null}
    import * as weave from 'weave';
    import OpenAI from 'openai';

    await weave.init('my-team/my-weave-project');

    const client = new OpenAI();

    const generateStatement = weave.op(async (ground_truth: string): Promise<string> => {
      if (Math.random() < 0.5) {
        const response = await client.chat.completions.create({
          model: 'gpt-4.1',
          messages: [
            {
              role: 'user',
              content: `Generate a statement that is incorrect based on this fact: ${ground_truth}`,
            },
          ],
        });
        return response.choices[0]?.message?.content ?? '';
      }
      return ground_truth;
    });

    await generateStatement("The Earth revolves around the Sun.");
    ```
  </Tab>
</Tabs>

2. Run the function at least once to log a trace in your project. This makes the op available for monitoring in the W\&B UI.

3. Open your Weave project in the W\&B UI and select **Monitors** from the sidebar. Then select **New Monitor**.

4. In the **Create new monitor** menu, configure the fields using the following values:
   * **Name**: `truthfulness-monitor`
   * **Description**: `Evaluates the truthfulness of generated statements.`
   * **Active monitor**: Toggle **on**.
   * **Operations**: Select `generate_statement`.
   * **Sampling rate**: Set to `100%` to score every call.
   * **Scorer name**: `truthfulness-scorer`
   * **Judge model**: `o3-mini-2025-01-31`
   * **System prompt**: `You are an impartial AI judge. Your task is to evaluate the truthfulness of statements.`
   * **Response format**: `json_object`
   * **Scoring prompt**:
     ```text theme={null}
     Evaluate whether the output statement is accurate based on the input statement.

     This is the input statement: {ground_truth}

     This is the output statement: {output}

     The response should be a JSON object with the following fields:
     - is_true: a boolean stating whether the output statement is true or false based on the input statement.
     - reasoning: your reasoning as to why the statement is true or false.
     ```

5. Select **Create monitor**. This adds the monitor to your Weave project.

6. In your script, invoke your function using statements with differing degrees of truthfulness to test the scoring function:

<Tabs>
  <Tab title="Python">
    ```python lines theme={null}
    generate_statement("The Earth revolves around the Sun.")
    generate_statement("Water freezes at 0 degrees Celsius.")
    generate_statement("The Great Wall of China was built over several centuries.")
    ```
  </Tab>

  <Tab title="TypeScript">
    ```typescript lines theme={null}
    await generateStatement("The Earth revolves around the Sun.");
    await generateStatement("Water freezes at 0 degrees Celsius.");
    await generateStatement("The Great Wall of China was built over several centuries.");
    ```
  </Tab>
</Tabs>

7. After running the script using several different statements, open the W\&B UI and go to the **Traces** tab. Select any **LLMAsAJudgeScorer.score** trace to see the results.

<img alt="Monitor trace" />

You now have a working truthfulness monitor that scores each invocation of `generate_statement` and stores the results alongside the original trace, ready for analysis and comparison in the Weave UI.


# Create dynamic Leaderboards in Evaluations
Source: https://docs.wandb.ai/weave/guides/evaluation/dynamic_leaderboards

Dynamic Leaderboards let you configure, customize, save, and update Leaderboard views directly from an evaluation.

This guide shows you how to use dynamic Leaderboards to visualize, customize, save, and update Leaderboard views directly from your Weave Evaluations. Dynamic Leaderboards help you track and compare model performance over time as new experiments accumulate, so you can maintain persistent, up-to-date views without re-creating them each time. This page is for users who are already running Weave Evaluations and want to organize results into shareable, reusable views.

When working with Weave Evaluations, you can visualize and customize your experiment results as Leaderboards.

Saved Leaderboard views are dynamic. As you add new evaluation runs and as results match the saved filters, the Leaderboard automatically updates to include them, without requiring manual reconfiguration.

This lets you use views as persistent Leaderboards that evolve alongside your experiments.

## Visualize evaluation results in a Leaderboard

When your project contains Weave Evaluation data, you can use the evaluation table to quickly create a Weave Leaderboard view based on a filtered subset of results.

To create a Weave Leaderboard:

1. Navigate to [wandb.ai](https://wandb.ai).
2. In the Weave sidebar menu, click **Evaluations**.
3. Apply filters to the evaluation table to narrow the data to the models, datasets, or runs you want to compare.
4. In the **evaluation table toolbar**, click **Visualize**.
   Weave automatically creates a Leaderboard panel using only the data currently filtered in the table.
5. In the Leaderboard panel header, click **Configure** to open the **Edit Leaderboard** panel.
   The **Edit Leaderboard** panel gives you fine-grained control over how models, datasets, scorers, and metrics appear.

The following shows how a filtered evaluation table is visualized as a Leaderboard and where to configure the resulting Leaderboard.

<Frame>
  <img alt="Evaluations page showing the evaluation table with filters applied, the Visualize button in the table toolbar, and the resulting Leaderboard panel on the right with the Configure button in the panel header." />
</Frame>

### Configure Leaderboard elements with visibility and custom names

The **Edit Leaderboard** panel lets you control which elements appear in the Leaderboard and how they're labeled. The following shows the **Edit Leaderboard** panel with four configuration tabs: Models, Datasets, Scorers, and Metrics.

<Frame>
  <img alt="Evaluations page showing the Edit Leaderboard panel open on the right, with tabs for Models, Datasets, Scorers, and Metrics used to configure the leaderboard." />
</Frame>

In the **Edit Leaderboard** panel, you can:

* **Activate or deactivate display**\
  Select which models, datasets, scorers, and metrics appear in the Leaderboard by checking or unchecking them.

* **Rename models, datasets, and scorers**\
  Assign display-friendly names (for example, renaming a model run to `GPT-4` or a dataset to `JokesV1`).

  Renamed items:

  * Update immediately in the Leaderboard.
  * Remain clickable so you can still open the underlying reference in the side panel.
  * Automatically propagate anywhere the Leaderboard view is used.

This makes it easier to compare experiments using meaningful, human-readable names without changing the underlying objects.

### Configure Leaderboard metric behavior and coloring

Different metrics require different interpretations of "good" performance. The **Edit Leaderboard** panel lets you set the direction for each metric so that visual highlighting matches your intent.

In the **Edit Leaderboard** panel, for each metric, you can specify one of the following directions:

* **Higher values are better**
* **Lower values are better**

This setting directly affects Leaderboard coloring:

* Green highlights the *better* value.
* Red highlights the *worse* value.
* Colors automatically invert when you switch between "higher is better" and "lower is better".

This ensures that visual cues remain accurate across different types of metrics (for example, accuracy compared to latency or error rate).

## Save and reuse Leaderboard views

After you customize a Leaderboard, you can save it as a named view so you and your team can return to the same configuration later. In the **Edit Leaderboard** panel, click **Save** to save your customized Leaderboard as a reusable view.

The saved Leaderboard view captures:

* Selected models, datasets, scorers, and metrics.
* Renamed display labels.
* Metric direction settings (higher or lower is better).
* Applied filters.

### Switch between saved views

Click the **menu (<Icon icon="bars" />)** button next to the Evaluations page title to open saved views. You can:

* Return to the **default** view to see the full dataset.
* Reopen a saved view to restore all customizations.

When you reopen a saved view, all renames and metric settings are preserved.


# Log evaluation data from your code
Source: https://docs.wandb.ai/weave/guides/evaluation/evaluation_logger

Flexible, incremental way to log evaluation data from Python and TypeScript code

This guide shows you how to use `EvaluationLogger` to record predictions and scores from your existing Python or TypeScript code, so you can evaluate model performance in Weave without first defining a full dataset and scorer suite.

This approach is helpful in complex workflows where the entire dataset or all scorers might not be defined upfront.

In contrast to the standard `Evaluation` object, which requires a predefined `Dataset` and list of `Scorer` objects, the `EvaluationLogger` lets you log individual predictions and their associated scores incrementally as they become available.

<Info>
  **Prefer a more structured evaluation?**

  If you prefer a more opinionated evaluation framework with predefined datasets and scorers, see [Weave's standard Evaluation framework](../core-types/evaluations).

  The `EvaluationLogger` offers flexibility while the standard framework offers structure and guidance.
</Info>

## Basic workflow

Following these steps records a complete evaluation in Weave, with per-prediction scores and an aggregated summary that you can review in the Weave UI.

1. *Initialize the logger:* Create an instance of `EvaluationLogger`, optionally providing metadata about the `model` and `dataset`. Weave uses defaults if omitted.
   <Note>
     To capture token usage and cost for LLM calls (for example, OpenAI), initialize `EvaluationLogger` before any LLM invocations.
     If you call your LLM first and then log predictions afterward, Weave doesn't capture token and cost data.
   </Note>
2. *Log predictions:* Call `log_prediction()` for each input and output pair from your system.
3. *Log scores:* Use the returned `ScoreLogger` to `log_score()` for the prediction. Multiple scores per prediction are supported.
4. *Finish prediction:* Always call `finish()` after logging scores for a prediction to finalize it.
5. *Log summary:* After all predictions are processed, call `log_summary()` to aggregate scores and add optional custom metrics.

<Warning>
  After calling `finish()` on a prediction, no more scores can be logged for it.
</Warning>

For a Python example that demonstrates this workflow, see the [Basic example](#basic-example). If the output and all scores are available at once, Python users can combine steps 2 through 4 into a single call using [`log_example()`](#simplified-logging-with-log_example).

## Basic example

The following example shows how to use `EvaluationLogger` to log predictions and scores inline with your existing code.

<Tabs>
  <Tab title="Python">
    The `user_model` model function is defined and applied to a list of inputs. For each example:

    * The input and output are logged using `log_prediction`.
    * A correctness score (`correctness_score`) is logged through `log_score`.
    * `finish()` finalizes logging for that prediction.

    Finally, `log_summary` records any aggregate metrics and triggers automatic score summarization in Weave.

    ```python lines theme={null}
    import weave
    from openai import OpenAI
    from weave import EvaluationLogger

    weave.init('your-team/your-project')

    # Initialize EvaluationLogger BEFORE calling the model to ensure token tracking
    eval_logger = EvaluationLogger(
        model="my_model",
        dataset="my_dataset"
    )

    # Example input data (this can be any data structure you want)
    eval_samples = [
        {'inputs': {'a': 1, 'b': 2}, 'expected': 3},
        {'inputs': {'a': 2, 'b': 3}, 'expected': 5},
        {'inputs': {'a': 3, 'b': 4}, 'expected': 7},
    ]

    # Example model logic using OpenAI
    @weave.op
    def user_model(a: int, b: int) -> int:
        oai = OpenAI()
        response = oai.chat.completions.create(
            messages=[{"role": "user", "content": f"What is {a}+{b}?"}],
            model="gpt-4o-mini"
        )
        # Use the response in some way (here we just return a + b for simplicity)
        return a + b

    # Iterate through examples, predict, and log
    for sample in eval_samples:
        inputs = sample["inputs"]
        model_output = user_model(**inputs) # Pass inputs as kwargs

        # Log the prediction input and output
        pred_logger = eval_logger.log_prediction(
            inputs=inputs,
            output=model_output
        )

        # Calculate and log a score for this prediction
        expected = sample["expected"]
        correctness_score = model_output == expected
        pred_logger.log_score(
            scorer="correctness", # Simple string name for the scorer
            score=correctness_score
        )

        # Finish logging for this specific prediction
        pred_logger.finish()

    # Log a final summary for the entire evaluation.
    # Weave auto-aggregates the 'correctness' scores logged above.
    summary_stats = {"subjective_overall_score": 0.8}
    eval_logger.log_summary(summary_stats)

    print("Evaluation logging complete. View results in the Weave UI.")
    ```
  </Tab>

  <Tab title="TypeScript">
    The TypeScript SDK provides two API patterns:

    * **Fire-and-forget API (recommended for most cases)**: Use `logPrediction()` without `await` for synchronous, non-blocking logging.
    * **Awaitable API**: Use `logPredictionAsync()` with `await` when you need to ensure operations complete before proceeding.

    Use fire-and-forget for:

    * **High throughput**: Process multiple predictions in parallel without waiting for each logging operation.
    * **Minimal code disruption**: Add evaluation logging without restructuring your existing async/await flow.
    * **Simplicity**: Less boilerplate code and cleaner syntax for most evaluation scenarios.

    The fire-and-forget pattern is safe because `logSummary()` automatically waits for all pending operations to complete before aggregating results.

    The following example evaluates model predictions with the fire-and-forget pattern. It sets up an evaluation logger, runs a model on three test samples, and then logs the prediction without using await:

    ```typescript lines {36,50} theme={null}
    import weave, {EvaluationLogger} from 'weave';
    import OpenAI from 'openai';

    await weave.init('your-team/your-project');

    // Initialize EvaluationLogger BEFORE calling the model to ensure token tracking
    const evalLogger = new EvaluationLogger({
      name: 'my-eval',
      model: 'my_model',
      dataset: 'my_dataset'
    });

    // Example input data
    const evalSamples = [
      {inputs: {a: 1, b: 2}, expected: 3},
      {inputs: {a: 2, b: 3}, expected: 5},
      {inputs: {a: 3, b: 4}, expected: 7},
    ];

    // Example model logic using OpenAI
    const userModel = weave.op(async function userModel(a: number, b: number): Promise<number> {
      const oai = new OpenAI();
      const response = await oai.chat.completions.create({
        messages: [{role: 'user', content: `What is ${a}+${b}?`}],
        model: 'gpt-4o-mini'
      });
      return a + b;
    });

    // Iterate through examples, predict, and log using fire-and-forget pattern
    for (const sample of evalSamples) {
      const {inputs} = sample;
      const modelOutput = await userModel(inputs.a, inputs.b);

      // Fire-and-forget: No await needed for logPrediction
      const scoreLogger = evalLogger.logPrediction(inputs, modelOutput);

      // Calculate and log a score for this prediction
      const correctnessScore = modelOutput === sample.expected;

      // Fire-and-forget: No await needed for logScore
      scoreLogger.logScore('correctness', correctnessScore);

      // Fire-and-forget: No await needed for finish
      scoreLogger.finish();
    }

    // logSummary waits for all pending operations to complete internally
    const summaryStats = {subjective_overall_score: 0.8};
    await evalLogger.logSummary(summaryStats);

    console.log('Evaluation logging complete. View results in the Weave UI.');
    ```

    Use the awaitable API when you need to ensure each operation completes before proceeding, such as when managing error handling or sequential dependencies.

    In the following example, instead of calling `logPrediction()` without `await`, it uses `logPredictionAsync()` with `await` to ensure each operation completes before proceeding to the next one:

    ```typescript lines theme={null}
    // Use logPredictionAsync instead of logPrediction
    const scoreLogger = await evalLogger.logPredictionAsync(inputs, modelOutput);

    // Await each operation
    await scoreLogger.logScore('correctness', correctnessScore);
    await scoreLogger.finish();
    ```
  </Tab>
</Tabs>

## Simplified logging with `log_example()`

Use `log_example()` to log inputs, an output, and scores in a single call. This convenience method combines `log_prediction()`, `log_score()`, and `finish()` into one step, and it's useful when you already have the inputs, model outputs, and scores ready to log, such as during batch or offline evaluations.

```python lines theme={null}
import weave
from weave import EvaluationLogger

weave.init('your-team-name/your-project-name')

eval_logger = EvaluationLogger(
    model="my_model",
    dataset="my_dataset"
)

eval_samples = [
    {'inputs': {'a': 1, 'b': 2}, 'expected': 3},
    {'inputs': {'a': 2, 'b': 3}, 'expected': 5},
    {'inputs': {'a': 3, 'b': 4}, 'expected': 7},
]

for sample in eval_samples:
    inputs = sample['inputs']
    output = inputs['a'] + inputs['b']

    eval_logger.log_example(
        inputs=inputs,
        output=output,
        scores={"correctness": output == sample['expected']}
    )

eval_logger.log_summary({"avg_score": 1.0})
```

The previous `log_example()` call is equivalent to:

```python theme={null}
pred = eval_logger.log_prediction(inputs=inputs, output=output)
pred.log_score(scorer="correctness", score=output == sample['expected'])
pred.finish()
```

<Note>
  `log_example()` is not available for the Weave TypeScript SDK. TypeScript users should use the `logPrediction()` and `logScore()` pattern shown in the [basic example](#basic-example).
</Note>

## Advanced usage

The `EvaluationLogger` offers flexible patterns beyond the basic workflow to accommodate more complex evaluation scenarios. The following sections describe advanced techniques, including how to use context managers for automatic resource management, separate model execution from logging, work with rich media data, and compare multiple model evaluations side by side.

### Use context managers

The `EvaluationLogger` supports context managers (`with` statements) for both predictions and scores. This can provide cleaner code, automatic resource cleanup, and better tracking of nested operations like LLM judge calls.

Using `with` statements in this context provides:

* Automatic `finish()` calls when exiting the context.
* Better token and cost tracking for nested LLM calls.
* Setting output after model execution within the prediction context.

<Tabs>
  <Tab title="Python">
    ```python lines {16,24,31,40} theme={null}
    import openai
    import weave

    weave.init("nested-evaluation-example")
    oai = openai.OpenAI()

    # Initialize the logger
    ev = weave.EvaluationLogger(
        model="gpt-4o-mini",
        dataset="joke_dataset"
    )

    user_prompt = "Tell me a joke"

    # Use context manager for prediction - no need to call finish()
    with ev.log_prediction(inputs={"user_prompt": user_prompt}) as pred:
        # Make your model call within the context
        result = oai.chat.completions.create(
            model="gpt-4o-mini",
            messages=[{"role": "user", "content": user_prompt}],
        )

        # Set the output after the model call
        pred.output = result.choices[0].message.content

        # Log simple scores
        pred.log_score("correctness", 1.0)
        pred.log_score("ambiguity", 0.3)
        
        # Use nested context manager for scores that require LLM calls
        with pred.log_score("llm_judge") as score:
            judge_result = oai.chat.completions.create(
                model="gpt-4o-mini",
                messages=[
                    {"role": "system", "content": "Rate how funny the joke is from 1-5"},
                    {"role": "user", "content": pred.output},
                ],
            )
            # Set the score value after computation
            score.value = judge_result.choices[0].message.content

    # finish() is automatically called when exiting the 'with' block

    ev.log_summary({"avg_score": 1.0})
    ```

    This pattern ensures that all nested operations are tracked and attributed to the parent prediction, giving you accurate token usage and cost data in the Weave UI.
  </Tab>

  <Tab title="TypeScript">
    TypeScript doesn't have Python's `with` statement pattern for context managers. Instead, use the fire-and-forget pattern with explicit `finish()` calls.

    The following example logs a prediction, adds scores and an LLM judge score, then finalizes the prediction with `finish()`:

    ```typescript lines {43} theme={null}
    import weave from 'weave';
    import OpenAI from 'openai';
    import {EvaluationLogger} from 'weave/evaluationLogger';

    await weave.init('your-team/your-project');
    const oai = new OpenAI();

    // Initialize the logger
    const ev = new EvaluationLogger({
      name: 'joke-eval',
      model: 'gpt-4o-mini',
      dataset: 'joke_dataset',
    });

    const userPrompt = 'Tell me a joke';

    // Get model output
    const result = await oai.chat.completions.create({
      model: 'gpt-4o-mini',
      messages: [{role: 'user', content: userPrompt}],
    });

    const modelOutput = result.choices[0].message.content;

    // Log prediction with output
    const pred = ev.logPrediction({user_prompt: userPrompt}, modelOutput);

    // Log simple scores
    pred.logScore('correctness', 1.0);
    pred.logScore('ambiguity', 0.3);

    // For LLM judge scores, make the call and log the result
    const judgeResult = await oai.chat.completions.create({
      model: 'gpt-4o-mini',
      messages: [
        {role: 'system', content: 'Rate how funny the joke is from 1-5'},
        {role: 'user', content: modelOutput || ''},
      ],
    });
    pred.logScore('llm_judge', judgeResult.choices[0].message.content);

    // Explicitly call finish when done scoring
    pred.finish();

    await ev.logSummary({avg_score: 1.0});
    ```

    <Note>
      While TypeScript doesn't have automatic cleanup with context managers, `logSummary()` automatically finishes any unfinished predictions before aggregating results. You can rely on this behavior if you prefer not to call `finish()` explicitly.
    </Note>
  </Tab>
</Tabs>

### Link to an existing dataset

When you pass raw datasets as `inputs` to `log_prediction`, Weave reimports the data with every evaluation run. This stores duplicate data, which can waste space if the dataset is large or if many evaluations reuse it.

To avoid this duplication, publish your dataset to Weave before running any evaluations, then pass the published dataset's rows as `inputs`. Weave resolves references to published rows using internal references instead of reimporting the data. This technique gives you the same linked experience as the standard [Evaluation framework](../core-types/evaluations), where each prediction links back to a specific dataset row in the Weave UI.

The following example publishes a dataset and links to it in the `EvaluationLogger`, before retrieving and iterating over it like any other dataset.

<Tabs>
  <Tab title="Python">
    ```python theme={null}
    import weave
    from weave import EvaluationLogger

    weave.init("your-team-name/your-project-name")

    # Publish the dataset (only needs to happen once)
    dataset = weave.Dataset(
        name="my_eval_dataset",
        rows=[
          {"question": "What is the capitol of France?", "expected": "Paris"},
          {"question": "What U.S. state is Seattle in?", "expected": "Washington"},
          {"question": "In what country is Mount Fuji located in?", "expected": "Japan"},
        ],
    )
    weave.publish(dataset)

    # Retrieve the published dataset
    dataset = weave.ref("my_eval_dataset").get()
    ```
  </Tab>

  <Tab title="TypeScript">
    ```typescript theme={null}
    import weave, {EvaluationLogger, Dataset} from 'weave';

    await weave.init('your-team-name/your-project-name');

    // Publish the dataset (only needs to happen once)
    const dataset = new Dataset({
      name: 'my_eval_dataset',
      rows: [
        {"question": "What is the capitol of France?", "expected": "Paris"},
        {"question": "What U.S. state is Seattle in?", "expected": "Washington"},
        {"question": "In what country is Mount Fuji located in?", "expected": "Japan"},
      ],
    });
    const datasetRef = await dataset.save();

    // Retrieve the published dataset
    const published = await datasetRef.get();
    ```
  </Tab>
</Tabs>

### Get outputs before logging

You can first compute your model outputs, then separately log predictions and scores. This separates evaluation and logging logic, which can make code easier to test and maintain when different parts of your system handle prediction generation and scoring.

<Tabs>
  <Tab title="Python">
    ```python lines theme={null}
    # Initialize EvaluationLogger BEFORE calling the model to ensure token tracking
    ev = EvaluationLogger(
        model="example_model",
        dataset="example_dataset"
    )

    # Model outputs (e.g. OpenAI calls) must happen after logger init for token tracking
    outputs = [your_output_generator(**inputs) for inputs in your_dataset]
    preds = [ev.log_prediction(inputs, output) for inputs, output in zip(your_dataset, outputs)]
    for pred, output in zip(preds, outputs):
        pred.log_score(scorer="greater_than_5_scorer", score=output > 5)
        pred.log_score(scorer="greater_than_7_scorer", score=output > 7)
        pred.finish()

    ev.log_summary()
    ```
  </Tab>

  <Tab title="TypeScript">
    The fire-and-forget pattern excels when processing multiple predictions in parallel.

    The following example batch-processes evaluations in parallel by creating multiple concurrent instances of the `EvaluationLogger`:

    ```typescript lines theme={null}
    // Initialize EvaluationLogger BEFORE calling the model to ensure token tracking
    const ev = new EvaluationLogger({
      name: 'parallel-eval',
      model: 'example_model',
      dataset: 'example_dataset'
    });

    // Model outputs, such as OpenAI calls, must happen after logger init for token tracking
    const outputs = await Promise.all(
      yourDataset.map(inputs => yourOutputGenerator(inputs))
    );

    // Fire-and-forget: Process all predictions without awaiting
    const preds = yourDataset.map((inputs, i) =>
      ev.logPrediction(inputs, outputs[i])
    );

    preds.forEach((pred, i) => {
      const output = outputs[i];
      // Fire-and-forget: No await needed
      pred.logScore('greater_than_5_scorer', output > 5);
      pred.logScore('greater_than_7_scorer', output > 7);
      pred.finish();
    });

    // logSummary waits for all pending operations
    await ev.logSummary();
    ```

    You can use the fire-and-forget pattern to process as many evaluations in parallel as your compute resources allow.
  </Tab>
</Tabs>

### Log rich media

Inputs, outputs, and scores can include rich media such as images, videos, audio, or structured tables. Logging rich media lets you inspect the actual content alongside scores in the Weave UI, which is helpful for qualitative analysis of multimodal models. Pass a dict or media object into the `log_prediction` or `log_score` methods.

<Tabs>
  <Tab title="Python">
    ```python lines theme={null}
    import io
    import wave
    import struct
    from PIL import Image
    import random
    from typing import Any
    import weave

    def generate_random_audio_wave_read(duration=2, sample_rate=44100):
        n_samples = duration * sample_rate
        amplitude = 32767  # 16-bit max amplitude

        buffer = io.BytesIO()

        # Write wave data to the buffer
        with wave.open(buffer, 'wb') as wf:
            wf.setnchannels(1)
            wf.setsampwidth(2)  # 16-bit
            wf.setframerate(sample_rate)

            for _ in range(n_samples):
                sample = random.randint(-amplitude, amplitude)
                wf.writeframes(struct.pack('<h', sample))

        # Rewind the buffer to the beginning so we can read from it
        buffer.seek(0)

        # Return a Wave_read object
        return wave.open(buffer, 'rb')

    rich_media_dataset = [
        {
            'image': Image.new(
                "RGB",
                (100, 100),
                color=(
                    random.randint(0, 255),
                    random.randint(0, 255),
                    random.randint(0, 255),
                ),
            ),
            "audio": generate_random_audio_wave_read(),
        }
        for _ in range(5)
    ]

    @weave.op
    def your_output_generator(image: Image.Image, audio) -> dict[str, Any]:
        return {
            "result": random.randint(0, 10),
            "image": image,
            "audio": audio,
        }

    ev = EvaluationLogger(model="example_model", dataset="example_dataset")

    for inputs in rich_media_dataset:
        output = your_output_generator(**inputs)
        pred = ev.log_prediction(inputs, output)
        pred.log_score(scorer="greater_than_5_scorer", score=output["result"] > 5)
        pred.log_score(scorer="greater_than_7_scorer", score=output["result"] > 7)

    ev.log_summary()
    ```
  </Tab>

  <Tab title="TypeScript">
    The TypeScript SDK supports logging images and audio using the `weaveImage` and `weaveAudio` functions. The following example loads image and audio files, processes them through a model, and logs the results with scores.

    ```typescript lines theme={null}
    import weave, {EvaluationLogger} from 'weave';
    import * as fs from 'fs';

    await weave.init('your-team/your-project');

    // Load images and audio from files
    const richMediaDataset = [
      {
        image: weave.weaveImage({data: fs.readFileSync('sample1.png')}),
        audio: weave.weaveAudio({data: fs.readFileSync('sample1.wav')}),
      },
      {
        image: weave.weaveImage({data: fs.readFileSync('sample2.png')}),
        audio: weave.weaveAudio({data: fs.readFileSync('sample2.wav')}),
      },
    ];

    // Model that processes media and returns results
    const yourOutputGenerator = weave.op(
      async (inputs: {image: any; audio: any}) => {
        const result = Math.floor(Math.random() * 10);
        return {
          result,
          image: inputs.image,
          audio: inputs.audio,
        };
      },
      {name: 'yourOutputGenerator'}
    );

    const ev = new EvaluationLogger({
      name: 'rich-media-eval',
      model: 'example_model',
      dataset: 'example_dataset',
    });

    for (const inputs of richMediaDataset) {
      const output = await yourOutputGenerator(inputs);

      // Log prediction with rich media in both inputs and outputs
      const pred = ev.logPrediction(inputs, output);
      pred.logScore('greater_than_5_scorer', output.result > 5);
      pred.logScore('greater_than_7_scorer', output.result > 7);
      pred.finish();
    }

    await ev.logSummary();
    ```
  </Tab>
</Tabs>

### Log and compare multiple evaluations

With `EvaluationLogger`, you can log and compare multiple evaluations side by side in the Weave UI. This is useful for assessing how different models perform on the same dataset.

1. Run the following code sample.
2. In the Weave UI, navigate to the `Evals` tab.
3. Select the evals that you want to compare.
4. Click the **Compare** button. In the Compare view, you can:
   * Choose which Evals to add or remove.
   * Choose which metrics to show or hide.
   * Page through specific examples to see how different models performed for the same input on a given dataset.

For more information about comparisons, see [Comparisons](../tools/comparison).

<Tabs>
  <Tab title="Python">
    ```python lines theme={null}
    import weave

    models = [
        "model1",
        "model2",
         {"name": "model3", "metadata": {"coolness": 9001}}
    ]

    for model in models:
        # EvalLogger must be initialized before model calls to capture tokens
        ev = EvaluationLogger(
            name="comparison-eval",
            model=model, 
            dataset="example_dataset",
            scorers=["greater_than_3_scorer", "greater_than_5_scorer", "greater_than_7_scorer"],
            eval_attributes={"experiment_id": "exp_123"}
        )
        for inputs in your_dataset:
            output = your_output_generator(**inputs)
            pred = ev.log_prediction(inputs=inputs, output=output)
            pred.log_score(scorer="greater_than_3_scorer", score=output > 3)
            pred.log_score(scorer="greater_than_5_scorer", score=output > 5)
            pred.log_score(scorer="greater_than_7_scorer", score=output > 7)
            pred.finish()

        ev.log_summary()
    ```
  </Tab>

  <Tab title="TypeScript">
    ```typescript lines theme={null}
    import weave from 'weave';
    import {EvaluationLogger} from 'weave/evaluationLogger';
    import {WeaveObject} from 'weave/weaveObject';

    await weave.init('your-team/your-project');

    const models = [
      'model1',
      'model2',
      new WeaveObject({name: 'model3', metadata: {coolness: 9001}})
    ];

    for (const model of models) {
      // EvalLogger must be initialized before model calls to capture tokens
      const ev = new EvaluationLogger({
        name: 'comparison-eval',
        model: model,
        dataset: 'example_dataset',
        description: 'Model comparison evaluation',
        scorers: ['greater_than_3_scorer', 'greater_than_5_scorer', 'greater_than_7_scorer'],
        attributes: {experiment_id: 'exp_123'}
      });

      for (const inputs of yourDataset) {
        const output = await yourOutputGenerator(inputs);

        // Fire-and-forget pattern for clean, efficient logging
        const pred = ev.logPrediction(inputs, output);
        pred.logScore('greater_than_3_scorer', output > 3);
        pred.logScore('greater_than_5_scorer', output > 5);
        pred.logScore('greater_than_7_scorer', output > 7);
        pred.finish();
      }

      await ev.logSummary();
    }
    ```
  </Tab>
</Tabs>

<Frame>
  <img alt="The Evals tab" />
</Frame>

<Frame>
  <img alt="The Comparison view" />
</Frame>

## Usage tips

The following tips help you get the most out of `EvaluationLogger`:

<Tabs>
  <Tab title="Python">
    * Call `finish()` promptly after each prediction.
    * Use `log_summary` to capture metrics not tied to single predictions (for example, overall latency).
    * Rich media logging is useful for qualitative analysis.
  </Tab>

  <Tab title="TypeScript">
    * **Auto-finish behavior**: While we recommend explicitly calling `finish()` on each prediction for clarity, `logSummary()` automatically finishes any unfinished predictions. However, once the script calls `finish()`, it can't log any more scores.
    * **Configuration options**: Use configuration options, including `name`, `description`, `dataset`, `model`, `scorers`, and `attributes` to organize and filter your evaluations in the Weave UI.
  </Tab>
</Tabs>


# Export evaluation data
Source: https://docs.wandb.ai/weave/guides/evaluation/export_eval

Programmatically export evaluation results using the Evaluation REST API.

Teams that run evaluations in W\&B Weave often need evaluation results outside of the Weave UI. Common use cases include:

* Pulling metrics into spreadsheets or notebooks for custom analysis and visualization.
* Feeding evaluation results into CI/CD pipelines to gate deployments.
* Sharing results through BI tools like Looker or internal dashboards with stakeholders who don't have W\&B seats.
* Building automated reporting pipelines that aggregate scores across projects.

The [v2 Evaluation REST API](https://trace.wandb.ai/docs) surfaces focused evaluation concepts: evaluation runs, predictions, scores, and scorers. The result is richer, more structured output with typed scorer statistics and resolved dataset inputs compared to the general-purpose Calls API.

## API endpoints used

The snippets on this page use the following endpoints from the [v2 Evaluation REST API](https://trace.wandb.ai/docs):

* `GET /v2/{entity}/{project}/evaluation_runs`: Lists evaluation runs in a project, with optional filters by evaluation reference, model reference, or run ID.
* `GET /v2/{entity}/{project}/evaluation_runs/{evaluation_run_id}`: Reads a single evaluation run to retrieve its model, evaluation reference, status, timestamps, and summary.
* `POST /v2/{entity}/{project}/eval_results/query`: Retrieves grouped evaluation result rows for one or more evaluations. Returns per-row trials with model output, scores, and optionally resolved dataset row inputs. Also returns aggregated scorer statistics when requested.
* `GET /v2/{entity}/{project}/predictions/{prediction_id}`: Reads an individual prediction with its inputs, output, and model reference.

Authentication uses HTTP Basic with `api` as the username and your W\&B API key as the password.

## Prerequisites

The examples on this page use Python, but the Evaluation REST API is language-agnostic: you can call the same endpoints from TypeScript or any HTTP client.

Before you begin, make sure you have the following:

* Python 3.7 or later.
* The `requests` library. Install it with `pip install requests`.
* A W\&B API key, set as the `WANDB_API_KEY` environment variable. Get your key at [wandb.ai/settings](https://wandb.ai/settings).

## Set up authentication

The following snippet imports the libraries used throughout this page and configures the base URL, authentication tuple, and target entity and project. Every later example reuses these variables.

```python theme={null}
import json
import os

import requests

TRACE_BASE = "https://trace.wandb.ai"
AUTH = ("api", os.environ["WANDB_API_KEY"])

entity = "my-team"
project = "my-project"
```

With authentication configured, you can call any of the endpoints described in the following sections.

## List evaluation runs

A list of evaluation runs is usually the first thing you need in an export workflow, because it gives you the `evaluation_run_id` values that the other endpoints require. Retrieve recent evaluation runs in a project and list details for each run, such as ID and status.

```python theme={null}
resp = requests.get(
    f"{TRACE_BASE}/v2/{entity}/{project}/evaluation_runs",
    auth=AUTH,
)
runs = [json.loads(line) for line in resp.text.strip().splitlines()]

for run in runs:
    print(run["evaluation_run_id"], run.get("status"))
```

## Read a single evaluation run

After you have an `evaluation_run_id`, you can fetch the full record for that run. Retrieve details for a specific evaluation run, including its model, evaluation reference, status, and timestamps. Replace `[EVALUATION_RUN_ID]` with the ID of the evaluation run you want to fetch.

```python theme={null}
eval_run_id = "[EVALUATION_RUN_ID]"

resp = requests.get(
    f"{TRACE_BASE}/v2/{entity}/{project}/evaluation_runs/{eval_run_id}",
    auth=AUTH,
)
eval_run = resp.json()
print(eval_run["evaluation_run_id"], eval_run.get("status"), eval_run.get("model"))
```

## Get predictions and scores

When you need the underlying data behind a run, such as for spreadsheet exports or row-level analysis, use the `eval_results/query` endpoint to retrieve per-row results for an evaluation run. Each row includes the resolved dataset inputs, model output, and individual scorer results. Set `include_rows`, `include_raw_data_rows`, and `resolve_row_refs` to get the full per-row detail. Replace `[EVALUATION_RUN_ID]` with the ID of the evaluation run you want to query.

```python theme={null}
eval_run_id = "[EVALUATION_RUN_ID]"

resp = requests.post(
    f"{TRACE_BASE}/v2/{entity}/{project}/eval_results/query",
    json={
        "evaluation_run_ids": [eval_run_id],
        "include_rows": True,
        "include_raw_data_rows": True,
        "resolve_row_refs": True,
    },
    auth=AUTH,
)
results = resp.json()

for row in results["rows"]:
    inputs = row.get("raw_data_row")
    for ev in row.get("evaluations", []):
        for trial in ev.get("trials", []):
            output = trial.get("model_output")
            scores = trial.get("scores", {})
            print("Input:", inputs)
            print("Output:", output)
            print("Scores:", scores)
```

## Get aggregated scores

When you only need high-level metrics, such as for dashboards or CI/CD gating, request summary statistics instead of per-row data. The same `eval_results/query` endpoint can also return aggregated scorer statistics instead of per-row data. Set `include_summary` to get summary-level metrics like pass rates for binary scorers and means for continuous scorers.

```python theme={null}
resp = requests.post(
    f"{TRACE_BASE}/v2/{entity}/{project}/eval_results/query",
    json={
        "evaluation_run_ids": [eval_run_id],
        "include_summary": True,
        "include_rows": False,
    },
    auth=AUTH,
)
results = resp.json()

for ev in results["summary"]["evaluations"]:
    for stat in ev["scorer_stats"]:
        print(stat["scorer_key"], stat.get("value_type"), stat.get("pass_rate") or stat.get("numeric_mean"))
```

## Read a single prediction

To inspect a single row in isolation, for example when investigating an unexpected score, you can fetch a prediction directly by its ID. Retrieve the full details of an individual prediction, including its inputs, output, and model reference. Replace `[PREDICTION_ID]` with the ID of the prediction you want to retrieve.

```python theme={null}
prediction_id = "[PREDICTION_ID]"

resp = requests.get(
    f"{TRACE_BASE}/v2/{entity}/{project}/predictions/{prediction_id}",
    auth=AUTH,
)
prediction = resp.json()
print(prediction)
```

## Row digests

Beyond the raw data each endpoint returns, the response from `eval_results/query` includes an additional identifier that helps you correlate rows across runs. Each result row from the `eval_results/query` endpoint includes a `row_digest`, a content hash that uniquely identifies a specific input in the evaluation dataset based on its contents, not its position. Row digests are useful for:

* **Cross-evaluation comparison**: When you run two different models against the same dataset, rows with the same digest represent the same input. You can join on `row_digest` to compare how different models performed on the exact same task.
* **Deduplication**: If the same task appears in multiple evaluation suites, the digest lets you identify it.
* **Reproducibility**: The digest is content-addressable, so if someone modifies a dataset row (changes the instruction text, rubric, or other fields), it gets a new digest. You can verify whether two evaluation runs used identical inputs or different versions.


# Set up guardrails
Source: https://docs.wandb.ai/weave/guides/evaluation/guardrails

Ensure LLM safety and measure output quality in production applications

Guardrails intervene in your LLM application's behavior based on scores from LLM judges. They run in real time before outputs reach users and can block or modify responses when scores exceed thresholds. You can use guardrails to block toxic content, filter responses for personally identifiable information (PII), or block abusive input from users.

This guide explains how Weave guardrails work, how to tune their performance, and walks through examples that use built-in scorers, custom scorers, and AWS Bedrock Guardrails to protect production LLM applications.

## How Weave guardrails work

Weave guardrails use inline [Weave Scorers](/weave/guides/evaluation/scorers) to assess the input from a user or the output from an LLM and adjust the LLM's responses in real time. You can configure custom scorers or use [built-in scorers](/weave/guides/evaluation/builtin_scorers) to assess content for a variety of purposes. This guide demonstrates how to use both types of scorers as guardrails.

If you want to passively score production traffic without modifying your application's control flow, use [monitors](/weave/guides/evaluation/monitors) instead.

Unlike monitors, guardrails require code changes because they affect your application's control flow. However, every scorer result from guardrails is automatically stored in Weave's database, so your guardrails also function as monitors without any extra configuration. You can analyze historical scorer results regardless of how they were originally used.

<Note>
  The Weave TypeScript SDK doesn't support the tools required to set up guardrails.
</Note>

### Optimize your Weave guardrail performance

Because guardrails can interrupt your application's control flow and change the course of its responses, they can impact performance if they're too complex. For best performance, follow these recommendations:

* Keep guardrail logic minimal and fast.
* Cache common results.
* Avoid heavy external API calls.
* Initialize guardrails outside of your main functions to avoid repeated initialization costs.

Initializing your guardrails outside of the main function is particularly important when:

* Your scorers load ML models.
* You're using local LLMs where latency is critical.
* Your scorers maintain network connections.
* You have high-traffic applications.

## Example: Create a guardrail using a built-in moderation scorer

The following example sends user prompts to OpenAI's GPT-4o mini model. The model's response is then passed to [OpenAI's moderation API](https://platform.openai.com/docs/guides/moderation) to assess whether the LLM's response contains harmful or toxic content. The model's response is passed to the guardrail function (`generate_safe_response()`), which uses the `OpenAIModerationScorer` to check the LLM's original response. The function's logic then checks OpenAI's assessment response for a boolean in the `passed` field, which determines how the application responds.

```python lines {28-45} theme={null}
import weave
import openai
from weave.scorers import OpenAIModerationScorer
import asyncio

# Initialize Weave
weave.init("your-team-name/your-project-name")

# Initialize OpenAI client
client = openai.OpenAI()  # Uses OPENAI_API_KEY env var

# Initialize the moderation scorer
moderation_scorer = OpenAIModerationScorer()

# Send prompts to OpenAI
@weave.op
def generate_response(prompt: str) -> str:
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {"role": "system", "content": "You are a helpful assistant."},
            {"role": "user", "content": prompt}
        ],
        max_tokens=200
    )
    return response.choices[0].message.content

# Guardrail function checks responses for toxicity
async def generate_safe_response(prompt: str) -> str:
    """Generate a response with content moderation guardrail."""
    # Get both the result and Call object
    result, call = generate_response.call(prompt)
    
    # Apply the moderation scorer before returning to user
    score = await call.apply_scorer(moderation_scorer)
    print("This is the score object:", score)
    
    # Check if content was flagged
    if not score.result.get("passed", True): 
        categories = score.result.get("categories", {})
        flagged_categories = list(categories.keys()) if categories else []
        print(f"Content blocked. Flagged categories: {flagged_categories}")
        return "I'm sorry, I can't provide that response due to content policy restrictions."
    
    return result

# Run the examples
if __name__ == "__main__":
    
    prompts = [
        "What's the capital of France?",
        "Tell me a funny fact about dogs.",
    ]
    
    for prompt in prompts:
        print(f"\nPrompt: {prompt}")
        response = asyncio.run(generate_safe_response(prompt))
        print(f"Response: {response}")
```

When using LLM-as-a-judge scorers, you can reference variables from your ops in your scoring prompts. For example, "Evaluate whether `{output}` is accurate based on `{ground_truth}`." See [prompt variables](/weave/guides/evaluation/scorers#access-variables-from-your-ops-in-scoring-prompts) for more information.

## Example: Create a guardrail using a custom scorer

The following example creates a custom guardrail that detects personally identifiable information (PII) in LLM responses, such as email addresses, phone numbers, or social security numbers. This prevents sensitive information from being exposed in generated content. The `generate_safe_response` function applies the custom `PIIDetectionScorer`.

```python lines {14-39, 57-69} theme={null}
import weave
import openai
import re
import asyncio
from weave import Scorer

weave.init("your-team-name/your-project-name")

client = openai.OpenAI()

class PIIDetectionScorer(Scorer):
    """Detects PII in LLM outputs to prevent data leaks."""
    
    @weave.op
    def score(self, output: str) -> dict:
        """
        Check for common PII patterns in the output.
        
        Returns:
            dict: Contains 'passed' (bool) and 'detected_types' (list)
        """
        detected_types = []
        
        # Email pattern
        if re.search(r'\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Z|a-z]{2,}\b', output):
            detected_types.append("email")
        
        # Phone number pattern (US format)
        if re.search(r'\b\d{3}[-.]?\d{3}[-.]?\d{4}\b', output):
            detected_types.append("phone")
        
        # SSN pattern
        if re.search(r'\b\d{3}-\d{2}-\d{4}\b', output):
            detected_types.append("ssn")
        
        return {
            "passed": len(detected_types) == 0,
            "detected_types": detected_types
        }

# Initialize scorer outside the function for best performance
pii_scorer = PIIDetectionScorer()

@weave.op
def generate_response(prompt: str) -> str:
    """Generate a response using an LLM."""
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {"role": "system", "content": "You are a helpful assistant."},
            {"role": "user", "content": prompt}
        ],
        max_tokens=200
    )
    return response.choices[0].message.content

async def generate_safe_response(prompt: str) -> str:
    """Generate a response with PII detection guardrail."""
    result, call = generate_response.call(prompt)
    
    # Apply PII detection scorer
    score = await call.apply_scorer(pii_scorer)
    
    # Block response if PII detected
    if not score.result.get("passed", True):
        detected_types = score.result.get("detected_types", [])
        return f"I cannot provide a response that may contain sensitive information (detected: {', '.join(detected_types)})."
    
    return result

# Example usage
if __name__ == "__main__":
    prompts = [
        "What's the weather like today?",
        "Can you help me contact someone at john.doe@example.com?",
        "Tell me about machine learning.",
    ]
    
    for prompt in prompts:
        print(f"\nPrompt: {prompt}")
        response = asyncio.run(generate_safe_response(prompt))
        print(f"Response: {response}")
```

## Integrate Weave with AWS Bedrock Guardrails

If you already manage content policies in AWS, you can apply them in Weave by using the `BedrockGuardrailScorer`, which uses AWS Bedrock Guardrails to detect and filter content based on configured policies.

Before you set up a Bedrock Guardrails integration, you need:

* An AWS account with Bedrock access.
* A [configured guardrail in the AWS Bedrock console](https://docs.aws.amazon.com/bedrock/latest/userguide/guardrails-components.html).
* The `boto3` [Python package](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3.html).

You don't need to create your own Bedrock client. Weave creates it for you. To specify a region, pass the region value in the scorer's `bedrock_runtime_kwargs` parameter.

For an example of how to create a guardrail in AWS Bedrock, see the [Bedrock guardrails notebook](https://github.com/aws-samples/amazon-bedrock-samples/blob/main/responsible_ai/bedrock-guardrails/guardrails-api.ipynb).

The following example checks text generation against AWS Bedrock Guardrails policies before returning results to users:

```python theme={null}
import weave
from weave.scorers.bedrock_guardrails import BedrockGuardrailScorer

weave.init("your-team-name/your-project-name")

guardrail_scorer = BedrockGuardrailScorer(
    guardrail_id="your-guardrail-id",
    guardrail_version="DRAFT",
    source="INPUT",
    bedrock_runtime_kwargs={"region_name": "us-east-1"}
)

@weave.op
def generate_text(prompt: str) -> str:
    # Your text generation logic here
    return "Generated text..."

async def generate_safe_text(prompt: str) -> str:
    result, call = generate_text.call(prompt)

    score = await call.apply_scorer(guardrail_scorer)

    if not score.result.passed:
        if score.result.metadata.get("modified_output"):
            return score.result.metadata["modified_output"]
        return "I cannot generate that content due to content policy restrictions."

    return result
```


# Monitor using built-in signals
Source: https://docs.wandb.ai/weave/guides/evaluation/monitors

W&B provides built-in scoring solutions for agents in production

<Note>
  When tracing agentic applications, use the **Signals** tab on the **Agents** page. See [Signals for agents](/weave/guides/tracking/view-agent-signals).
</Note>

Standard system metrics like latency, token count, and cost don't fully capture agent behavior. Inspecting individual traces provides deep insight but doesn't scale to the millions of traces generated in a live environment.

Signals provide a high-level monitoring solution to this problem by offering automated, behavioral scoring for agents in production:

* Automated scoring: Every incoming production trace is automatically processed and scored on common quality issues and errors.
* Infrastructure: Processing is powered by CoreWeave compute and CoreWeave GPUs for scalability across millions of traces.

By using signals within production, you can:

* Gain behavioral insight. Move beyond system metrics to understand if your agent is hallucinating, failing to follow conversation patterns, or losing grounding in its evidence.
* Accelerate the research loop. Use the scores and failure analyses generated by signals to identify weaknesses, which you can use to research model improvement, data annotation, or reinforcement learning.

## Available signals

W\&B Weave offers monitors with built-in signals: preset scorers that evaluate production traces for common quality issues and errors by default, with no custom setup. Each built-in signal uses a benchmarked LLM prompt to classify traces and saves the results as comma-delimited tags representing the detected issues.

Signals use a [Serverless Inference](/inference/) model to score traces, so you don't need external API keys.

W\&B Weave provides 13 preset signals organized into two groups.

### Quality signals

Quality signals evaluate successful root-level traces for output quality and safety issues.

| Signal               | What it detects                                                                                           |
| -------------------- | --------------------------------------------------------------------------------------------------------- |
| **Hallucination**    | Fabricated facts or claims that contradict the provided input context                                     |
| **Low quality**      | Responses with poor format, insufficient effort, or incomplete content                                    |
| **User frustration** | Signs of user frustration such as repeated questions, negative sentiment, or complaints                   |
| **Jailbreaking**     | Prompt injection and jailbreak attempts that try to bypass safety guidelines                              |
| **NSFW**             | Explicit, violent, or otherwise inappropriate content in inputs or outputs                                |
| **Lazy**             | Low-effort responses such as excessive brevity, refusals to help, or deferred work                        |
| **Forgetful**        | Failure to use context from earlier in the conversation, ignoring previously stated facts or instructions |

### Error signals

Error signals categorize failed traces by root cause to help you identify and resolve infrastructure and application issues.

| Signal                | What it detects                                                               |
| --------------------- | ----------------------------------------------------------------------------- |
| **Network Error**     | DNS failures, timeouts, connection resets, and other connectivity issues      |
| **Ratelimited**       | HTTP `429` responses, quota exhaustion, and throttling from upstream APIs     |
| **Request Too Large** | Requests exceeding size or token limits, such as context window exceeded      |
| **Bad Request**       | Client-side errors where the server rejected the request (`4xx` except `429`) |
| **Bad Response**      | Invalid, unexpected, or unusable responses from remote services (`5xx`)       |
| **Bug**               | Flaws in application code such as `KeyError`, `TypeError`, or logic errors    |

### How signals work

Each signal uses an LLM-as-a-judge approach to classify traces:

* **Trace selection**: Quality signals evaluate successful root-level traces. Error signals evaluate failed traces. Weave doesn't score child spans and intermediate Calls.
* **Prompt construction**: Weave constructs a prompt that includes the trace metadata, inputs, outputs, exception details (if any), and the operation's source code. Weave appends the signal's classifier prompt with instructions for the specific issue to detect.
* **LLM scoring**: For each signal, a Serverless Inference model performs a binary classification (whether that issue is present on the trace). Detected issues are returned as comma-delimited string tags (for example, `"Low-quality, User-frustration, Forgetful"`).

When multiple signals from the same group (Quality or Error) are active, Weave batches the signals into a single LLM call. The model evaluates all active classifiers in one pass and returns a result for each.

## Add a signal from the Monitors page

Adding a signal turns on automated scoring so that Weave evaluates new production traces against that signal's criteria. To enable signals:

1. Navigate to [wandb.ai](https://wandb.ai/) and then open your Weave project.
2. In the Weave project sidebar, select **Monitors**.
3. At the top of the Monitors page, a row of suggested signal cards appears. Each card shows the signal name, a description, and an **+ Add signal** button.
4. To add a signal, select the **Add signal** button on the signal card. The signal starts scoring new traces immediately.
5. To add multiple signals at once, select the **\[X] more signals** button. This opens a drawer that lists all available signals grouped by category.
6. Select the signals you want to turn on, then select **Add signals**.

After you add signals, Weave automatically scores incoming traces.

## Manage active signals

After your signals are running, you can review which ones are active or turn off signals you no longer need. To view or remove active signals:

1. From the Monitors page, select the **Manage signals** (<Icon icon="gear" />) button. This opens a drawer listing all currently active signals grouped by category.
2. Hover over a signal and select the **Remove signal** (<Icon icon="trash-can" />) button to deactivate the signal.

Removing a signal stops scoring new traces. Weave preserves existing scores from the signal.

## Use built-in signals

After signals are active and scoring your traces, you can review the results in several places across Weave. You can also set up alerts when Weave detects issues. The following sections describe where to find signal results and how to act on them.

### See tagged Call traces on the Traces page

If you trace individual functions as Ops with the `@weave.op` decorator, Weave stores signal results as feedback on the Call object. You can query these results from the **Traces** page.

You can scan your traces for certain behavior in the **Traces** page using the **Signals** column. The Signals column shows tags when their criteria are met. Hover over these tags to see the confidence in the score and the reasoning.

<img alt="Weave Traces view with hover over of a Signals tag in the Signals column showing confidence and reasoning." />

Use the trace table toolbar to filter the trace table to only show traces that triggered certain signals.

You can view additional signal details in the Traces page by selecting the classifier Call that the signal generates and reviewing the **Trace Details view**. Under Call **Output**, review `classifier_meta` for the reasoning. For example, the following screenshot shows a `Quality-classifiers` signal with `Low-quality` match and confidence (0.9) with a reason for this rating.

<img alt="Weave Traces view with a quality-classifier trace selected. The details panel shows Call details with classifiers metadata including a confidence score and reason." />

### See signals in the project dashboard

You can also review signals at a project level:

1. In the project sidebar, select **Project**.
2. At the top of the Project dashboard, select the **Weave** tab.
3. In the Weave dashboard panels, locate **Monitor Scores**.

In the Monitor Scores project panel, you can see time-based graphs of signals that occurred for the project.

<img alt="Weave project dashboard Monitor Scores panel showing signal graphs from project activity." />

### Alert on signals

Beyond reviewing signals in the UI, Weave can notify you when signals trigger. You can set up automated triggers that notify your team through tools such as Slack when an agent's performance drops below a certain threshold.

To get notified when a signal is triggered, set up an [automation](/weave/guides/evaluation/automations).

***

For specific monitoring beyond what the built-in signals provide, see [Set up custom monitors](/weave/guides/evaluation/custom-monitors).


# Scoring overview
Source: https://docs.wandb.ai/weave/guides/evaluation/scorers

Evaluate AI outputs and return evaluation metrics with Weave Scorers

In Weave, Scorers evaluate AI outputs and return evaluation metrics. They take the AI's output, analyze it, and return a dictionary of results. Scorers can use your input data as reference if needed and can also output extra information, such as explanations or reasoning from the evaluation.

This guide is for developers who want to measure the quality of their AI system's outputs. It explains how Scorers fit into Weave evaluations, how to build your own Scorers, how to apply Scorers to individual calls, and how to analyze the resulting scores.

<Tabs>
  <Tab title="Python">
    Pass Scorers to a `weave.Evaluation` object during evaluation. Weave supports two types of Scorers:

    1. **Function-based Scorers:** Python functions decorated with `@weave.op`.
    2. **Class-based Scorers:** Python classes that inherit from `weave.Scorer` for more complex evaluations.

    Scorers must return a dictionary and can return multiple metrics, nested metrics, and non-numeric values such as text returned from an LLM-evaluator about its reasoning.
  </Tab>

  <Tab title="TypeScript">
    Scorers are special ops that you pass to a `weave.Evaluation` object during evaluation.
  </Tab>
</Tabs>

## Create your own scorers

Custom Scorers let you encode evaluation criteria that are specific to your use case, beyond what the built-in Scorers cover. The following sections describe the two ways to define a Scorer: as a function, or as a class for more complex logic.

<Tip>
  **Ready-to-use scorers**
  Although this guide shows you how to create custom scorers, Weave comes with [predefined scorers](./builtin_scorers) and [local SLM scorers](./weave_local_scorers) that you can use right away, including:

  * [Hallucination detection](./builtin_scorers#hallucinationfreescorer)
  * [Summarization quality](./builtin_scorers#summarizationscorer)
  * [Embedding similarity](./builtin_scorers#embeddingsimilarityscorer)
  * [Toxicity detection (local)](./weave_local_scorers#weavetoxicityscorerv1)
  * [Context relevance scoring (local)](./weave_local_scorers#weavecontextrelevancescorerv1)
</Tip>

### Function-based scorers

<Tabs>
  <Tab title="Python">
    Function-based Scorers are functions decorated with `@weave.op` that return a dictionary. They work well for straightforward evaluations like:

    ```python lines theme={null}
    import weave

    @weave.op
    def evaluate_uppercase(text: str) -> dict:
        return {"text_is_uppercase": text.isupper()}

    my_eval = weave.Evaluation(
        dataset=[{"text": "HELLO WORLD"}],
        scorers=[evaluate_uppercase]
    )
    ```

    When you run the evaluation, `evaluate_uppercase` checks whether the text is all uppercase.
  </Tab>

  <Tab title="TypeScript">
    These are functions wrapped with `weave.op` that accept an object with `modelOutput` and optionally `datasetRow`. They work well for straightforward evaluations like the following example:

    ```typescript lines theme={null}
    import * as weave from 'weave'

    const evaluateUppercase = weave.op(
        ({modelOutput}) => modelOutput.toUpperCase() === modelOutput,
        {name: 'textIsUppercase'}
    );

    const myEval = new weave.Evaluation({
        dataset: [{text: 'HELLO WORLD'}],
        scorers: [evaluateUppercase],
    })
    ```
  </Tab>
</Tabs>

### Class-based scorers

<Tabs>
  <Tab title="Python">
    For more advanced evaluations, especially when you need to track additional scorer metadata, try different prompts for your LLM-evaluators, or make multiple function calls, use the `Scorer` class.

    **Requirements:**

    1. Inherit from `weave.Scorer`.
    2. Define a `score` method decorated with `@weave.op`.
    3. The `score` method must return a dictionary.

    Example:

    ```python lines {7} theme={null}
    import weave
    from openai import OpenAI
    from weave import Scorer

    llm_client = OpenAI()

    class SummarizationScorer(Scorer):
        model_id: str = "gpt-4o"
        system_prompt: str = "Evaluate whether the summary is good."

        @weave.op
        def some_complicated_preprocessing(self, text: str) -> str:
            processed_text = "Original text: \n" + text + "\n"
            return processed_text

        @weave.op
        def call_llm(self, summary: str, processed_text: str) -> dict:
            res = llm_client.chat.completions.create(
                messages=[
                    {"role": "system", "content": self.system_prompt},
                    {"role": "user", "content": (
                        f"Analyze how good the summary is compared to the original text."
                        f"Summary: {summary}\n{processed_text}"
                    )}])
            return {"summary_quality": res}

        @weave.op
        def score(self, output: str, text: str) -> dict:
            """Score the summary quality.

            Args:
                output: The summary generated by an AI system
                text: The original text to summarize
            """
            processed_text = self.some_complicated_preprocessing(text)
            eval_result = self.call_llm(summary=output, processed_text=processed_text)
            return {"summary_quality": eval_result}

    evaluation = weave.Evaluation(
        dataset=[{"text": "The quick brown fox jumps over the lazy dog."}],
        scorers=[summarization_scorer])
    ```

    This class evaluates the quality of a summary by comparing it to the original text.
  </Tab>

  <Tab title="TypeScript">
    ```plaintext theme={null}
    This feature is not available in TypeScript yet.
    ```
  </Tab>
</Tabs>

## How scorers work

This section explains how Scorers receive data from your evaluation, how to map dataset columns to Scorer arguments, how to reference op variables in scoring prompts, and how Weave summarizes per-row scores into a final result.

### Scorer keyword arguments

<Tabs>
  <Tab title="Python">
    Scorers can access both the output from your AI system and the input data from the dataset row.

    * **Input:** If you want your scorer to use data from your dataset row, such as a `label` or `target` column, make this available to the scorer by adding a `label` or `target` keyword argument to your scorer definition.

    For example, if you want to use a column called `label` from your dataset, your scorer function (or `score` class method) has a parameter list like this:

    ```python lines theme={null}
    @weave.op
    def my_custom_scorer(output: str, label: int) -> dict:
        ...
    ```

    When a Weave `Evaluation` runs, it passes the output of the AI system to the `output` parameter. The `Evaluation` also automatically tries to match any additional scorer argument names to your dataset columns. If customizing your scorer arguments or dataset columns isn't feasible, you can use column mapping. See the following section.

    * **Output:** Include an `output` parameter in your scorer function's signature to access the AI system's output.

    ### Mapping column names with `column_map`

    Sometimes the `score` method's argument names don't match the column names in your dataset. You can fix this using a `column_map`.

    If you're using a class-based scorer, pass a dictionary to the `column_map` attribute of `Scorer` when you initialize your scorer class. This dictionary maps your `score` method's argument names to the dataset's column names, in the order `{scorer_keyword_argument: dataset_column_name}`.

    Example:

    ```python lines theme={null}
    import weave
    from weave import Scorer

    # A dataset of news articles to summarize
    dataset = [
        {"news_article": "The news today was great...", "date": "2030-04-20", "source": "Bright Sky Network"},
        ...
    ]

    # Scorer class
    class SummarizationScorer(Scorer):

        @weave.op
        def score(self, output, text) -> dict:
            """
                output: output summary from an LLM summarization system
                text: the text to summarize
            """
            ...  # evaluate the quality of the summary

    # create a scorer with a column mapping the `text` argument to the `news_article` data column
    scorer = SummarizationScorer(column_map={"text" : "news_article"})
    ```

    Now, the `text` argument in the `score` method receives data from the `news_article` dataset column.

    **Notes:**

    * Another equivalent option to map your columns is to subclass the `Scorer` and overload the `score` method, mapping the columns explicitly.

    ```python lines theme={null}
    import weave
    from weave import Scorer

    class MySummarizationScorer(SummarizationScorer):

        @weave.op
        def score(self, output: str, news_article: str) -> dict:  # Added type hints
            # overload the score method and map columns manually
            return super().score(output=output, text=news_article)
    ```
  </Tab>

  <Tab title="TypeScript">
    Scorers can access both the output from your AI system and the contents of the dataset row.

    You can access relevant columns from the dataset row by adding a `datasetRow` keyword argument to your scorer definition.

    ```typescript lines theme={null}
    const myScorer = weave.op(
        ({modelOutput, datasetRow}) => {
            return modelOutput * 2 === datasetRow.expectedOutputTimesTwo;
        },
        {name: 'myScorer'}
    );
    ```

    ### Mapping column names with `columnMapping`

    <Warning>
      In TypeScript, this feature is on the `Evaluation` object, not individual scorers.
    </Warning>

    Sometimes your `datasetRow` keys don't exactly match the scorer's naming scheme, but they're semantically similar. You can map the columns using the `Evaluation`'s `columnMapping` option.

    The mapping is always from the scorer's perspective, that is, `{scorer_key: dataset_column_name}`.

    Example:

    ```typescript lines theme={null}
    const myScorer = weave.op(
        ({modelOutput, datasetRow}) => {
            return modelOutput * 2 === datasetRow.expectedOutputTimesTwo;
        },
        {name: 'myScorer'}
    );

    const myEval = new weave.Evaluation({
        dataset: [{expected: 2}],
        scorers: [myScorer],
        columnMapping: {expectedOutputTimesTwo: 'expected'}
    });
    ```
  </Tab>
</Tabs>

### Access variables from your ops in scoring prompts

In scoring prompts for LLM-as-a-judge scorers, you can reference variables from your op. Weave automatically extracts these values when the scorer runs.

For a function like:

```python theme={null}
@weave.op
def summarize_article(article: str, max_length: int) -> str:
    # Your summarization logic here
    return summary
```

The following variables are available:

| Variable       | Description                                  |
| -------------- | -------------------------------------------- |
| `{article}`    | The value of the input argument `article`    |
| `{max_length}` | The value of the input argument `max_length` |
| `{inputs}`     | A JSON dictionary of all input arguments     |
| `{output}`     | The result returned by your op               |

Example scoring prompt:

```text theme={null}
Evaluate the quality of this summary.

Original article: {article}
Summary: {output}
Maximum length requested: {max_length}

Rate the summary on a scale of 1-10 based on:
- Accuracy: Does it accurately represent the article?
- Completeness: Does it cover the key points?
- Conciseness: Is it appropriately brief?

Return a JSON object with your rating and reasoning.
```

### Final summarization of the scorer

<Tabs>
  <Tab title="Python">
    During evaluation, Weave computes the scorer for each row of your dataset. To provide a final score for the evaluation, Weave runs `auto_summarize` based on the return type of the output.

    You can override the `summarize` method on the `Scorer` class and provide your own way to compute the final scores. The `summarize` function expects:

    * A single parameter `score_rows`: a list of dictionaries, where each dictionary contains the scores returned by the `score` method for a single row of your dataset.
    * It returns a dictionary containing the summarized scores.

    **Why this is useful**

    When you need to score all rows before deciding on the final value of the score for the dataset.

    ```python lines theme={null}
    class MyBinaryScorer(Scorer):
        """
        Returns True if the full output matches the target, False if not
        """

        @weave.op
        def score(self, output, target):
            return {"match": output == target}

        def summarize(self, score_rows: list) -> dict:
            full_match = all(row["match"] for row in score_rows)
            return {"full_match": full_match}
    ```

    > In this example, the default `auto_summarize` would return the count and proportion of True.

    For more information, see the implementation of [CorrectnessLLMJudge](/weave/tutorial-rag#optional-defining-a-scorer-class).
  </Tab>

  <Tab title="TypeScript">
    During evaluation, Weave computes the scorer for each row of your dataset. To provide a final score, Weave uses an internal `summarizeResults` function that aggregates depending on the output type.

    Weave doesn't support custom summarization.
  </Tab>
</Tabs>

### Apply scorers to a call

In addition to running Scorers as part of a `weave.Evaluation`, you can apply them directly to an individual call. This is useful when you want to score production traffic or attach evaluation metrics to a specific op invocation.

To apply scorers to your Weave ops, use the `.call()` method, which provides access to both the operation's result and its tracking information. This lets you associate scorer results with specific calls in Weave's database.

For more information about using the `.call()` method, see the [Calling Ops](../tracking/tracing#getting-a-handle-to-the-call-object-during-execution) guide.

<Tabs>
  <Tab title="Python">
    Here's a basic example:

    ```python lines theme={null}
    # Get both result and Call object
    result, call = generate_text.call("Say hello")

    # Apply a scorer
    score = await call.apply_scorer(MyScorer())
    ```

    You can also apply multiple scorers to the same call:

    ```python lines theme={null}
    # Apply multiple scorers in parallel
    await asyncio.gather(
        call.apply_scorer(quality_scorer),
        call.apply_scorer(toxicity_scorer)
    )
    ```

    **Notes:**

    * Weave automatically stores scorer results in its database.
    * Scorers run asynchronously after the main operation completes.
    * You can view scorer results in the UI or query them through the API.

    For more detailed information about using scorers as guardrails or monitors, including production best practices and complete examples, see the [Guardrails and Monitors guide](./monitors).
  </Tab>

  <Tab title="TypeScript">
    ```plaintext theme={null}
    This feature is not available in TypeScript yet.
    ```
  </Tab>
</Tabs>

### Use `preprocess_model_input`

You can use the `preprocess_model_input` parameter to modify dataset examples before they reach your model during evaluation.

<Important>
  The `preprocess_model_input` function only transforms inputs before Weave passes them to the model's prediction function.

  Scorer functions always receive the original dataset examples, without any preprocessing applied.
</Important>

For usage information and an example, see [Using `preprocess_model_input` to format dataset rows before evaluating](../core-types/evaluations#using-preprocess_model_input-to-format-dataset-rows-before-evaluating).

## Score analysis

After your Scorers run, you'll often want to inspect the scores they produced to understand model behavior or compare versions. The following sections describe how to analyze the scores for a single call, multiple calls, and all calls scored by a specific Scorer, using both the API and the Weave UI.

### Analyze a single call's scores

#### Single call API

To retrieve a single call, use the `get_call` method.

```python lines theme={null}
client = weave.init("my-project")

# Get a single call
call = client.get_call("call-uuid-here")

# Get the feedback for the call which contains the scores
feedback = list(call.feedback)
```

#### Single call UI

<Frame>
  <img alt="Call Scores Tab" />
</Frame>

The Call details panel displays scores for an individual call under the **Scores** tab.

### Analyze multiple calls' scores

#### Multiple calls API

To retrieve multiple calls, use the `get_calls` method.

```python lines theme={null}
client = weave.init("my-project")

# Get multiple calls - use whatever filters you want and include feedback
calls = client.get_calls(..., include_feedback=True)

# Iterate over the calls and access the feedback which contains the scores
for call in calls:
    feedback = list(call.feedback)
```

#### Multiple calls UI

<Frame>
  <img alt="Multiple Calls Tab" />
</Frame>

The traces table displays scores for multiple calls under the **Scores** column.

### Analyze all calls scored by a specific scorer

#### All calls by scorer API

To retrieve all calls scored by a specific scorer, use the `get_calls` method.

```python lines theme={null}
client = weave.init("my-project")

# To get all the calls scored by any version of a scorer, use the scorer name (typically the class name)
calls = client.get_calls(scored_by=["MyScorer"], include_feedback=True)

# To get all the calls scored by a specific version of a scorer, use the entire ref
# Refs can be obtained from the scorer object or via the UI.
calls = client.get_calls(scored_by=[myScorer.ref.uri()], include_feedback=True)

# Iterate over the calls and access the feedback which contains the scores
for call in calls:
    feedback = list(call.feedback)
```

#### All calls by scorer UI

Finally, if you want to see all the calls scored by a Scorer, navigate to the **Scorers** tab in the UI and select the **Programmatic Scorer** tab. Click your Scorer to open the Scorer details page.

<Frame>
  <img alt="Scorer Details Page" />
</Frame>

Next, click the **View Traces** button under **Scores** to view all the calls scored by your Scorer.

<Frame>
  <img alt="Filtered Calls to Scorer Version" />
</Frame>

This defaults to the selected version of the Scorer. You can remove the version filter to see all the calls scored by any version of the Scorer.

<Frame>
  <img alt="Filtered Calls to Scorer Name" />
</Frame>


# Evaluate using local scorers
Source: https://docs.wandb.ai/weave/guides/evaluation/weave_local_scorers

Small language models that run locally to evaluate AI system safety and quality

<a href="https://colab.research.google.com/github/wandb/examples/blob/master/weave/docs/scorers_local_weave_scorers.ipynb" aria-label="Open in Google Colab">
  <img alt="Open In Colab" />
</a>

Weave's local scorers are a suite of small language models that run locally on your machine with minimal latency. These models evaluate the safety and quality of your AI system's inputs, context, and outputs.

This page describes how to evaluate the safety and quality of your AI system using Weave's local scorers, and walks through each available scorer with usage notes and examples. Use this guide to choose a scorer for your use case, install the dependencies, and add scorers to your AI system's inputs, context, or outputs.

W\&B fine-tunes some of these models, while others are state-of-the-art open source models trained by the community. W\&B Reports document the training and evaluation process. For the full details, see this [list of W\&B Reports](https://wandb.ai/c-metrics/weave-scorers/reports/Weave-Scorers-v1--VmlldzoxMDQ0MDE1OA).

The model weights are publicly available in W\&B Artifacts, and Weave automatically downloads them when you instantiate the scorer class. To download them yourself, see the artifact paths in `weave.scorers.default_models`.

These scorers return an object with a `passed` boolean attribute that indicates whether the input text is safe or high quality. The object also includes a `metadata` attribute with more detail, such as the raw score from the model.

You can run local scorers on CPUs, but W\&B recommends GPUs for best performance.

<Note>
  Local scorers are only available for the Weave Python SDK. They aren't yet available for the Weave TypeScript SDK.

  To use Weave scorers in TypeScript, see [function-based scorers](/weave/guides/evaluation/scorers#function-based-scorers).
</Note>

## Prerequisites

Before you can use Weave local scorers, install the additional dependencies that bundle the required scorer libraries:

```bash theme={null}
pip install weave[scorers]
```

With the dependencies installed, you're ready to choose a scorer for your evaluation.

## Select a scorer

The following local scorers are available. Select a scorer based on your use case, then continue to the matching section below for usage notes and an example.

| Scorer                                                          | Scenario                                                                                                                             |
| --------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------ |
| [WeaveToxicityScorerV1](#weavetoxicityscorerv1)                 | Identify toxic or harmful content in your AI system's inputs and outputs, including hate speech or threats.                          |
| [WeaveBiasScorerV1](#weavebiasscorerv1)                         | Detect biased or stereotypical content in your AI system's inputs and outputs. Useful for reducing harmful biases in generated text. |
| [WeaveHallucinationScorerV1](#weavehallucinationscorerv1)       | Identify whether your RAG system generates hallucinations in its output based on the input and context provided.                     |
| [WeaveContextRelevanceScorerV1](#weavecontextrelevancescorerv1) | Measure whether the AI system's output is relevant to the input and context provided.                                                |
| [WeaveCoherenceScorerV1](#weavecoherencescorerv1)               | Evaluate the coherence and logical structure of the AI system's output.                                                              |
| [WeaveFluencyScorerV1](#weavefluencyscorerv1)                   | Measure whether the AI system's output is fluent.                                                                                    |
| [WeaveTrustScorerV1](#weavetrustscorerv1)                       | Combine the toxicity, hallucination, context relevance, fluency, and coherence scorers into a single trust signal.                   |
| [PresidioScorer](#presidioscorer)                               | Detect personally identifiable information (PII) in your AI system's inputs and outputs using the Presidio library from Microsoft.   |

## `WeaveBiasScorerV1`

This scorer assesses gender and race or origin bias along two dimensions:

* Race and Origin: Racism and bias against a country or region of origin, immigration status, or ethnicity.
* Gender and Sexuality: Sexism, misogyny, homophobia, transphobia, or sexual harassment.

`WeaveBiasScorerV1` uses a fine-tuned [deberta-small-long-nli](https://huggingface.co/tasksource/deberta-small-long-nli) model. For more details on the model, dataset, and calibration process, see the [WeaveBiasScorerV1 W\&B Report](https://wandb.ai/c-metrics/bias-benchmark/reports/Bias-Scorer--VmlldzoxMDM2MTgzNw).

### Usage notes

* The `score` method expects a string to be passed to the `output` parameter.
* A higher score means that there is a stronger prediction of bias in the text.
* The `threshold` parameter is set but can also be overridden on initialization.

### Usage example

```python lines theme={null}
import weave
from weave.scorers import WeaveBiasScorerV1

bias_scorer = WeaveBiasScorerV1()
result = bias_scorer.score(output="Martian men are terrible at cleaning")

print(f"The text is biased: {not result.passed}")
print(result)
```

***

## `WeaveToxicityScorerV1`

This scorer assesses the input text for toxicity along five dimensions:

* Race and Origin: Racism and bias against a country or region of origin, immigration status, or ethnicity.
* Gender and Sexuality: Sexism, misogyny, homophobia, transphobia, or sexual harassment.
* Religious: Bias or stereotypes against someone's religion.
* Ability: Bias related to someone's physical, mental, or intellectual ability or disability.
* Violence and Abuse: Overly graphic descriptions of violence, threats of violence, or incitement of violence.

The `WeaveToxicityScorerV1` uses the open source [Celadon](https://huggingface.co/PleIAs/celadon) model from PleIAs. For more information, see the [WeaveToxicityScorerV1 W\&B Report](https://wandb.ai/c-metrics/toxicity-benchmark/reports/Toxicity-Scorer--VmlldzoxMDMyNjc0NQ).

### Usage notes

* The `score` method expects a string to be passed to the `output` parameter.
* The model returns scores from `0` to `3` across five different categories:
  * If the sum of these scores is above `total_threshold` (default value `5`), the input is flagged as toxic.
  * If any single category has a score higher than `category_threshold` (default `2`), the input is flagged as toxic.
* To make filtering more aggressive, override `category_threshold` or `total_threshold` during initialization.

### Usage example

```python lines theme={null}
import weave
from weave.scorers import WeaveToxicityScorerV1

toxicity_scorer = WeaveToxicityScorerV1()
result = toxicity_scorer.score(output="people from the south pole of Mars are the worst")

print(f"Input is toxic: {not result.passed}")
print(result)
```

***

## `WeaveHallucinationScorerV1`

This scorer checks whether your AI system's output contains hallucinations based on the input data.

The `WeaveHallucinationScorerV1` uses the open source [HHEM 2.1 model](https://huggingface.co/vectara/hallucination_evaluation_model) from Vectara. For more information, see the [WeaveHallucinationScorerV1 W\&B Report](https://wandb.ai/c-metrics/hallucination/reports/Hallucination-Scorer--VmlldzoxMDM3NDA3MA).

### Usage notes

* The `score` method expects values to be passed to the `query` and `output` parameters.
* The context should be passed to the `output` parameter (as a string or list of strings).
* A higher output score means a stronger prediction of hallucination in the output.
* The `threshold` parameter is set but can be overridden on initialization.

### Usage example

```python lines theme={null}
import weave
from weave.scorers import WeaveHallucinationScorerV1

hallucination_scorer = WeaveHallucinationScorerV1()

result = hallucination_scorer.score(
    query="What is the capital of Antarctica?",
    context="People in Antarctica love the penguins.",
    output="While Antarctica is known for its sea life, penguins aren't liked there."
)

print(f"Output is hallucinated: {not result.passed}")
print(result)
```

***

## `WeaveContextRelevanceScorerV1`

Use this scorer to evaluate RAG systems. It scores the relevance of the context to the query.

The `WeaveContextRelevanceScorerV1` uses a fine-tuned [deberta-small-long-nli](https://huggingface.co/tasksource/deberta-small-long-nli) model from tasksource. For more details, see the [WeaveContextRelevanceScorerV1 W\&B Report](https://wandb.ai/c-metrics/context-relevance-scorer/reports/Context-Relevance-Scorer--VmlldzoxMDYxNjEyNA).

### Usage notes

* The `score` method expects values for `query` and `output`.
* The context should be passed to the `output` parameter (string or list of strings).
* A higher score means a stronger prediction that the context is relevant to the query.
* You can pass `verbose=True` to the `score` method to get per-chunk scores.

### Usage example

```python lines theme={null}
import weave
from weave.scorers import WeaveContextRelevanceScorerV1

context_relevance_scorer = WeaveContextRelevanceScorerV1()

result = context_relevance_scorer.score(
    query="What is the capital of Antarctica?",
    output="The Antarctic has the happiest penguins."  # context is passed to the output parameter
)

print(f"Output is relevant: {result.passed}")
print(result)
```

***

## `WeaveCoherenceScorerV1`

This scorer checks whether the input text is coherent.

The `WeaveCoherenceScorerV1` uses a fine-tuned [deberta-small-long-nli](https://huggingface.co/tasksource/deberta-small-long-nli) model from tasksource. For more information, see the [WeaveCoherenceScorerV1 W\&B Report](https://wandb.ai/c-metrics/coherence_scorer/reports/Coherence-Scorer--VmlldzoxMDI5MjA1MA).

### Usage notes

* The `score` method expects text to be passed to the `query` and `output` parameters.
* A higher output score means a stronger prediction of coherence.

### Usage example

```python lines theme={null}
import weave
from weave.scorers import WeaveCoherenceScorerV1

coherence_scorer = WeaveCoherenceScorerV1()

result = coherence_scorer.score(
    query="What is the capital of Antarctica?",
    output="but why not monkey up day"
)

print(f"Output is coherent: {result.passed}")
print(result)
```

***

## `WeaveFluencyScorerV1`

This scorer checks whether the input text is fluent (readable and similar to natural human language). It evaluates grammar, syntax, and overall readability.

The `WeaveFluencyScorerV1` uses a fine-tuned [ModernBERT-base](https://huggingface.co/answerdotai/ModernBERT-base) model from AnswerDotAI. For more information, see the [WeaveFluencyScorerV1 W\&B Report](https://wandb.ai/c-metrics/fluency-eval/reports/Fluency-Scorer--VmlldzoxMTA3NzE2Ng).

### Usage notes

* The `score` method expects text to be passed to the `output` parameter.
* A higher output score indicates higher fluency.

### Usage example

```python lines theme={null}
import weave
from weave.scorers import WeaveFluencyScorerV1

fluency_scorer = WeaveFluencyScorerV1()

result = fluency_scorer.score(
    output="The cat did stretching lazily into warmth of sunlight."
)

print(f"Output is fluent: {result.passed}")
print(result)
```

***

## `WeaveTrustScorerV1`

Use this scorer when you want a single trust signal that combines multiple scorers instead of running each one individually.

The `WeaveTrustScorerV1` is a composite scorer for RAG systems. It evaluates the trustworthiness of model outputs by grouping other scorers into two categories: Critical and Advisory. Based on the composite score, it returns a trust level:

* `high`: No issues detected.
* `medium`: Only Advisory issues detected.
* `low`: Critical issues detected or input is empty.

Any input that fails a Critical scorer results in a `low` trust level. Failing an Advisory scorer results in `medium`.

* **Critical:**
  * `WeaveToxicityScorerV1`
  * `WeaveHallucinationScorerV1`
  * `WeaveContextRelevanceScorerV1`

* **Advisory:**
  * `WeaveFluencyScorerV1`
  * `WeaveCoherenceScorerV1`

### Usage notes

* Use this scorer to evaluate RAG pipelines.
* It requires `query`, `context`, and `output` keys for correct scoring.

### Usage example

```python lines theme={null}
import weave
from weave.scorers import WeaveTrustScorerV1

trust_scorer = WeaveTrustScorerV1()

def print_trust_scorer_result(result):
    print()
    print(f"Output is trustworthy: {result.passed}")
    print(f"Trust level: {result.metadata['trust_level']}")
    if not result.passed:
        print("Triggered scorers:")
        for scorer_name, scorer_data in result.metadata['raw_outputs'].items():
            if not scorer_data.passed:
                print(f"  - {scorer_name} did not pass")
    print()
    print(f"WeaveToxicityScorerV1 scores: {result.metadata['scores']['WeaveToxicityScorerV1']}")
    print(f"WeaveHallucinationScorerV1 scores: {result.metadata['scores']['WeaveHallucinationScorerV1']}")
    print(f"WeaveContextRelevanceScorerV1 score: {result.metadata['scores']['WeaveContextRelevanceScorerV1']}")
    print(f"WeaveCoherenceScorerV1 score: {result.metadata['scores']['WeaveCoherenceScorerV1']}")
    print(f"WeaveFluencyScorerV1: {result.metadata['scores']['WeaveFluencyScorerV1']}")
    print()

result = trust_scorer.score(
    query="What is the capital of Antarctica?",
    context="People in Antarctica love the penguins.",
    output="The cat stretched lazily in the warm sunlight."
)

print_trust_scorer_result(result)
print(result)
```

***

## `PresidioScorer`

This scorer uses the [Presidio library](https://github.com/microsoft/presidio) to detect personally identifiable information (PII) in your AI system's inputs and outputs.

### Usage notes

* To detect specific entity types, such as emails or phone numbers, pass a list of Presidio entities to the `selected_entities` parameter. Otherwise, Presidio detects all entity types in its default entities list.
* You can pass custom recognizers through the `custom_recognizers` parameter as a list of `presidio.EntityRecognizer` instances.
* To handle non-English input, use the `language` parameter to specify the language.

### Usage example

```python lines theme={null}
import weave
from weave.scorers import PresidioScorer

presidio_scorer = PresidioScorer()

result = presidio_scorer.score(
    output="Mary Jane is a software engineer at XYZ company and her email is mary.jane@xyz.com."
)

print(f"Output contains PII: {not result.passed}")
print(result)
```


# Integrations overview
Source: https://docs.wandb.ai/weave/guides/integrations

Trace agents and LLM applications with Weave's integrations for agent SDKs, harnesses, LLM providers, and orchestration frameworks.

Weave provides two ways to integrate with your AI stack:

* **Trace agents**: for multi-turn agentic applications built with agent SDKs (such as the OpenAI Agents SDK or Google ADK) or run inside agent harnesses (such as Claude Code, Codex, or Pi.dev). These integrations capture sessions, turns, LLM calls, and tool calls, and render them in the Weave **Agents** view.
* **Trace LLM applications**: for code that calls LLM providers (OpenAI, Anthropic, Bedrock, and others) or uses orchestration frameworks (LangChain, LlamaIndex, DSPy, and others). These integrations capture individual LLM calls and pipeline steps as Weave **Calls** in the **Traces** view.

If you're not sure which path to take, start with [Trace your agents](/weave/guides/tracking/trace-agents) for new multi-turn agent work, or [Trace LLM applications](/weave/guides/tracking/tracing) for existing LLM-call workflows.

## Trace agents

Use these integrations when you're building multi-turn agentic applications and want sessions, turns, and tool calls to appear in the Weave **Agents** view. For agent SDKs, harnesses, and custom instrumentation, see [Trace agent integrations](/weave/guides/tracking/trace-agent-integrations).

## Trace LLM applications

If your application calls an LLM provider's API directly or uses an orchestration framework, Weave can automatically intercept traces (using autopatching) for many libraries and frameworks. After you import the Weave SDK into your code and initialize it with `weave.init`, Weave records each request as a **Call** with inputs, outputs, latency, token usage, and cost.

For libraries Weave doesn't autopatch, you can manually apply [Weave Ops](/weave/guides/tracking/create-call#2-tracking-of-custom-functions) to your code to capture traces.

### LLM providers

LLM providers are the vendors that offer access to large language models to generate predictions. Weave integrates with these providers to log and trace the interactions with their APIs:

* **[Serverless Inference Service](/inference)**
* **[Amazon Bedrock](/weave/guides/integrations/bedrock)**
* **[Anthropic](/weave/guides/integrations/anthropic)**
* **[Cerebras](/weave/guides/integrations/cerebras)**
* **[Cohere](/weave/guides/integrations/cohere)**
* **[Google](/weave/guides/integrations/google)**
* **[Groq](/weave/guides/integrations/groq)**
* **[Hugging Face Hub](/weave/guides/integrations/huggingface)**
* **[LiteLLM](/weave/guides/integrations/litellm)**
* **[Microsoft Azure](/weave/guides/integrations/azure)**
* **[MistralAI](/weave/guides/integrations/mistral)**
* **[NVIDIA NIM](/weave/guides/integrations/nvidia_nim)**
* **[OpenAI](/weave/guides/integrations/openai)**
* **[OpenRouter](/weave/guides/integrations/openrouter)**
* **[Together AI](/weave/guides/integrations/together_ai)**
* **[Local Models](/weave/guides/integrations/local_models)** (for models running on your own infrastructure)

### Frameworks

Frameworks help orchestrate the execution pipelines in AI applications. They provide tools and abstractions for building complex workflows. Weave integrates with these frameworks to trace the entire pipeline:

* **[OpenAI Agents SDK (call-level tracing)](/weave/guides/integrations/openai_agents)**
* **[Claude Agent SDK (call-level tracing)](/weave/guides/integrations/claude_agent)**
* **[Claude Code (call-level tracing)](/weave/guides/integrations/claude_code)**
* **[LangChain](/weave/guides/integrations/langchain)**
* **[LlamaIndex](/weave/guides/integrations/llamaindex)**
* **[Haystack](/weave/guides/integrations/haystack)**
* **[DSPy](/weave/guides/integrations/dspy)**
* **[Instructor](/weave/guides/integrations/instructor)**
* **[CrewAI](/weave/guides/integrations/crewai)**
* **[Smolagents](/weave/guides/integrations/smolagents)**
* **[PydanticAI](/weave/guides/integrations/pydantic_ai)**
* **[Google Agent Development Kit (ADK, call-level tracing)](/weave/guides/integrations/google_adk)**
* **[AutoGen](/weave/guides/integrations/autogen)**
* **[Verdict](/weave/guides/integrations/verdict)**
* **[TypeScript SDK](/weave/guides/integrations/js)**
* **[Vercel AI SDK](/weave/guides/integrations/vercel_ai_sdk)**
* **[Agno](/weave/guides/integrations/agno)**
* **[Koog](/weave/guides/integrations/koog)**

### Reinforcement learning frameworks

Weave integrates with reinforcement learning frameworks to trace training and evaluation runs:

* **[Verifiers](/weave/guides/integrations/verifiers)**

### Protocols

Weave integrates with standardized protocols that enable communication between AI applications and their supporting services:

* **[Model Context Protocol (MCP)](/weave/guides/integrations/mcp)**


# Claude Agent SDK
Source: https://docs.wandb.ai/weave/guides/integrations/agents/claude-agents-sdk

Trace an agent built with the Claude Agent SDK using Weave.

<Note>
  Weave for Agents is in public preview. Features, APIs, and the Agents view UI may change before general availability.
</Note>

The [Claude Agent SDK](https://github.com/anthropics/claude-agent-sdk-python) is a Python SDK for building agent applications with Claude.

Weave automatically traces Claude Agent SDK calls, including agent queries, model responses, tool use, and multi-turn conversations. Weave displays the captured data in the **Agents** view of your project.

## Trace Claude Agent SDK agents with Weave

The Weave SDK autopatches the Claude Agent SDK, letting you capture traces from your Claude agents with minimal setup.

This doc shows how to initialize Weave and run a Claude agent with MCP tools through `ClaudeSDKClient`. Weave automatically traces the conversation, model calls, and tool calls end-to-end. When you're done, you can view a complete trace of the agent's query, model responses, and tool invocations in the **Agents** view of your project.

### Prerequisites

* A W\&B account and [API key](https://wandb.ai/authorize) set as a `WANDB_API_KEY` environment variable.
* An Anthropic API key set as an `ANTHROPIC_API_KEY` environment variable.
* Python 3.10+.

### Install packages

Install the following packages in your developer environment. The `weave` package captures traces, and `claude-agent-sdk` provides the agent runtime.

```bash theme={null}
pip install weave claude-agent-sdk
```

### Initialize Weave in your code

Add `weave.init` to the project, update your W\&B team and project names, and then build an agent the way you normally would. `weave.init` enables the autopatching that captures traces from the Claude Agent SDK.

The following code creates a Claude agent with two MCP math tools and runs it while Weave captures the traces.

```python lines theme={null}
import anyio
import weave

from claude_agent_sdk import (
    ClaudeAgentOptions,
    ClaudeSDKClient,
    create_sdk_mcp_server,
    tool,
)

weave.init("[YOUR-TEAM]/[YOUR-PROJECT]")


@tool("add", "Add two numbers", {"a": float, "b": float})
async def add(args: dict) -> dict:
    return {"content": [{"type": "text", "text": str(args["a"] + args["b"])}]}


@tool("multiply", "Multiply two numbers", {"a": float, "b": float})
async def multiply(args: dict) -> dict:
    return {"content": [{"type": "text", "text": str(args["a"] * args["b"])}]}


math_server = create_sdk_mcp_server(
    name="math",
    version="1.0.0",
    tools=[add, multiply],
)


async def main():
    options = ClaudeAgentOptions(
        mcp_servers={"math": math_server},
        allowed_tools=["mcp__math__add", "mcp__math__multiply"],
    )

    async with ClaudeSDKClient(options=options) as client:
        await client.query("Using the math tools, compute (3 + 7) * 2.")

        async for message in client.receive_response():
            print(message)


anyio.run(main) 
```

When the script runs, `weave.init()` prints a link to your project. Open the link to inspect the captured traces for the agent's query, model responses, and tool calls.

For details about viewing Agents data in Weave, see [View agent activity](/weave/guides/tracking/view-agent-activity).


# Claude Code plugin
Source: https://docs.wandb.ai/weave/guides/integrations/agents/claude-code-harness

Track Claude Code sessions in W&B Weave for observability and debugging.

<Note>
  Weave for Agents is in public preview. Features, APIs, and the Agents view UI may change before general availability.
</Note>

The Weave Claude Code plugin automatically traces every Claude Code session and sends the structured data to W\&B Weave. The plugin logs every turn, tool call, and subagent with no code changes required. Use these traces to debug sessions, audit tool usage, and monitor cost and latency across runs.

This guide walks you through installing the plugin, viewing your Claude Code traces in Weave, configuring the plugin, and managing its lifecycle.

<Warning>
  This plugin sends Claude Code session data to Weave. That data can include user prompts, Claude responses, tool inputs and outputs, file contents read by Claude Code tools, shell commands and output, and fetched URLs and page content.

  PII scrubbing and sensitive-data redaction aren't implemented. If you can't send this data to Weave under your security or compliance requirements, don't install this plugin.
</Warning>

## Prerequisites

* [Node.js](https://nodejs.org/) v18 or later (includes `npm`).
* [Claude Code](https://claude.ai/code) installed and authenticated.
* A W\&B account and [API key](https://wandb.ai/authorize) set as a `WANDB_API_KEY` environment variable.
* A Weave project (`[YOUR-TEAM]/[YOUR-PROJECT]`) to receive traces.

## Install the plugin

Install the CLI, run the installer to register the plugin with Claude Code, then start a Claude Code session to begin tracing.

<Steps>
  <Step title="Install the CLI">
    ```bash lines theme={null}
    npm install -g weave-claude-code
    ```
  </Step>

  <Step title="Run the installer">
    ```bash lines theme={null}
    weave-claude-code install
    ```

    The installer does the following:

    * Creates `~/.weave-claude-code/settings.json`.
    * Registers the plugin in Claude Code.
    * Prompts for your Weave project (`[YOUR-TEAM]/[YOUR-PROJECT]`) and W\&B API key if they aren't already set.

    To skip prompts in CI, bootstrap scripts, or other automated systems, set environment variables before running:

    ```bash lines theme={null}
    WEAVE_PROJECT=[YOUR-TEAM]/[YOUR-PROJECT] \
    WANDB_API_KEY=[YOUR-API-KEY] \
    weave-claude-code install --non-interactive
    ```

    In non-interactive mode, the installer still creates the config file and registers the plugin. It uses `WEAVE_PROJECT` and `WANDB_API_KEY` from the environment and warns if either is missing.
  </Step>

  <Step title="Start Claude Code">
    ```bash lines theme={null}
    claude
    ```

    The plugin traces sessions automatically from this point. Run a prompt or two, then open your Weave project to see the traces appear.
  </Step>
</Steps>

## View Claude Code traces in Weave

After running at least one Claude Code session, open your project in the Weave UI:

1. Navigate to [https://wandb.ai](https://wandb.ai) and select your project.
2. In the sidebar menu, select **Agents** to view all agent conversations saved for your project.
3. Select a conversation to inspect the full session tree.

For more information about the Agents view, see [View agent activity](/weave/guides/tracking/view-agent-activity).

Each user prompt produces one OTEL trace that follows the [GenAI semantic conventions](https://opentelemetry.io/docs/specs/semconv/gen-ai/). The trace shows a full turn hierarchy:

```text theme={null}
invoke_agent claude-code               (Root, one trace per user prompt.)
├─ chat <model>                        (Each LLM call within the turn.)
├─ execute_tool <tool_name>            (Each tool call such as Read, Bash, Grep.)
└─ invoke_agent <subagent_type>        (Subagent dispatched through the Agent tool.)
   ├─ chat <model>
   └─ execute_tool <tool_name>
```

Multi-turn conversations are linked server-side by session ID, so you can follow a conversation across multiple traces. Each span includes token usage, model name, tool inputs and outputs, timing, and the textual content of prompts and responses. For more information about traced data, see [What Gets Traced](https://github.com/wandb/weave-claude-code/blob/main/README.md#what-gets-traced) in GitHub.

## Configure the plugin

Use the `weave-claude-code config` commands to view or update plugin settings after installation:

```bash lines theme={null}
# Show all current settings.
weave-claude-code config show

# Set your Weave project.
weave-claude-code config set weave_project [YOUR-TEAM]/[YOUR-PROJECT]

# Set your W&B API key.
weave-claude-code config set wandb_api_key [YOUR-API-KEY]
```

Environment variables take precedence over the settings file:

```bash lines theme={null}
export WEAVE_PROJECT=[YOUR-TEAM]/[YOUR-PROJECT]
export WANDB_API_KEY=[YOUR-API-KEY]
```

## Weave skills

After installation, three Weave-specific skills are available directly inside any Claude Code session.

| Skill   | Command                | Description                                                                                                                                                             |
| ------- | ---------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Install | `/weave:weave-install` | Walks through the installation and configuration flow interactively. Use this on a fresh machine or to diagnose a broken setup.                                         |
| Status  | `/weave:weave-status`  | Checks the current plugin status and explains any issues. Equivalent to running `weave-claude-code status`, but Claude interprets the output and tells you what to fix. |
| Config  | `/weave:weave-config`  | Read or update plugin configuration without leaving Claude Code.                                                                                                        |

You can use the `weave:weave-config` skill to set Weave values from within Claude Code:

```text theme={null}
/weave:weave-config set weave_project [YOUR-TEAM]/[YOUR-PROJECT]
/weave:weave-config set wandb_api_key [YOUR-API-KEY]
```

## Check plugin status

You can use these CLI commands to check plugin status or troubleshoot issues:

```bash lines theme={null}
weave-claude-code status
```

Each line shows `✓` (OK), `✗` (action needed), or `-` (not yet active but not an error).

If sessions aren't appearing in Weave, check the daemon log:

```bash lines theme={null}
weave-claude-code logs
```

To tail the log in real time:

```bash lines theme={null}
weave-claude-code logs --follow
```

The log file is also at `~/.weave-claude-code/logs/daemon.log`.

### W\&B Dedicated Cloud or self-hosted instances

If you use W\&B Dedicated Cloud or a self-hosted instance, set `WANDB_BASE_URL` before launching Claude Code:

```bash lines theme={null}
export WANDB_BASE_URL=https://[YOUR-INSTANCE].wandb.io
```

The plugin runs a background daemon that reads `WANDB_BASE_URL` at startup. If the daemon is already running when you set the variable, it won't detect the change. To restart the daemon:

1. Shut down the daemon:
   ```bash lines theme={null}
   printf '{"command":"shutdown"}' | nc -U -w1 ~/.weave-claude-code/daemon.sock
   ```
2. Set `WANDB_BASE_URL`, or run `wandb login --host https://[YOUR-INSTANCE].wandb.io` to persist the setting across sessions.
3. Launch Claude Code again. The daemon restarts automatically and uses the correct URL.

## Uninstall

To remove the plugin from Claude Code, run:

```bash lines theme={null}
weave-claude-code uninstall
```

Pass `--keep-logs` to preserve the log directory.


# Codex plugin
Source: https://docs.wandb.ai/weave/guides/integrations/agents/codex-harness

Trace Codex agentic sessions, LLM calls, and tool executions in W&B Weave.

<Note>
  Weave for Agents is in public preview. Features, APIs, and the Agents view UI may change before general availability.
</Note>

The Weave Codex plugin automatically traces every Codex turn and sends the structured data to W\&B Weave. Every model call, tool execution, and reasoning step is logged with no changes to your Codex workflow. Use these traces to debug sessions, audit tool usage, and monitor cost and latency across runs.

The plugin reads Codex's own rollout session files (`~/.codex/sessions/**/rollout-*.jsonl`) to reconstruct spans. It runs entirely off Codex's critical path through a fire-and-forget Stop hook, so Codex never waits on the network.

<Warning>
  By default, this plugin captures span content: your prompts, the model's responses and reasoning, tool-call arguments, and tool results. Tool results include shell commands, command output, and file contents. This data is sent to your Weave instance.

  PII scrubbing and sensitive-data redaction aren't implemented. To send structure, token usage, model, and timing only (with no prompts, code, or output), set `WEAVE_CODEX_CAPTURE_CONTENT=0`. If you can't send this data to Weave under your security or compliance requirements, don't install this plugin.
</Warning>

## Prerequisites

* [Node.js](https://nodejs.org/) v20 or later.
* [OpenAI Codex CLI](https://github.com/openai/codex) with the hooks system.
* A W\&B account and [API key](https://wandb.ai/authorize) set as a `WANDB_API_KEY` environment variable.
* A Weave project (`[YOUR-TEAM]/[YOUR-PROJECT]`) to receive traces.

## Install the plugin

<Steps>
  <Step title="Install the package">
    ```bash lines theme={null}
    npm install -g weave-codex
    ```
  </Step>

  <Step title="Set credentials and project">
    ```bash lines theme={null}
    wandb login
    export WEAVE_PROJECT="YOUR-TEAM/YOUR-PROJECT"
    ```

    You can also set `WANDB_API_KEY` directly as an environment variable instead of using `wandb login`. See [Credential resolution order](#credential-resolution-order) for full precedence rules.
  </Step>

  <Step title="Install the Stop hook">
    ```bash lines theme={null}
    weave-codex install
    ```

    This merges a Stop hook into `~/.codex/hooks.json`. When each Codex turn completes, the hook spawns a detached worker that reads new rollout lines since a per-session cursor, reconstructs spans, and exports them to Weave.
  </Step>

  <Step title="Approve the hook in Codex">
    Codex marks newly added hooks as untrusted and won't run them until you approve. On your next `codex` launch, approve the `weave-codex` hook when prompted.

    Alternatively, set `bypass_hook_trust = true` in `~/.codex/config.toml` to skip the prompt.

    Run `weave-codex status` to confirm everything is configured correctly.
  </Step>
</Steps>

Now you can run Codex normally, and each completed turn appears in Weave within approximately one second.

## View Codex traces in Weave

After running at least one Codex session, open your project in the Weave UI:

1. Navigate to [https://wandb.ai](https://wandb.ai) and select your project.
2. In the sidebar, select **Agents** for the multi-turn chat view and per-agent version grouping, or select **Traces** for the raw span tree.
3. Select a conversation to inspect the full turn hierarchy.

For more on the Agents view, see [View agent activity](/weave/guides/tracking/view-agent-activity).

The plugin emits one OTEL trace per Codex turn, following the [GenAI semantic conventions](https://opentelemetry.io/docs/specs/semconv/gen-ai/):

| Span                  | Emitted on           | Key attributes                                                                                                                             |
| --------------------- | -------------------- | ------------------------------------------------------------------------------------------------------------------------------------------ |
| `invoke_agent codex`  | Per turn (root span) | Agent name or version, `gen_ai.conversation.id`, model, summed token usage, user prompt and final answer (when content capture is on)      |
| `chat <model>`        | Per model call       | `gen_ai.usage.*` (input, output, cached, reasoning tokens), finish reason, `server.address`, assistant output (when content capture is on) |
| `execute_tool <name>` | Per tool execution   | `gen_ai.tool.name`, `gen_ai.tool.call.id`, arguments and result (when content capture is on), and MCP calls also carry `mcp.server.name`   |

Weave groups turns into a single conversation in the Agents view using `gen_ai.conversation.id`, which is set to the Codex session ID on every span. Span timestamps are backdated from rollout file timestamps, so durations reflect actual execution time.

Traces also render in any OTEL-compatible backend, since all attributes follow the GenAI semantic conventions.

### Known limitations

* The `codex` (interactive TUI) and `codex exec` commands are supported. The `codex mcp` and `app-server` commands are not covered because they fire no hooks.
* A spawned subagent appears as its `spawn_agent` tool call only. Its own model calls and tool executions aren't captured.
* The Stop hook doesn't fire on aborted or errored turns, so those aren't captured.

## Configuration reference

This section lists the settings you can use to customize plugin behavior. Configuration and runtime files are stored in `~/.weave-codex/`, including `settings.json`, the hook shim, per-session cursors, and the log file at `logs/collector.log`.

| Setting         | Environment variable          | `settings.json` key | Default                            |
| --------------- | ----------------------------- | ------------------- | ---------------------------------- |
| W\&B API key    | `WANDB_API_KEY`               | `wandb_api_key`     | `~/.netrc` (through `wandb login`) |
| Weave project   | `WEAVE_PROJECT`               | `weave_project`     | None (required, `entity/project`)  |
| Base URL        | `WANDB_BASE_URL`              | `wandb_base_url`    | `https://trace.wandb.ai`           |
| Capture content | `WEAVE_CODEX_CAPTURE_CONTENT` | `capture_content`   | `true`                             |
| Debug logging   | `WEAVE_CODEX_DEBUG`           | `debug`             | Off (errors always log)            |

### Credential resolution order

The plugin resolves credentials in this order:

1. Environment variables (`WANDB_API_KEY`, `WEAVE_PROJECT`).
2. `~/.weave-codex/settings.json`.
3. `~/.netrc` entry for the Weave host.

### W\&B Dedicated Cloud or self-hosted instances

Set `WANDB_BASE_URL` to your install host before running Codex:

```bash lines theme={null}
export WANDB_BASE_URL=https://YOUR-INSTANCE.wandb.io
```

## Check plugin status

You can use these CLI commands to check plugin status or troubleshoot issues:

```bash lines theme={null}
weave-codex status
```

Each line shows `✓` (OK), `✗` (action needed), or `-` (not yet active but not an error). If turns don't appear in Weave, check the collector log:

```bash lines theme={null}
cat ~/.weave-codex/logs/collector.log
```

## Troubleshooting

The following sections describe common issues and how to resolve them. The collector log at `~/.weave-codex/logs/collector.log` is the primary diagnostic source. The plugin always logs errors regardless of the `debug` setting.

### No traces appear after running Codex

1. Run `weave-codex status`. Confirm all checks pass.
2. Confirm the hook is trusted. If you skipped the approval prompt on first launch, run `codex` again and approve when prompted, or set `bypass_hook_trust = true` in `~/.codex/config.toml`.
3. Confirm `WEAVE_PROJECT` is set to a valid `entity/project` slug. `weave-codex status` prints the resolved project.
4. Confirm the auth source. `weave-codex status` prints the resolved credential source. If it shows `WANDB_API_KEY env` but you set the key elsewhere, the plugin reads the wrong value.

### Turns appear but input/output text is empty

Content capture may be disabled. Check that `WEAVE_CODEX_CAPTURE_CONTENT` isn't set to `0` and that `capture_content` isn't set to `false` in `~/.weave-codex/settings.json`.

### Errors sending traces to Weave

If the plugin is active and generates spans that don't appear in Weave, check the collector log for an export error and match it against this table.

| Symptom                          | Most likely cause                | Fix                                                                                                                                 |
| -------------------------------- | -------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------- |
| 401 or 403 from `trace.wandb.ai` | Invalid or scope-limited API key | Verify the key is current and the team owns the entity and project. Check the resolved credential source with `weave-codex status`. |
| 404 from the agents endpoint     | Wrong base URL                   | For Dedicated Cloud installs, set `WANDB_BASE_URL` to your install host.                                                            |
| Connection refused or DNS error  | DNS, proxy, or firewall          | Confirm the host can reach `trace.wandb.ai` (cloud) or your install host (Dedicated Cloud) on port 443.                             |

### Hook-locked environments

If `allow_managed_hooks_only` is set in your Codex config, you can't add custom hooks directly. Use Codex's `notify` program as a fallback trigger instead:

```toml lines theme={null}
# ~/.codex/config.toml
notify = ["sh", "/Users/you/.weave-codex/stop-hook.sh"]
```

## Uninstall

```bash lines theme={null}
weave-codex uninstall
```

This removes only the `weave-codex` entries from `~/.codex/hooks.json`.


# Google ADK
Source: https://docs.wandb.ai/weave/guides/integrations/agents/google-adk

Trace an agent built with Google's Agent Development Kit (ADK) using Weave.

<Note>
  Weave for Agents is in public preview. Features, APIs, and the Agents view UI may change before general availability.
</Note>

[Google's Agent Development Kit (ADK)](https://google.github.io/adk-docs/) is a flexible, model-agnostic Python framework for building and orchestrating agents. While optimized for Gemini, ADK supports any model and supports both simple tasks and complex multi-agent workflows. Weave automatically traces agents built with ADK, including each agent invocation, sub-agent handoff, model call, and tool call. Weave displays the captured data in the **Agents** view of your project.

## Trace Google ADK agents with Weave

The Weave SDK autopatches with Google ADK so you can capture traces from your ADK agents with minimal setup. This guide shows how to initialize Weave and then run a multi-turn research agent built with Google ADK so that Weave captures every agent invocation, model call, and tool call across the session.

### Prerequisites

* A W\&B account and [API key](https://wandb.ai/authorize) set as a `WANDB_API_KEY` environment variable.
* A [Google API key](https://aistudio.google.com/apikey) for Gemini.
* Python 3.10+.

### Install packages

Install the following packages in your developer environment. These provide the Weave SDK, the Google ADK framework, and the HTTP client used by the example tool:

```bash theme={null}
pip install weave google-adk requests
```

### Initialize Weave in your code

Add `weave.init` to the project, along with your W\&B team and project names, and then build an agent the way you normally would. The following code creates a `research_assistant` agent that uses `gemini-2.5-flash` and a `wikipedia_search` tool, then runs three questions through a single ADK session while Weave captures the trace.

```python theme={null}
import asyncio
import requests
import weave
from google.adk.agents import Agent
from google.adk.runners import InMemoryRunner
from google.genai import types

weave.init("[YOUR-TEAM]/[YOUR-PROJECT-NAME]")

def wikipedia_search(query: str) -> dict:
    """Search Wikipedia for a topic and return its title and intro paragraph.

    Args:
        query: The topic to search for.

    Returns:
        A dictionary with the article title and intro extract.
    """
    r = requests.get(
        "https://en.wikipedia.org/w/api.php",
        params={
            "action": "query", "generator": "search", "gsrsearch": query, "gsrlimit": 1,
            "prop": "extracts", "exintro": True, "explaintext": True, "format": "json",
        },
        headers={"User-Agent": "weave-demo"},
    ).json()
    page = next(iter(r["query"]["pages"].values()))
    return {"title": page["title"], "extract": page["extract"]}

agent = Agent(
    name="research_assistant",
    model="gemini-2.5-flash",
    instruction=(
        "You are a research assistant. Use the wikipedia_search tool to look up "
        "topics when needed, and cite the article titles you used."
    ),
    tools=[wikipedia_search],
)

async def main():
    runner = InMemoryRunner(agent=agent, app_name="research-app")
    session = await runner.session_service.create_session(
        app_name="research-app", user_id="user-1"
    )

    questions = [
        "Who founded Anthropic?",
        "What is Claude (the AI assistant)?",
        "Summarize what we discussed in one sentence.",
    ]

    for question in questions:
        print(f"USER: {question}")
        async for event in runner.run_async(
            user_id="user-1",
            session_id=session.id,
            new_message=types.Content(
                role="user",
                parts=[types.Part(text=question)],
            ),
        ):
            if event.is_final_response() and event.content:
                print(f"AGENT: {event.content.parts[0].text}\n")

asyncio.run(main())
```

The example runs three turns in a single ADK session. The first two turns trigger Wikipedia lookups, and the third uses the previous conversation context to produce a summary without a tool call.

### See your agent traces in the Agents view

After the script runs, open your project's **Agents** view to confirm that Weave captured the session. `weave.init()` prints a link to your project when it runs. In the **Agents** view, inspect the following:

* A row in the **Agents** tab for `research_assistant`.
* A single session containing three turns.
* Each turn rendered as an `invoke_agent` span with nested model calls and tool calls.
* The full input, model, output, token usage, and Wikipedia results at each step.

For details on viewing Agents data in Weave, see [View agent activity](/weave/guides/tracking/view-agent-activity).


# OpenAI Agents SDK
Source: https://docs.wandb.ai/weave/guides/integrations/agents/openai-agents-sdk

Trace an agent built with the OpenAI Agents SDK using Weave.

<Note>
  Weave for Agents is in public preview. Features, APIs, and the Agents view UI may change before general availability.
</Note>

The [OpenAI Agents SDK](https://github.com/openai/openai-agents-python) is a lightweight framework for building agents and multi-agent workflows on top of OpenAI's API. Weave automatically traces agents built with the OpenAI Agents SDK, including each agent invocation, sub-agent handoff, model call, and tool call. Weave displays the captured data in the **Agents** view of your project.

<Note>
  The Weave TypeScript SDK does not support autopatching for the OpenAI Agents SDK.
</Note>

## Trace OpenAI Agents SDK agents with Weave

The Weave SDK autopatches with the OpenAI Agents SDK so you can capture traces from your agents with minimal setup. This guide explains how to initialize Weave and then run a multi-turn research agent built with the OpenAI Agents SDK so that Weave captures every agent invocation, model call, and tool call across the session.

### Prerequisites

Before you begin, make sure you have the following:

* A W\&B account and [API key](https://wandb.ai/authorize) set as a `WANDB_API_KEY` environment variable.
* An [OpenAI API key](https://platform.openai.com/api-keys).
* Python 3.10+.

### Install packages

Install the following packages in your developer environment so that Weave and the OpenAI Agents SDK are available to your script:

```bash theme={null}
pip install weave openai-agents requests
```

### Initialize Weave in your code

Add `weave.init` to the project, along with your W\&B team and project names, and then build your agent as usual. The following code defines a `wikipedia_search` function tool and a `Research assistant` agent, then runs three questions through the OpenAI Agents SDK `Runner` while Weave captures the trace.

```python lines theme={null}
import asyncio
import requests
import weave
from agents import Agent, Runner, function_tool

weave.init("<your-team>/<your-project-name>")

@function_tool
def wikipedia_search(query: str) -> str:
    """Search Wikipedia for a topic and return its title and intro paragraph."""
    r = requests.get(
        "https://en.wikipedia.org/w/api.php",
        params={
            "action": "query", "generator": "search", "gsrsearch": query, "gsrlimit": 1,
            "prop": "extracts", "exintro": True, "explaintext": True, "format": "json",
        },
        headers={"User-Agent": "weave-demo"},
    ).json()
    page = next(iter(r["query"]["pages"].values()))
    return f"{page['title']}: {page['extract']}"

agent = Agent(
    name="Research assistant",
    instructions=(
        "You are a research assistant. Use the wikipedia_search tool to look up "
        "topics when needed, and cite the article titles you used."
    ),
    tools=[wikipedia_search],
)

async def main():
    history = []
    for question in [
        "Who founded Anthropic?",
        "What is Claude (the AI assistant)?",
        "Summarize what we discussed in one sentence.",
    ]:
        history.append({"role": "user", "content": question})
        print(f"USER: {question}")
        result = await Runner.run(agent, input=history)
        print(f"AGENT: {result.final_output}\n")
        history = result.to_input_list()

asyncio.run(main())
```

The example runs three turns in a single conversation. The first two turns trigger Wikipedia lookups, and the third uses the prior conversation context to produce a summary without a tool call. Each call to `Runner.run` continues the conversation by passing the previous result's input list back as the next request.

### See your agent traces in the Agents view

After the script runs, you can review the captured activity in Weave. `weave.init()` prints a link to your project when it runs. Open the **Agents** view to inspect:

* A row in the **Agents** tab for `Research assistant`.
* A session containing three turns.
* Each turn rendered as an `invoke_agent` span with nested model calls and tool calls.
* The full input, model, output, token usage, and Wikipedia results at each step.

For details about viewing Agents data in Weave, see [View agent activity](/weave/guides/tracking/view-agent-activity).


# OpenClaw plugin
Source: https://docs.wandb.ai/weave/guides/integrations/agents/openclaw-harness

Track OpenClaw agent sessions in W&B Weave for observability and debugging.

<Note>
  Weave for Agents is in public preview. Features, APIs, and the Agents view UI may change before general availability.
</Note>

The Weave OpenClaw plugin automatically traces every agent session running through the OpenClaw gateway and sends the structured data to W\&B Weave. The plugin logs every turn, model call, and tool execution without requiring any application code changes. Use these traces to debug sessions, audit tool usage, and monitor cost and latency across runs.

This guide is for OpenClaw gateway operators who want to enable Weave tracing for the agents running behind the gateway. It walks you through how to install the plugin, configure it, view the resulting traces, and troubleshoot common issues.

<Warning>
  This plugin sends OpenClaw session data to Weave. That data can include user prompts, model responses, tool inputs and outputs, tool results, and conversation history.

  The plugin doesn't implement personally identifiable information (PII) scrubbing or sensitive-data redaction. If you need to suppress content capture, set `captureContent: false` in the plugin config. If you can't send this data to Weave under your security or compliance requirements, don't install this plugin.
</Warning>

## Prerequisites

* [Node.js](https://nodejs.org/) v22.14 or later.
* [OpenClaw](https://openclaw.dev) `2026.4.25` or later with the plugin API.
* A W\&B account and [API key](https://wandb.ai/authorize).
* A Weave project (`[YOUR-TEAM]/[YOUR-PROJECT]`) to receive traces.

## Install the plugin

Complete the following steps to install the plugin, register it with the OpenClaw gateway, and confirm that traces reach your Weave project.

<Steps>
  <Step title="Install the package">
    ```bash lines theme={null}
    openclaw plugins install weave-openclaw
    ```

    Use the full name `weave-openclaw` (`weave` on its own is the W\&B SDK, not this plugin). The OpenClaw gateway loads the plugin through its config. You don't import it from application code.
  </Step>

  <Step title="Add the plugin to your gateway config">
    The default config location is `~/.openclaw/openclaw.json` (JSON5, which allows comments and trailing commas). Run `openclaw onboard` to scaffold one if you don't have it yet.
    Update `[YOUR-TEAM]` and `[YOUR-PROJECT]` for your project.

    ```json lines theme={null}
    {
      plugins: {
        allow: ["weave"],
        entries: {
          weave: {
            enabled: true,
            config: { entity: "YOUR-TEAM", project: "YOUR-PROJECT" },
            hooks: { allowConversationAccess: true },
          },
        },
      },
    }
    ```

    Set **`hooks.allowConversationAccess`** to **`true`** so OpenClaw runs the content-bearing hooks (`llm_input`, `llm_output`, `agent_end`) and spans include input and output text, tool arguments, and tool results.

    `diagnostics.enabled` is on by default. Only set it explicitly if you need to turn it off.
  </Step>

  <Step title="Restart the gateway and verify">
    Restart the OpenClaw gateway, then run `/weave status` in any OpenClaw chat surface to confirm the plugin is active. Traces appear at `https://wandb.ai/[YOUR-TEAM]/[YOUR-PROJECT]` within a few seconds of your first agent run.
  </Step>
</Steps>

## View OpenClaw traces in Weave

Once the plugin is active, each agent session produces a trace you can inspect in the Weave UI. After running at least one agent session, open your project in the Weave UI:

1. Navigate to [https://wandb.ai](https://wandb.ai) and select your project.
2. In the sidebar, select **Agents** for the multi-turn chat view and per-agent version grouping, or select **Traces** for the raw span tree.
3. Select a conversation to inspect the full session tree.

For more information about the Agents view, see [View agent activity](/weave/guides/tracking/view-agent-activity).

The plugin emits spans per the [OpenTelemetry (OTel) GenAI semantic conventions](https://opentelemetry.io/docs/specs/semconv/gen-ai/):

| Span                   | Emitted on         | Key attributes                                                                    |
| ---------------------- | ------------------ | --------------------------------------------------------------------------------- |
| `invoke_agent <agent>` | Per agent run      | `gen_ai.agent.name`, `gen_ai.conversation.id`, cumulative cost, token usage       |
| `chat <model>`         | Per model call     | `gen_ai.request.model`, `gen_ai.usage.input_tokens`, `gen_ai.usage.output_tokens` |
| `execute_tool <tool>`  | Per tool execution | `gen_ai.tool.name`, `gen_ai.tool.call.id`                                         |

## Configuration reference

This section is the full configuration reference for the `weave` plugin entry in `openclaw.json`.

The `apiKey` field supports four auth sources, resolved in this order:

1. A `SecretRef` object with `source: "env"` or `source: "file"` (see line 10 in the following example).
2. A literal `apiKey` string (supported but not recommended).
3. The `WANDB_API_KEY` environment variable.
4. A `~/.netrc` entry for the Weave host, populated by `wandb login`.

```json lines theme={null}
{
  plugins: {
    entries: {
      weave: {
        enabled: true,
        config: {
          entity: "YOUR-TEAM",
          project: "YOUR-PROJECT",
          // Reads WANDB_API_KEY from env if apiKey is omitted.
          // SecretRef supports source: "env" or "file":
          //   { source: "env",  provider: "default", id: "WANDB_API_KEY" }
          //   { source: "file", provider: "default", id: "/run/secrets/wandb" }
          // Plain string is supported but discouraged.
          apiKey: { source: "env", provider: "default", id: "WANDB_API_KEY" },
          serviceName: "openclaw-agent",
          // Optional, improves Agents tab grouping.
          agentName: "my-agent",
          agentVersion: "v1.0",
          agentDescription: "What my agent does.",
          // ON by default. Set to false for a hard off (compliance or
          // retention policy). The plugin does not redact captured
          // strings; scrub upstream if needed.
          captureContent: true,
          flushIntervalMs: 1000,
        },
        hooks: { allowConversationAccess: true },
      },
    },
  },
}
```

`captureContent` is `true` by default. When `captureContent` is `true`, the plugin also emits input and output messages, tool arguments, and tool results following the `gen_ai.input.messages` and `gen_ai.output.messages` payload shape. The plugin stamps subagents, compaction events, loop detection, retry attempts, and context sizing as additional attributes and span events.

Set `captureContent` to `false` to turn off capture for compliance or retention policies.

### W\&B Dedicated Cloud or self-hosted instances

The plugin delegates endpoint and auth to the [Weave Node SDK](https://github.com/wandb/weave/tree/master/sdks/node). It reads the following environment variables using the same conventions as the Weave Python and Node SDKs:

| Variable              | Description                                                                                               |
| --------------------- | --------------------------------------------------------------------------------------------------------- |
| `WANDB_BASE_URL`      | W\&B API base URL. Default: `https://api.wandb.ai`. Set this for Dedicated Cloud or self-hosted installs. |
| `WF_TRACE_SERVER_URL` | Full trace-server URL override. Use for self-managed or proxied setups.                                   |

## Troubleshooting

If traces aren't reaching Weave or content fields are empty, use the following sections to diagnose the most common causes.

The gateway log is the terminal output from the process running `openclaw`, or your process manager's log stream if you daemonized it.

### Plugin loaded but no spans show up

1. Run `/weave status`. If lifecycle is `disabled`, `config-error`, or `not-started`, the plugin didn't activate. Check the gateway log for `weave: config.entity is required`, `weave: configuration error`, or `[weave] incompatible plugin SDK`.
2. Check you haven't set `diagnostics.enabled: false` in the gateway config. This field should be `true`.
3. Confirm the entity and project match the URL slug of the Weave project you're inspecting. `/weave status` should print `project=[YOUR-TEAM]/[YOUR-PROJECT]`.
4. Confirm the auth source. `/weave status` should print `auth=...`. If it says `WANDB_API_KEY env` but you set the key in a different environment variable, the plugin is reading the wrong key.

### Spans land but input/output text is empty

Look in the gateway log for:

```text theme={null}
[plugins] typed hook "llm_input"  blocked because non-bundled plugins must set
                                  plugins.entries.weave.hooks.allowConversationAccess=true
[plugins] typed hook "llm_output" blocked ...
[plugins] typed hook "agent_end"  blocked ...
```

OpenClaw gates content-bearing hooks behind an operator opt-in. Set `plugins.entries.weave.hooks.allowConversationAccess: true` in your config and restart the gateway. Span structure and cost and usage data come through diagnostic events, not hooks, so those keep working even when `allowConversationAccess` is `false`.

### Errors sending traces to Weave

If the plugin is active and generating spans but they don't appear in Weave, check the gateway log for an export error and match it against the following table.

| Symptom                              | Most likely cause                | Fix                                                                                                                                              |
| ------------------------------------ | -------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------ |
| `401` or `403` from `trace.wandb.ai` | Invalid or scope-limited API key | Verify the key is current and the team owns the entity and project. `wandb login` refreshes `~/.netrc`.                                          |
| `404` from the agents endpoint       | Wrong base or trace-server URL   | For dedicated installs, set `WANDB_BASE_URL` to your install host. For self-managed or proxy, set `WF_TRACE_SERVER_URL` to the trace-server URL. |
| Connection refused or DNS error      | DNS, proxy, or firewall          | Confirm the gateway host can reach `trace.wandb.ai` (cloud) or your install host (dedicated) on `443`.                                           |


# Pi extension
Source: https://docs.wandb.ai/weave/guides/integrations/agents/pi-dev-harness

Trace Pi agentic sessions, LLM calls, and tool executions in Weave.

<Note>
  Weave for Agents is in public preview. Features, APIs, and the Agents view UI may change before general availability.
</Note>

[Pi](https://pi.dev/) is a terminal-based coding agent. Weave traces Pi sessions, LLM calls, and tool executions automatically using the `createOtelExtension` integration, which conforms to the [GenAI semantic conventions](https://opentelemetry.io/docs/specs/semconv/gen-ai/). This page shows you how to enable Weave tracing in a Pi application so you can observe agent behavior, debug runs, and analyze token usage and cost.

<Warning>
  This integration sends Pi session data to Weave. That data can include user prompts, model responses, tool inputs and outputs, file contents read by Pi tools, shell commands and output, and fetched URLs and page content.

  The integration doesn't implement PII scrubbing or sensitive-data redaction. If you can't send this data to Weave under your security or compliance requirements, don't enable Weave tracing in your Pi application.
</Warning>

## Prerequisites

* [Node.js](https://nodejs.org/) (v18 or later).
* A W\&B account and [API key](https://wandb.ai/authorize) set as a `WANDB_API_KEY` environment variable.

<Note>
  Pi is a TypeScript and Node.js framework with no Python equivalent. Pi requires the ESM module system. Your project must use `"type": "module"` in `package.json`, or compile TypeScript to ESM output. CommonJS projects error. For more information about setting up an ESM project, see [TypeScript SDK integration](/weave/guides/integrations/js#set-up-an-esm-project).
</Note>

## Install packages

Install Weave, Pi, and Node type definitions as local project dependencies:

```bash lines theme={null}
npm install weave @earendil-works/pi-coding-agent
npm install --save-dev @types/node tsx typescript
```

## Trace a Pi prompt and response

The following example shows the minimum setup needed to trace a single Pi prompt and response. Call `weave.init()` before you create your agent session, then pass `createOtelExtension()` as an extension factory. Weave traces the full agent lifecycle: the session, each prompt-and-response cycle (`invoke_agent`), individual LLM calls (`chat`), and tool executions (`execute_tool`). `SessionManager.inMemory()` generates the session ID automatically.

```typescript lines theme={null}
import {init, createOtelExtension} from 'weave';

import {
  createAgentSession,
  DefaultResourceLoader,
  SessionManager,
  getAgentDir,
} from '@earendil-works/pi-coding-agent';

async function main() {
  // 1. Initialize Weave. Sets up the OTEL TracerProvider that points at your
  //    Weave project. All spans created by createOtelExtension() are
  //    automatically exported here.
  await init('[YOUR-TEAM]/[YOUR-PROJECT]');

  // 2. Create a resource loader and inject the Weave OTEL extension.
  //    The resource loader provides the Pi runtime environment and
  //    extension lifecycle used for tracing agent activity.
  const resourceLoader = new DefaultResourceLoader({
    cwd: process.cwd(),
    agentDir: getAgentDir(),
    extensionFactories: [createOtelExtension({})],
  });

  await resourceLoader.reload();

  // 3. Start the agent session
  const {session} = await createAgentSession({
    resourceLoader,
    sessionManager: SessionManager.inMemory(),
  });

  // 4. Bind extensions. Triggers session_start event so the OTEL adapter
  //    creates the root session span and captures the conversation ID.
  await session.bindExtensions({});

  // 5. Stream assistant output to stdout
  session.subscribe(event => {
    if (
      event.type === 'message_update' &&
      event.assistantMessageEvent.type === 'text_delta'
    ) {
      process.stdout.write(event.assistantMessageEvent.delta);
    }
  });

  // 6. Send a prompt and wait for the full response
  await session.prompt('What files are in the current directory?');
  console.log();
}

main();
```

Compile and execute the script with `tsx`, replacing `[FILENAME]` with the name of your TypeScript file:

```bash theme={null}
npx tsx [FILENAME].ts
```

When you run your code, your traces appear in the **Agents** tab of your Weave project at `https://wandb.ai/[YOUR-TEAM]/[YOUR-PROJECT]/weave/agents`. Weave captures Pi sessions, LLM calls, and tool executions for every run of your application.

### Next steps

To turn this example into a multi-turn session, add more prompts. Weave traces each call to `session.prompt()` as a separate `invoke_agent` span, all nested under a single root span. The agent retains context across prompts automatically.

After running the code, the **Agents** tab shows the full multi-turn timeline with nested LLM calls, tool executions, token usage, and cost.


# Anthropic
Source: https://docs.wandb.ai/weave/guides/integrations/anthropic

Use Weave to automatically track and log LLM calls made with the Anthropic SDK

<a href="https://colab.research.google.com/github/wandb/examples/blob/master/weave/docs/quickstart_anthropic.ipynb" aria-label="Open in Google Colab">
  <img alt="Open In Colab" />
</a>

When you integrate Weave into your code, it automatically tracks and logs LLM calls made with the Anthropic SDK for both [Python](https://github.com/anthropics/anthropic-sdk-python) and [TypeScript](https://github.com/anthropics/anthropic-sdk-typescript). Weave does this by automatically invoking Anthropic's `Messages.create()` method.

This guide shows you how to capture traces, wrap your own functions as ops, build a reusable `Model` for experimentation, and track tool use when working with the Anthropic SDK. Use these patterns to debug, compare, and iterate on Claude-powered applications without writing custom logging code.

## Traces

Weave automatically captures traces for the Anthropic SDK when you add `weave.init("your-team-name/your-project-name")` to your code. If you don't specify a team name as an argument in `weave.init()`, Weave logs output to your [default W\&B entity](/platform/app/settings-page/user-settings/#default-team). If you don't specify a project name, Weave fails to initialize.

The following examples demonstrate how to integrate Weave into a basic call to Anthropic:

<Tabs>
  <Tab title="Python">
    ```python lines {6} theme={null}
    import weave    
    # use the anthropic library as usual
    import os
    from anthropic import Anthropic

    weave.init("anthropic_project")

    client = Anthropic(
        api_key=os.environ.get("ANTHROPIC_API_KEY"),
    )

    message = client.messages.create(
        max_tokens=1024,
        messages=[
            {
                "role": "user",
                "content": "Tell me a joke about a dog",
            }
        ],
        model="claude-3-opus-20240229",
    )
    print(message.content)
    ```
  </Tab>

  <Tab title="TypeScript">
    ```typescript theme={null}
    import Anthropic from '@anthropic-ai/sdk';
    import * as weave from 'weave';
    import { wrapAnthropic } from 'weave';

    await weave.init('anthropic_project');

    // Wrap the Anthropic client to enable tracing
    const client = wrapAnthropic(new Anthropic());

    const message = await client.messages.create({
        max_tokens: 1024,
        messages: [
            {
                role: 'user',
                content: 'Tell me a joke about a dog',
            }
        ],
        model: 'claude-3-opus-20240229',
    });

    console.log(message.content);
    ```
  </Tab>
</Tabs>

By including `weave.init()` in the code, Weave automatically captures tracing information and outputs links. You can view the traces in the Weave UI by clicking on the links.

[<img alt="anthropic_trace.png" />](https://wandb.ai/capecape/anthropic_project/weave/calls)

## Wrap with your own ops

While `weave.init()` alone captures Anthropic SDK calls, decorating your own functions with ops gives you richer traces that include your application logic around each model call.

Weave ops automatically version your code as you experiment, and capture its inputs and outputs. Decorate a function with [`@weave.op()`](https://docs.wandb.ai/weave/guides/tracking/ops) in Python or wrap it with [`weave.op()`](/weave/reference/typescript-sdk/functions/op) in TypeScript to call [`Anthropic.messages.create()`](https://platform.claude.com/docs/en/build-with-claude/working-with-messages), and Weave tracks the inputs and outputs for you.

The following examples show you how to track a function:

<Tabs>
  <Tab title="Python">
    ```python lines {5,10,24} theme={null}
    import weave
    import os
    from anthropic import Anthropic

    weave.init("anthropic_project")
    client = Anthropic(
        api_key=os.environ.get("ANTHROPIC_API_KEY"),
    )

    @weave.op()
    def call_anthropic(user_input:str, model:str) -> str:
        message = client.messages.create(
        max_tokens=1024,
        messages=[
            {
                "role": "user",
                "content": user_input,
            }
            ],
            model=model,
        )
        return message.content[0].text

    @weave.op()
    def generate_joke(topic: str) -> str:
        return call_anthropic(f"Tell me a joke about {topic}", model="claude-3-haiku-20240307")

    print(generate_joke("chickens"))
    print(generate_joke("cars"))
    ```
  </Tab>

  <Tab title="TypeScript">
    ```typescript theme={null}
    import Anthropic from '@anthropic-ai/sdk';
    import * as weave from 'weave';
    import { wrapAnthropic } from 'weave';

    await weave.init('anthropic_project');
    const client = wrapAnthropic(new Anthropic());

    const callAnthropic = weave.op(async function callAnthropic(
        userInput: string,
        model: string
    ): Promise<string> {
        const message = await client.messages.create({
            max_tokens: 1024,
            messages: [
                {
                    role: 'user',
                    content: userInput,
                }
            ],
            model: model,
        });
        const content = message.content[0];
        return content.type === 'text' ? content.text : '';
    });

    const generateJoke = weave.op(async function generateJoke(
        topic: string
    ): Promise<string> {
        return callAnthropic(`Tell me a joke about ${topic}`, 'claude-3-haiku-20240307');
    });

    console.log(await generateJoke('chickens'));
    console.log(await generateJoke('cars'));
    ```
  </Tab>
</Tabs>

When you decorate a function with `weave.op()`, Weave captures the function's code, input, and output. You can use ops to track any function you want, including nested functions.

[<img alt="anthropic_ops.png" />](https://docs.wandb.ai/weave/guides/tracking/ops)

## Create a `Model` for easier experimentation

Once you're tracing individual calls, the next step is to organize related parameters together so you can compare different configurations.

<Note>
  The `weave.Model` class is only available in the Weave Python SDK. For TypeScript, use the `weave.op()` wrapper to track functions with structured parameters.
</Note>

Organizing experimentation is difficult when many pieces are moving. By using the [`Model`](/weave/guides/core-types/models) class, you can capture and organize the experimental details of your app like your system prompt or the model you're using. This helps you organize and compare different iterations of your app.

In addition to versioning code and capturing inputs and outputs, models capture structured parameters that control your application's behavior. This can help you find which parameters work best. You can also use Weave models with `serve` and [evaluations](/weave/guides/core-types/evaluations).

In the following example, you can experiment with `model` and `temperature`:

```python lines theme={null}
import weave    
# use the anthropic library as usual
import os
from anthropic import Anthropic
weave.init('joker-anthropic')

class JokerModel(weave.Model): # Change to `weave.Model`
  model: str
  temperature: float
  
  @weave.op()
  def predict(self, topic): # Change to `predict`
    client = Anthropic()
    message = client.messages.create(
    max_tokens=1024,
    messages=[
        {
            "role": "user",
            "content": f"Tell me a joke about {topic}",
        }
        ],
        model=self.model,
        temperature=self.temperature
    )
    return message.content[0].text


joker = JokerModel(
    model="claude-3-haiku-20240307",
    temperature = 0.1)
result = joker.predict("Chickens and Robots")
print(result)
```

Every time you change one of these values, Weave creates and tracks a new version of `JokerModel`. This lets you associate trace data with your code changes and can help you determine which configurations work best for your use case.

[<img alt="anthropic_model.png" />](https://wandb.ai/capecape/anthropic_project/weave/calls)

## Tools (function calling)

In addition to standard messages, Weave instruments Anthropic's tool use interface so you can trace agentic workflows end-to-end.

Anthropic provides a [tools](https://platform.claude.com/docs/en/agents-and-tools/tool-use/overview) interface that lets Claude request function calls. Weave automatically tracks tool definitions, tool use requests, and tool results throughout the conversation.

The following truncated examples demonstrate an Anthropic tool configuration:

<Tabs>
  <Tab title="Python">
    ```python lines theme={null}
    message = client.messages.create(
        max_tokens=1024,
        messages=[
            {
                "role": "user",
                "content": "What's the weather like in San Francisco?",
            }
        ],
        tools=[
            {
                "name": "get_weather",
                "description": "Get the current weather in a given location",
                "input_schema": {
                    "type": "object",
                    "properties": {
                        "location": {
                            "type": "string",
                            "description": "The city and state, e.g. San Francisco, CA",
                        }
                    },
                    "required": ["location"],
                },
            },
        ],
        model=model,
    )

    print(message)
    ```
  </Tab>

  <Tab title="TypeScript">
    ```typescript theme={null}
    const message = await client.messages.create({
        max_tokens: 1024,
        messages: [
            {
                role: 'user',
                content: "What's the weather like in San Francisco?",
            }
        ],
        tools: [
            {
                name: 'get_weather',
                description: 'Get the current weather in a given location',
                input_schema: {
                    type: 'object',
                    properties: {
                        location: {
                            type: 'string',
                            description: 'The city and state, e.g. San Francisco, CA',
                        }
                    },
                    required: ['location'],
                },
            },
        ],
        model: 'claude-3-opus-20240229',
    });

    console.log(message);
    ```
  </Tab>
</Tabs>

Weave automatically captures the tool definitions, Claude's tool use requests, and tool results at each step of the conversation.

[<img alt="anthropic_tool.png" />](https://wandb.ai/capecape/anthropic_project/weave/calls)


# Control automatic LLM call tracking
Source: https://docs.wandb.ai/weave/guides/integrations/autopatching

Control how W&B Weave automatically records calls to OpenAI, Anthropic, and other LLM libraries

If you use LLM provider libraries (such as OpenAI, Anthropic, Cohere, or Mistral) in your application, autopatching is how Weave traces your LLM calls for you. When you call `weave.init()`, Weave intercepts (patches) supported LLM client libraries. Your application code stays unchanged. You use the provider SDK as usual, and Weave records each request as a Weave Call.

This page describes when and how to change that behavior: turn off automatic tracking, limit it to specific providers, or post-process inputs and outputs (for example, to redact PII).

## Default behavior

By default, Weave automatically patches and tracks calls to common LLM libraries such as `openai` and `anthropic`. Call `weave.init(...)` at the start of your program and use those libraries normally. Their calls appear in your project's Traces.

## Configure autopatching

The following sections describe how to enable, disable, or fine-tune autopatching in Python and TypeScript. Choose the tab for your SDK.

<Tabs>
  <Tab title="Python">
    <Warning>
      The `autopatch_settings` argument is deprecated. Use `implicitly_patch_integrations=False` to disable implicit patching, or call specific patch functions like `patch_openai(settings={...})` to configure settings per integration.
    </Warning>

    Weave provides automatic implicit patching for all supported integrations by default.

    Implicit patching (automatic): Weave automatically patches libraries regardless of when you import them.

    ```python lines theme={null}
    # Option 1: Import before weave.init()
    import openai
    import weave
    weave.init('your-team-name/your-project-name')  # OpenAI is automatically patched.

    # Option 2: Import after weave.init()
    import weave
    weave.init('your-team-name/your-project-name')
    import anthropic  # Automatically patched via import hook.
    ```

    Disabling implicit patching: You can turn off automatic patching if you prefer explicit control.

    ```python lines theme={null}
    import weave

    # Option 1: Via settings parameter
    weave.init('your-team-name/your-project-name', settings={'implicitly_patch_integrations': False})

    # Option 2: Via environment variable
    # Set WEAVE_IMPLICITLY_PATCH_INTEGRATIONS=false before running your script

    # When implicit patching is disabled, you must explicitly patch integrations
    import openai
    weave.patch_openai()  # Now required for OpenAI tracing
    ```

    Explicit patching (manual): You can explicitly patch integrations for fine-grained control. This is useful when implicit patching is off or when you only want to trace a subset of providers.

    ```python lines theme={null}
    import weave
    weave.init('your-team-name/your-project-name')
    weave.integrations.patch_openai()  # Enable OpenAI tracing
    weave.integrations.patch_anthropic()  # Enable Anthropic tracing
    ```

    After you run the preceding code, Weave only traces the providers you explicitly patched.

    ### Post-process inputs and outputs

    In addition to turning patching on or off, you can transform what Weave records for each call. You can customize how Weave records inputs and outputs (for example, to redact PII or secrets) by passing settings to the patch function:

    ```python lines theme={null}
    import weave.integrations

    def redact_inputs(inputs: dict) -> dict:
        if "email" in inputs:
            inputs["email"] = "[REDACTED]"
        return inputs

    weave.init(...)
    weave.integrations.patch_openai(
        settings={
            "op_settings": {"postprocess_inputs": redact_inputs}
        }
    )
    ```

    For more information about handling sensitive data, see [How to use Weave with PII data](/weave/cookbooks/pii).
  </Tab>

  <Tab title="TypeScript">
    The TypeScript SDK only supports autopatching for OpenAI and Anthropic. Weave patches OpenAI when you import Weave, and it doesn't require any additional configuration.

    Additionally, the TypeScript SDK doesn't support:

    * Autopatching configuration or disabling.
    * Input and output post-processing.

    For edge cases where automatic patching doesn't work (ESM, bundlers like Next.js), use explicit wrapping to ensure Weave traces your OpenAI client:

    ```typescript theme={null}
    import OpenAI from 'openai'
    import * as weave from 'weave'
    import { wrapOpenAI } from 'weave'

    const client = wrapOpenAI(new OpenAI())
    await weave.init('your-team-name/your-project-name')
    ```

    For more information about ESM setup and troubleshooting, see the [TypeScript SDK Integration Guide](/weave/guides/integrations/js).
  </Tab>
</Tabs>


# Microsoft Azure
Source: https://docs.wandb.ai/weave/guides/integrations/azure

Integrate Weave with Microsoft Azure OpenAI services

Weights & Biases (W\&B) Weave integrates with Microsoft Azure OpenAI services, helping teams optimize their Azure AI applications. This page shows you how to start tracking calls to Azure OpenAI with Weave and points to additional resources for more advanced workflows.

<Tip>
  For the latest tutorials, visit [Weights & Biases on Microsoft Azure](https://wandb.ai/site/partners/azure).
</Tip>

## Get started

To start tracking Azure OpenAI calls with Weave, decorate the functions you want to track with `weave.op`. Once decorated, Weave automatically captures inputs, outputs, and metadata for each call to the function.

```python lines theme={null}
@weave.op()
def call_azure_chat(model_id: str, messages: list, max_tokens: int = 1000, temperature: float = 0.5):
    response = client.chat.completions.create(
        model=model_id,
        messages=messages,
        max_tokens=max_tokens,
        temperature=temperature
    )
    return {"status": "success", "response": response.choices[0].message.content}

```

## Learn more

The following sections describe advanced Azure with Weave topics.

### Use the Azure AI Model Inference API with Weave

Learn how to [use the Azure AI Model Inference API with Weave to gain insights into Azure models](https://wandb.ai/byyoung3/ML-NEWS2/reports/A-guide-to-using-the-Azure-AI-model-inference-API--Vmlldzo4OTY1MjEy#tutorial:-implementing-azure-ai-model-inference-api-with-w\&b-weave-).

### Trace Azure OpenAI models with Weave

Learn how to [trace Azure OpenAI models using Weave](https://wandb.ai/a-sh0ts/azure-weave-cookbook/reports/How-to-use-Azure-OpenAI-and-Azure-AI-Studio-with-W-B-Weave--Vmlldzo4MTI0NDgy).


# Bedrock
Source: https://docs.wandb.ai/weave/guides/integrations/bedrock

Track and monitor Amazon Bedrock LLM calls with Weave, capturing foundation model interactions and converse API usage.

Weave automatically tracks and logs LLM calls made through Amazon Bedrock, AWS's managed service that offers foundation models from multiple AI providers through a unified API. Use this integration to capture Bedrock foundation model interactions and `converse` API usage so you can debug, evaluate, and monitor your Bedrock-based applications.

You can log LLM calls to Weave from Amazon Bedrock in multiple ways. Use `weave.op` to create reusable operations for tracking any calls to a Bedrock model. Optionally, if you're using Anthropic models, you can use Weave's built-in integration with Anthropic.

<Tip>
  For the latest tutorials, visit [W\&B on Amazon Web Services](https://wandb.ai/site/partners/aws/).
</Tip>

## Traces

Weave automatically captures traces for Bedrock API calls once you patch the client. After you initialize Weave and patch the client, use the Bedrock client as usual:

```python lines theme={null}
import weave
import boto3
import json
from weave.integrations.bedrock.bedrock_sdk import patch_client

weave.init("my_bedrock_app")

# Create and patch the Bedrock client
client = boto3.client("bedrock-runtime")
patch_client(client)

# Use the client as usual
response = client.invoke_model(
    modelId="anthropic.claude-3-5-sonnet-20240620-v1:0",
    body=json.dumps({
        "anthropic_version": "bedrock-2023-05-31",
        "max_tokens": 100,
        "messages": [
            {"role": "user", "content": "What is the capital of France?"}
        ]
    }),
    contentType='application/json',
    accept='application/json'
)
response_dict = json.loads(response.get('body').read())
print(response_dict["content"][0]["text"])
```

The same patched client also captures traces when using the `converse` API:

```python lines theme={null}
messages = [{"role": "user", "content": [{"text": "What is the capital of France?"}]}]

response = client.converse(
    modelId="anthropic.claude-3-5-sonnet-20240620-v1:0",
    system=[{"text": "You are a helpful AI assistant."}],
    messages=messages,
    inferenceConfig={"maxTokens": 100},
)
print(response["output"]["message"]["content"][0]["text"])

```

## Wrap calls with your own ops

Wrap Bedrock calls in your own ops to group related logic, capture custom inputs, and reuse the same tracked function across your application. Create reusable operations with the `@weave.op()` decorator. The following example shows both the `invoke_model` and `converse` APIs:

```python lines theme={null}
@weave.op
def call_model_invoke(
    model_id: str,
    prompt: str,
    max_tokens: int = 100,
    temperature: float = 0.7
) -> dict:
    body = json.dumps({
        "anthropic_version": "bedrock-2023-05-31",
        "max_tokens": max_tokens,
        "temperature": temperature,
        "messages": [
            {"role": "user", "content": prompt}
        ]
    })

    response = client.invoke_model(
        modelId=model_id,
        body=body,
        contentType='application/json',
        accept='application/json'
    )
    return json.loads(response.get('body').read())

@weave.op
def call_model_converse(
    model_id: str,
    messages: str,
    system_message: str,
    max_tokens: int = 100,
) -> dict:
    response = client.converse(
        modelId=model_id,
        system=[{"text": system_message}],
        messages=messages,
        inferenceConfig={"maxTokens": max_tokens},
    )
    return response
```

<Frame>
  <img alt="Bedrock Converse integration with Weave and API call traces and response data" />
</Frame>

## Create a `Model` for easier experimentation

A Weave `Model` bundles configuration and prediction logic together so you can iterate on parameters and compare runs side by side. Create a Weave `Model` to better organize your experiments and capture parameters. The following example uses the `converse` API:

```python lines theme={null}
class BedrockLLM(weave.Model):
    model_id: str
    max_tokens: int = 100
    system_message: str = "You are a helpful AI assistant."

    @weave.op
    def predict(self, prompt: str) -> str:
        "Generate a response using Bedrock's converse API"
        
        messages = [{
            "role": "user",
            "content": [{"text": prompt}]
        }]

        response = client.converse(
            modelId=self.model_id,
            system=[{"text": self.system_message}],
            messages=messages,
            inferenceConfig={"maxTokens": self.max_tokens},
        )
        return response["output"]["message"]["content"][0]["text"]

# Create and use the model
model = BedrockLLM(
    model_id="anthropic.claude-3-5-sonnet-20240620-v1:0",
    max_tokens=100,
    system_message="You are an expert software engineer that knows a lot of programming. You prefer short answers."
)
result = model.predict("What is the best way to handle errors in Python?")
print(result)
```

This approach lets you version your experiments and track different configurations of your Bedrock-based application.

## Learn more

The following resources provide additional ways to explore and evaluate Amazon Bedrock with Weave.

### Try Bedrock in the Weave Playground

To experiment with Amazon Bedrock models in the Weave UI without any setup, try the [LLM Playground](../tools/playground).

### Report: Compare LLMs on Bedrock for text summarization with Weave

The [Evaluating LLMs on Amazon Bedrock](https://wandb.ai/byyoung3/ML_NEWS3/reports/Compare-LLMs-on-Amazon-Bedrock-for-text-summarization-with-W-B-Weave--VmlldzoxMDI1MTIzNw) report explains how to use Bedrock in combination with Weave to evaluate and compare LLMs for summarization tasks. The report includes code samples.


# Cerebras
Source: https://docs.wandb.ai/weave/guides/integrations/cerebras

Use Weave to trace and log LLM calls made via the Cerebras Cloud SDK

Weave automatically tracks and logs LLM calls made via the [Cerebras Cloud SDK](https://inference-docs.cerebras.ai/introduction).

## Traces

Tracking LLM calls is crucial for debugging and performance monitoring. Weave helps you do this by automatically capturing traces for the Cerebras Cloud SDK.

Here's an example of how to use Weave with Cerebras:

```python lines theme={null}
import os
import weave
from cerebras.cloud.sdk import Cerebras

# Initialise the weave project
weave.init("cerebras_speedster")

# Use the Cerebras SDK as usual
api_key = os.environ["CEREBRAS_API_KEY"]
model = "llama3.1-8b"  # Cerebras model

client = Cerebras(api_key=api_key)

response = client.chat.completions.create(
    model=model,
    messages=[{"role": "user", "content": "What's the fastest land animal?"}],
)

print(response.choices[0].message.content)
```

Weave now tracks and logs all LLM calls made through the Cerebras SDK. You can view the traces in the Weave web interface, including details like token usage and response time.

[<img alt="Weave trace view showing Cerebras LLM calls" />](https://wandb.ai/capecape/cerebras_speedster/weave/traces)

## Wrap with your own ops

Weave Ops enhance reproducibility and traceability in your experiments by automatically versioning your code and capturing inputs and outputs. Here's an example of how to use Weave Ops with the Cerebras SDK:

```python lines theme={null}
import os
import weave
from cerebras.cloud.sdk import Cerebras

# Initialise the weave project
weave.init("cerebras_speedster")

client = Cerebras(api_key=os.environ["CEREBRAS_API_KEY"])

# Weave will track the inputs, outputs and code of this function
@weave.op
def animal_speedster(animal: str, model: str) -> str:
    "Find out how fast an animal can run"
    
    response = client.chat.completions.create(
        model=model,
        messages=[{"role": "user", "content": f"How fast can a {animal} run?"}],
    )
    return response.choices[0].message.content

animal_speedster("cheetah", "llama3.1-8b")
animal_speedster("ostrich", "llama3.1-8b")
animal_speedster("human", "llama3.1-8b")
```

## Create a `Model` for easier experimentation

The [Model](/weave/guides/core-types/models) class in Weave helps you organize and compare different iterations of your app. This is useful when experimenting with Cerebras models. Here's an example:

```python lines theme={null}
import os
import weave
from cerebras.cloud.sdk import Cerebras

# Initialise the weave project
weave.init("cerebras_speedster")

client = Cerebras(api_key=os.environ["CEREBRAS_API_KEY"])

class AnimalSpeedModel(weave.Model):
    model: str
    temperature: float

    @weave.op
    def predict(self, animal: str) -> str:
        "Predict the top speed of an animal"        

        response = client.chat.completions.create(
            model=self.model,
            messages=[{"role": "user", "content": f"What's the top speed of a {animal}?"}],
            temperature=self.temperature
        )
        return response.choices[0].message.content

speed_model = AnimalSpeedModel(
    model="llama3.1-8b",
    temperature=0.7
)
result = speed_model.predict(animal="cheetah")
print(result)
```

With this setup, you can experiment with different models and parameters while tracking your Cerebras-powered inferences.

[<img alt="Weave Model traces for Cerebras experiments" />](https://wandb.ai/capecape/cerebras_speedster/weave/traces)


# Claude Agent SDK
Source: https://docs.wandb.ai/weave/guides/integrations/claude_agent

Use W&B Weave with the Claude Agent SDK to trace your agent's queries, tool calls, and multi-turn conversations.

The [Claude Agent SDK](https://github.com/anthropics/claude-agent-sdk-python) is a Python SDK by Anthropic for building agent applications with Claude. W\&B Weave automatically traces Claude Agent SDK calls when you call `weave.init()`, capturing queries, tool use, and multi-turn conversations.

This guide shows you how to instrument a Claude Agent SDK application with Weave so you can inspect every query, tool invocation, and conversation turn in the Weave UI. It's intended for developers building Claude-based agents who want end-to-end observability without writing custom tracing code.

## Prerequisites

Before you run the code example, complete the following setup so Weave can authenticate and route traces to your project:

* Replace `your-team-name` in the call to `weave.init()` with your W\&B team name.
* Set `WANDB_API_KEY` and `ANTHROPIC_API_KEY` in your environment. For more information on W\&B API keys, see [API keys](/platform/app/settings-page/user-settings#api-keys).

## Installation

Install the required dependencies with `pip` so both Weave and the Claude Agent SDK are available in your environment:

```bash theme={null}
pip install weave claude-agent-sdk
```

## Get started

This section walks through a complete example that demonstrates how Weave traces different Claude Agent SDK usage patterns in a single run.

When you call `weave.init()` before you use the Claude Agent SDK, Weave automatically patches the SDK and traces every query.

The following example showcases three capabilities:

* **Simple one-shot queries** with the `query()` function, including model responses and cost.
* **MCP tool use** with `ClaudeSDKClient`. The Claude Agent SDK supports custom tools through in-process MCP servers.
  * Weave traces tool definitions, tool invocations, and results alongside the agent's queries.
  * The MCP demo defines two MCP tools and runs a query that uses them.
* **Multi-turn conversations** that `weave.thread()` groups into a single trace. Weave captures each turn as a child of the thread, so you can see the full conversation flow in the Weave UI.

```python lines theme={null}
"""Weave + Claude Agent SDK: tracing queries, tools, and conversations.

Call `weave.init()` to auto-patch the SDK — every query, tool call, and
multi-turn conversation is recorded as a trace you can inspect in the Weave UI.
"""

import anyio

import weave
from claude_agent_sdk import (
    AssistantMessage,
    ClaudeAgentOptions,
    ClaudeSDKClient,
    ResultMessage,
    TextBlock,
    ToolUseBlock,
    create_sdk_mcp_server,
    tool,
)

# --- 1. Define MCP tools ---

@tool("add", "Add two numbers", {"a": float, "b": float})
async def add(args: dict) -> dict:
    return {"content": [{"type": "text", "text": str(args["a"] + args["b"])}]}


@tool("multiply", "Multiply two numbers", {"a": float, "b": float})
async def multiply(args: dict) -> dict:
    return {"content": [{"type": "text", "text": str(args["a"] * args["b"])}]}


math_server = create_sdk_mcp_server(
    name="math", version="1.0.0", tools=[add, multiply],
)


# --- 2. Simple one-shot query ---

async def simple_query():
    from claude_agent_sdk import query

    async for msg in query(prompt="What is 2+2?"):
        if isinstance(msg, AssistantMessage):
            for block in msg.content:
                if isinstance(block, TextBlock):
                    print(block.text, end="")
        elif isinstance(msg, ResultMessage):
            print(f"\ncost=${msg.total_cost_usd:.4f}")


# --- 3. MCP tool use ---

async def tool_query():
    options = ClaudeAgentOptions(
        mcp_servers={"math": math_server},
        allowed_tools=["mcp__math__add", "mcp__math__multiply"],
    )

    async with ClaudeSDKClient(options=options) as client:
        await client.query("Using the math tools, compute (3 + 7) * 2.")
        async for msg in client.receive_response():
            if isinstance(msg, AssistantMessage):
                for block in msg.content:
                    if isinstance(block, ToolUseBlock):
                        print(f"  [tool] {block.name}({block.input})")
                    elif isinstance(block, TextBlock):
                        print(block.text, end="")
            elif isinstance(msg, ResultMessage):
                print(f"\ncost=${msg.total_cost_usd:.4f}")


# --- 4. Multi-turn conversation with thread context ---

async def threaded_conversation():
    with weave.thread("my-conversation") as t:
        print(f"thread_id={t.thread_id}")

        async with ClaudeSDKClient() as client:
            for prompt in [
                "Name a famous sorting algorithm.",
                "What is its time complexity?",
            ]:
                await client.query(prompt)
                reply = ""
                async for msg in client.receive_response():
                    if isinstance(msg, AssistantMessage):
                        reply += "".join(
                            b.text for b in msg.content if isinstance(b, TextBlock)
                        )
                    elif isinstance(msg, ResultMessage):
                        print(f"Q: {prompt}\nA: {reply.strip()[:120]}\n")


# --- Run all examples ---

async def main():
    print("=== Simple Query ===")
    await simple_query()

    print("\n=== MCP Tool Use ===")
    await tool_query()

    print("\n=== Multi-Turn Thread ===")
    await threaded_conversation()


if __name__ == "__main__":
    weave.init("your-team-name/claude-agent-sdk-demo")
    anyio.run(main)
```

## View traces

After you run the example, Weave records a set of traces in your project that you can use to debug and analyze your agent's behavior.

When you run the example, Weave prints links to the Weave dashboard. Follow the links to see your traces, including query inputs, model responses, tool invocations, and conversation threads.


# Cohere
Source: https://docs.wandb.ai/weave/guides/integrations/cohere

Use Weave to automatically track and log LLM calls made through the Cohere Python library

<a href="https://colab.research.google.com/github/wandb/examples/blob/master/weave/docs/quickstart_cohere.ipynb" aria-label="Open in Google Colab">
  <img alt="Open In Colab" />
</a>

Weave automatically tracks and logs LLM calls made through the [Cohere Python library](https://github.com/cohere-ai/cohere-python) after you call `weave.init()`.

## Traces

It's important to store traces of LLM applications in a central database, both during development and in production. You use these traces for debugging and as a dataset that helps you improve your application.

Weave automatically captures traces for `cohere-python`. Use the library as usual. Start by calling `weave.init()`:

```python lines {8} theme={null}
import cohere
import os
import weave

# Use the Cohere library as usual
co = cohere.Client(api_key=os.environ["COHERE_API_KEY"])

weave.init("cohere_project")

response = co.chat(
    message="How is the weather in Boston?",
    # perform web search before answering the question. You can also use your own custom connector.
    connectors=[{"id": "web-search"}],
)
print(response.text)
```

If you don't specify a W\&B team when you call `weave.init()`, Weave uses your default entity. To find or update your default entity, refer to [User Settings](https://docs.wandb.ai/platform/app/settings-page/user-settings/#default-team) in the W\&B Models documentation.

Cohere models support [connectors](https://docs.cohere.com/docs/overview-rag-connectors#using-connectors-to-create-grounded-generations), which let you make requests to other APIs on the endpoint side. The response then contains the generated text with citation elements that link to the documents returned from the connector.

[<img alt="cohere_trace.png" />](https://wandb.ai/capecape/cohere_dev/weave/calls)

<Note>
  Weave patches the Cohere `Client.chat()`, `AsyncClient.chat()`, `Client.chat_stream()`, and `AsyncClient.chat_stream()` methods to track your LLM calls.
</Note>

## Wrap with your own ops

Weave ops make results *reproducible* by automatically versioning code as you experiment, and they capture their inputs and outputs. Create a function decorated with [`@weave.op()`](/weave/guides/tracking/ops) that calls into Cohere's chat methods, and Weave tracks the inputs and outputs for you. Here's an example:

```python lines {9} theme={null}
import cohere
import os
import weave

co = cohere.Client(api_key=os.environ["COHERE_API_KEY"])

weave.init("cohere_project")

@weave.op()
def weather(location: str, model: str) -> str:
    response = co.chat(
        model=model,
        message=f"How is the weather in {location}?",
        # perform web search before answering the question. You can also use your own custom connector.
        connectors=[{"id": "web-search"}],
    )
    return response.text

print(weather("Boston", "command"))
```

[<img alt="cohere_ops.png" />](https://wandb.ai/capecape/cohere_dev/weave/calls)

## Create a `Model` for easier experimentation

Organizing experimentation is difficult when there are many moving pieces. By using the [`Model`](/weave/guides/core-types/models) class, you can capture and organize the experimental details of your app like your system prompt or the model you're using. This helps organize and compare different iterations of your app.

Beyond versioning code and capturing inputs/outputs, [`Model`](/weave/guides/core-types/models)s capture structured parameters that control your application's behavior, helping you find what parameters worked best. You can also use Weave Models with `serve`, and [`Evaluation`](/weave/guides/core-types/evaluations)s.

In the following example, you can experiment with `model` and `temperature`. Every time you change one of these, you'll get a new *version* of `WeatherModel`.

```python lines theme={null}
import weave
import cohere
import os

weave.init('weather-cohere')

class WeatherModel(weave.Model):
    model: str
    temperature: float
  
    @weave.op()
    def predict(self, location: str) -> str:
        co = cohere.Client(api_key=os.environ["COHERE_API_KEY"])
        response = co.chat(
            message=f"How is the weather in {location}?",
            model=self.model,
            temperature=self.temperature,
            connectors=[{"id": "web-search"}]
        )
        return response.text

weather_model = WeatherModel(
    model="command",
    temperature=0.7
)
result = weather_model.predict("Boston")
print(result)
```

[<img alt="cohere_model.png" />](https://wandb.ai/capecape/cohere_dev/weave/models)


# Google
Source: https://docs.wandb.ai/weave/guides/integrations/google

Use Weave to trace and log Google GenAI model calls

<a href="https://github.com/wandb/examples/blob/master/weave/docs/quickstart_google.ipynb">
  <img alt="Open In Colab" />
</a>

<Note>
  You can experiment with Google AI models on Weave without any setup using the [LLM Playground](../tools/playground).
</Note>

This page describes how to use W\&B Weave with the Google Vertex AI API and the Google Gemini API so you can evaluate, monitor, and iterate on your Google GenAI applications.

Weave automatically captures traces for the:

* [Google GenAI SDK](https://github.com/googleapis/python-genai), which is accessible through Python SDK, Node.js SDK, Go SDK, and REST.
* [Google Vertex AI API](https://cloud.google.com/vertex-ai/docs), which provides access to Google's Gemini models and [partner models](https://cloud.google.com/vertex-ai/generative-ai/docs/partner-models/use-partner-models).

<Note>
  Weave also supports the deprecated [Google AI Python SDK for the Gemini API](https://github.com/google-gemini/deprecated-generative-ai-python). This support is deprecated as well and is scheduled for removal in a future version.
</Note>

## Get started

The following examples show how to enable tracing for each supported SDK. In both cases, calling `weave.init` is the only Weave-specific step required. Your existing Google GenAI or Vertex AI code stays the same.

Weave automatically captures traces for the [Google GenAI SDK](https://github.com/googleapis/python-genai). To start tracking, call `weave.init(project_name="[YOUR-WANDB-PROJECT-NAME]")` and use the library as normal.

```python lines theme={null}
import os
from google import genai
import weave

weave.init(project_name="google-genai")

google_client = genai.Client(api_key=os.getenv("GOOGLE_GENAI_KEY"))
response = google_client.models.generate_content(
    model="gemini-2.0-flash",
    contents="What's the capital of France?",
)
```

[<img alt="dspy_trace.png" />](https://wandb.ai/geekyrakshit/google-genai/weave/traces)

Weave also automatically captures traces for [Vertex APIs](https://github.com/googleapis/python-aiplatform/tree/main/vertexai/generative_models). To start tracking, call `weave.init(project_name="[YOUR-WANDB-PROJECT-NAME]")` and use the library as normal.

```python lines theme={null}
import vertexai
import weave
from vertexai.generative_models import GenerativeModel

weave.init(project_name="vertex-ai-test")
vertexai.init(project="[YOUR-VERTEX-AI-PROJECT-NAME]", location="[YOUR-VERTEX-AI-PROJECT-LOCATION]")
model = GenerativeModel("gemini-1.5-flash-002")
response = model.generate_content(
    "What's a good name for a flower shop specialising in selling dried flower bouquets?"
)
```

## Track your own ops

Wrap a function with `@weave.op` to start capturing inputs, outputs, and app logic so you can debug how data flows through your app. You can deeply nest ops and build a tree of functions that you want to track. This also starts automatically versioning code as you experiment to capture ad-hoc details that haven't been committed to git.

Create a function decorated with [`@weave.op`](/weave/guides/tracking/ops).

In the following example, the function `recommend_places_to_visit` is wrapped with `@weave.op` and recommends places to visit in a city.

```python lines theme={null}
import os
from google import genai
import weave

weave.init(project_name="google-genai")
google_client = genai.Client(api_key=os.getenv("GOOGLE_GENAI_KEY"))


@weave.op()
def recommend_places_to_visit(city: str, model: str = "gemini-1.5-flash"):
    response = google_client.models.generate_content(
        model=model,
        contents="You are a helpful assistant meant to suggest all budget-friendly places to visit in a city",
    )
    return response.text


recommend_places_to_visit("New York")
recommend_places_to_visit("Paris")
recommend_places_to_visit("Kolkata")
```

[<img alt="dspy_trace.png" />](https://wandb.ai/geekyrakshit/google-genai/weave/traces)

## Create a `Model` for easier experimentation

Organizing experimentation is difficult when there are many moving pieces. By using the [`Model`](../core-types/models) class, you can capture and organize the experimental details of your app like your system prompt or the model you're using. This helps organize and compare different iterations of your app.

In addition to versioning code and capturing inputs/outputs, [`Model`](../core-types/models)s capture structured parameters that control your application's behavior, helping you find what parameters worked best. You can also use Weave Models with `serve`, and [`Evaluation`](../core-types/evaluations)s.

In the following example, you can experiment with `CityVisitRecommender`. Every time you change one of these, you'll get a new *version* of `CityVisitRecommender`. This gives you a versioned record of each configuration you try, which you can later compare or evaluate.

```python lines theme={null}
import os
from google import genai
import weave

weave.init(project_name="google-genai")
google_client = genai.Client(api_key=os.getenv("GOOGLE_GENAI_KEY"))


class CityVisitRecommender(weave.Model):
    model: str

    @weave.op()
    def predict(self, city: str) -> str:
        response = google_client.models.generate_content(
            model=self.model,
            contents="You are a helpful assistant meant to suggest all budget-friendly places to visit in a city",
        )
        return response.text


city_recommender = CityVisitRecommender(model="gemini-1.5-flash")
print(city_recommender.predict("New York"))
print(city_recommender.predict("San Francisco"))
print(city_recommender.predict("Los Angeles"))
```


# Groq
Source: https://docs.wandb.ai/weave/guides/integrations/groq

Track and monitor Groq LPU inference with Weave, capturing model calls, performance metrics, and function chains for LLM applications.

<a href="https://colab.research.google.com/github/wandb/examples/blob/master/weave/docs/quickstart_groq.ipynb" aria-label="Open in Google Colab">
  <img alt="Open In Colab" />
</a>

<Note>
  Do you want to experiment with Groq models on Weave without any setup? Try the [LLM Playground](../tools/playground).
</Note>

[Groq](https://groq.com/) is an AI infrastructure company that delivers fast AI inference. The LPU Inference Engine by Groq is a hardware and software platform built for compute speed, quality, and energy efficiency. Weave automatically tracks and logs Groq chat completion calls.

This page explains how to use Weave to trace Groq chat completion calls, wrap your own functions as Weave ops, and organize your experiments using `Model` objects.

## Tracing

It's important to store traces of language model applications in a central location, both during development and in production. These traces can help you debug your application and serve as a dataset to improve it.

Weave automatically captures traces for [Groq](https://groq.com/). To start tracking, call `weave.init(project_name="<your-wandb-project-name>")` and use the library as normal. Replace values enclosed in `<>` with your own.

```python lines theme={null}
import os
import weave
from groq import Groq

weave.init(project_name="groq-project")

client = Groq(
    api_key=os.environ.get("GROQ_API_KEY"),
)
chat_completion = client.chat.completions.create(
    messages=[
        {
            "role": "user",
            "content": "Explain the importance of fast language models",
        }
    ],
    model="llama3-8b-8192",
)
```

| <img alt="Groq Weave dashboard with tracked LLM calls, performance metrics, and trace information" />                      |
| -------------------------------------------------------------------------------------------------------------------------- |
| Weave now tracks and logs all LLM calls made through the Groq library. You can view the traces in the Weave web interface. |

## Track your own ops

Wrap a function with `@weave.op` to capture inputs, outputs, and app logic so you can debug how data flows through your app. You can deeply nest ops and build a tree of functions that you want to track. This also automatically versions code as you experiment to capture ad-hoc details that haven't been committed to git.

Create a function decorated with [`@weave.op`](/weave/guides/tracking/ops).

In the following example, the `recommend_places_to_visit` function is wrapped with `@weave.op` and recommends places to visit in a city.

```python lines theme={null}
import os
import weave
from groq import Groq


weave.init(project_name="groq-test")

client = Groq(
    api_key=os.environ.get("GROQ_API_KEY"),
)

@weave.op()
def recommend_places_to_visit(city: str, model: str="llama3-8b-8192"):
    chat_completion = client.chat.completions.create(
        messages=[
            {
                "role": "system",
                "content": "You are a helpful assistant meant to suggest places to visit in a city",
            },
            {
                "role": "user",
                "content": city,
            }
        ],
        model="llama3-8b-8192",
    )
    return chat_completion.choices[0].message.content


recommend_places_to_visit("New York")
recommend_places_to_visit("Paris")
recommend_places_to_visit("Kolkata")
```

| <img alt="Groq Weave tracing interface with op decoration and function call hierarchy and trace details" />                                          |
| ---------------------------------------------------------------------------------------------------------------------------------------------------- |
| Decorate the `recommend_places_to_visit` function with `@weave.op` to trace its inputs, outputs, and all internal LM calls made inside the function. |

## Create a `Model` for easier experimentation

Organizing experimentation is difficult when there are many moving pieces. By using the [`Model`](../core-types/models) class, you can capture and organize the experimental details of your app like your system prompt or the model you're using. This helps organize and compare different iterations of your app.

In addition to versioning code and capturing inputs and outputs, [`Model`](../core-types/models)s capture structured parameters that control your application's behavior, helping you find what parameters worked best. You can also use Weave Models with `serve`, and [`Evaluation`](../core-types/evaluations)s.

In the following example, you can experiment with `GroqCityVisitRecommender`. Every time you change one of these, you get a new *version* of `GroqCityVisitRecommender`.

```python lines theme={null}
import os
from groq import Groq
import weave


class GroqCityVisitRecommender(weave.Model):
    model: str
    groq_client: Groq

    @weave.op()
    def predict(self, city: str) -> str:
        system_message = {
            "role": "system",
            "content": """
You are a helpful assistant meant to suggest places to visit in a city
""",
        }
        user_message = {"role": "user", "content": city}
        chat_completion = self.groq_client.chat.completions.create(
            messages=[system_message, user_message],
            model=self.model,
        )
        return chat_completion.choices[0].message.content


weave.init(project_name="groq-test")
city_recommender = GroqCityVisitRecommender(
    model="llama3-8b-8192", groq_client=Groq(api_key=os.environ.get("GROQ_API_KEY"))
)
print(city_recommender.predict("New York"))
print(city_recommender.predict("San Francisco"))
print(city_recommender.predict("Los Angeles"))
```

| <img alt="Groq Weave Model tracing and versioning interface with model versions, trace history, and performance metrics" /> |
| --------------------------------------------------------------------------------------------------------------------------- |
| Trace and version your calls using a [`Model`](../core-types/models).                                                       |

### Serve a Weave Model

After you have a versioned `Model`, you can deploy it as a service for testing or downstream applications. Given a Weave reference to any `weave.Model` object, you can spin up a FastAPI server and [serve](https://docs.wandb.ai/weave/guides/tools/serve) it.

| [<img alt="dspy_weave_model_serve.png" />](https://wandb.ai/geekyrakshit/groq-test/weave/objects/GroqCityVisitRecommender/versions/6O1xPTJ9yFx8uuCjJAlI7KgcVYxXKn7JxfmVD9AQT5Q) |
| ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| You can find the Weave reference of any `weave.Model` by navigating to the model and copying it from the UI.                                                                    |

Serve your model with the following command in the terminal:

```shell theme={null}
weave serve weave://your_entity/project-name/YourModel:<hash>
```


# Hugging Face Hub
Source: https://docs.wandb.ai/weave/guides/integrations/huggingface

Integrate Hugging Face Hub with W&B Weave to track and analyze your machine learning applications

<Note>
  All code samples shown on this page are in Python.
</Note>

This page explains how to integrate [Hugging Face Hub](https://huggingface.co/) with W\&B Weave to track and analyze your machine learning applications. You'll learn how to log model inferences, monitor function calls, and organize experiments using Weave's tracing and versioning capabilities. By following the examples provided, you can capture insights, debug your applications, and compare different model configurations, all within the Weave web interface.

<Tip>
  **Try Hugging Face Hub with Weave in Google Colab**
  Do you want to experiment with Hugging Face Hub and Weave without any of the setup? You can try the code samples shown here as a Jupyter Notebook on Google Colab.

  <a href="https://colab.research.google.com/github/wandb/examples/blob/master/weave/docs/quickstart_huggingface.ipynb" aria-label="Open in Google Colab">
    <img alt="Open In Colab" />
  </a>
</Tip>

## Overview

Hugging Face Hub is a machine learning platform for creators and collaborators, offering a large collection of pre-trained models and datasets for many types of projects.

The `huggingface_hub` Python library provides a unified interface to run inference across multiple services for models hosted on the Hub. You can invoke these models using the [`InferenceClient`](https://huggingface.co/docs/huggingface_hub/en/package_reference/inference_client).

Weave automatically captures traces for `InferenceClient`. To start tracking, call `weave.init()` and use the library as usual.

## Prerequisites

Before you start, complete the following setup steps to install the required libraries and configure access to the Hugging Face Hub:

1. Before you can use `huggingface_hub` with Weave, you must install the necessary libraries, or upgrade to the latest versions. The following command installs or upgrades `huggingface_hub` and `weave` to the latest version if they're already installed, and reduces installation output.

   ```python lines theme={null}
   pip install -U huggingface_hub weave -qqq
   ```

2. To use inference with a model on the Hugging Face Hub, set your [User Access Token](https://huggingface.co/docs/hub/security-tokens). You can either set the token from your [Hugging Face Hub Settings page](https://huggingface.co/login?next=%2Fsettings%2Ftokens) or programmatically. The following code sample prompts you to enter your `HUGGINGFACE_TOKEN` and sets the token as an environment variable.

   ```python lines theme={null}
   import os
   import getpass

   os.environ["HUGGINGFACE_TOKEN"] = getpass.getpass("Enter your Hugging Face Hub Token: ")
   ```

## Basic tracing

Storing traces of language model applications in a central location is essential during development and production. These traces help with debugging and serve as datasets for improving your application.

Weave automatically captures traces for the `InferenceClient`. To start tracking, initialize Weave by calling `weave.init()`, then use the library as usual.

The following example demonstrates how to log inference calls to the Hugging Face Hub using Weave:

```python lines theme={null}
import weave
from huggingface_hub import InferenceClient

# Initialize Weave
weave.init(project_name="quickstart-huggingface")

# Initialize Hugging Face Inference Client
huggingface_client = InferenceClient(
    api_key=os.environ.get("HUGGINGFACE_TOKEN")
)

# Make a chat completion inference call to the Hugging Face Hub with the Llama-3.2-11B-Vision-Instruct model
image_url = "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
response = huggingface_client.chat_completion(
    model="meta-llama/Llama-3.2-11B-Vision-Instruct",
    messages=[
        {
            "role": "user",
            "content": [
                {"type": "image_url", "image_url": {"url": image_url}},
                {"type": "text", "text": "Describe this image in one sentence."},
            ],
        }
    ],
    max_tokens=500,
    seed=42,
)
```

After the previous code runs, Weave tracks and logs all LLM calls made with the Hugging Face Inference Client. You can view these traces in the Weave web interface.

<Frame>
  <img alt="Weave logs each inference call, providing details about inputs, outputs, and metadata." />
</Frame>

Weave logs each inference call, providing details about inputs, outputs, and metadata.

<img alt="Weave chat view in the UI with the entire chat history with the model" />

Weave also renders the call as a chat view in the UI, displaying the entire chat history with the model.

## Trace a function

To gain deeper insights into how data flows through your application, you can use `@weave.op` to track function calls. This captures inputs, outputs, and execution logic, helping with debugging and performance analysis.

By nesting multiple ops, you can build a structured tree of tracked functions. Weave also automatically versions your code, preserving intermediate states as you experiment, even before you commit changes to Git.

To start tracking, decorate the functions that you want to track with `@weave.op`.

In the following example, Weave tracks three functions: `generate_image`, `check_image_correctness`, and `generate_image_and_check_correctness`. These functions generate an image and validate whether it matches a given prompt.

```python lines theme={null}
import base64
from PIL import Image


def encode_image(pil_image):
    import io
    buffer = io.BytesIO()
    pil_image.save(buffer, format="JPEG")
    buffer.seek(0)
    encoded_image = base64.b64encode(buffer.read()).decode("utf-8")
    return f"data:image/jpeg;base64,{encoded_image}"


@weave.op
def generate_image(prompt: str):
    return huggingface_client.text_to_image(
        prompt=prompt,
        model="black-forest-labs/FLUX.1-schnell",
        num_inference_steps=4,
    )


@weave.op
def check_image_correctness(image: Image.Image, image_generation_prompt: str):
    return huggingface_client.chat_completion(
        model="meta-llama/Llama-3.2-11B-Vision-Instruct",
        messages=[
            {
                "role": "user",
                "content": [
                    {"type": "image_url", "image_url": {"url": encode_image(image)}},
                    {
                        "type": "text",
                        "text": f"Is this image correct for the prompt: {image_generation_prompt}? Answer with only one word: yes or no",
                    },
                ],
            }
        ],
        max_tokens=500,
        seed=42,
    ).choices[0].message.content


@weave.op
def generate_image_and_check_correctness(prompt: str):
    image = generate_image(prompt)
    return {
        "image": image,
        "is_correct": check_image_correctness(image, prompt),
    }


response = generate_image_and_check_correctness("A cute puppy")
```

Weave now logs all function calls wrapped with `@weave.op`, letting you analyze execution details in the Weave UI.

<Frame>
  <img alt="Weave now logs all function calls wrapped with @weave.op, letting you analyze execution details in the Weave UI. Weave also captures and visualizes function execution, helping you to understand data flow and logic within your application." />

  Weave also captures and visualizes function execution, helping you to understand data flow and logic within your application.
</Frame>

## Use `Model`s for experimentation

Managing LLM experiments can be challenging when multiple components are involved. The Weave [`Model`](../core-types/models) class helps capture and organize experimental details, such as system prompts and model configurations, letting you compare different iterations.

In addition to versioning code and capturing inputs and outputs, a `Model` stores structured parameters that control application behavior. This helps you track which configurations produced the best results. You can also integrate a Weave `Model` with Weave [Serve](../tools/serve) and [Evaluations](../evaluation/scorers) for further insights.

The following example defines a `CityVisitRecommender` model for travel recommendations. Each modification to its parameters generates a new version, supporting iterative experimentation.

```python lines theme={null}
import rich


class CityVisitRecommender(weave.Model):
    model: str
    temperature: float = 0.7
    max_tokens: int = 500
    seed: int = 42

    @weave.op()
    def predict(self, city: str) -> str:
        return huggingface_client.chat_completion(
            model=self.model,
            messages=[
                {
                    "role": "system",
                    "content": "You are a helpful assistant meant to suggest places to visit in a city",
                },
                {"role": "user", "content": city},
            ],
            max_tokens=self.max_tokens,
            temperature=self.temperature,
            seed=self.seed,
        ).choices[0].message.content


city_visit_recommender = CityVisitRecommender(
    model="meta-llama/Llama-3.2-11B-Vision-Instruct",
    temperature=0.7,
    max_tokens=500,
    seed=42,
)
rich.print(city_visit_recommender.predict("New York City"))
rich.print(city_visit_recommender.predict("Paris"))
```

Weave automatically logs models and tracks different versions so you can analyze performance and experiment history.

<Frame>
  <img alt="Weave automatically logs models and tracks different versions so you can analyze performance and experiment history." />
</Frame>


# TypeScript SDK: third-party integration guide
Source: https://docs.wandb.ai/weave/guides/integrations/js

Integrate third-party libraries with the Weave TypeScript SDK

This guide describes how to integrate third-party libraries (for example, OpenAI) with the Weave TypeScript SDK. It's intended for TypeScript developers who want Weave to automatically trace calls to supported libraries in their application.

Weave supports automatic instrumentation, which streamlines setup and reduces the need for manual configuration.

<Warning>
  **What changed?**
  As of [PR #4554](https://github.com/wandb/weave/pull/4554), Weave automatically patches supported libraries such as OpenAI when it loads. You no longer need to manually wrap them:

  ```ts lines theme={null}
  weave.wrapOpenAI(new OpenAI());
  ```

  Weave handles this automatically in most cases. However, [edge cases](#advanced-usage) may occur.
</Warning>

## Use Weave integrations with your TypeScript project

The following sections describe how to identify which module system your project uses and then configure that project so Weave can automatically instrument supported third-party libraries. TypeScript projects can use either the CommonJS or ESM module system, and the required setup differs slightly between the two.

### If you're unsure which type of project you have

If you run a TypeScript file directly with a tool such as:

```bash theme={null}
npx tsx test.ts
```

your environment may determine the module system implicitly. For consistent behavior, explicitly define `package.json` and `tsconfig.json` files.

To determine whether a project uses CommonJS or ESM, check the `type` field in `package.json`:

```json theme={null}
"type": "module"
```

* If `type` is `"module"`, the project uses ESM.
* If the `type` field is missing or set to `"commonjs"`, the project defaults to using CommonJS.

### Set up a CommonJS project

For CommonJS projects, automatic instrumentation works without additional configuration.

To configure your project for CommonJS:

1. Create or update your `package.json`:

   ```json theme={null}
   {
     "type": "commonjs"
   }
   ```

2. Create a `tsconfig.json` with CommonJS-compatible settings:

   ```json theme={null}
   {
     "compilerOptions": {
       "module": "CommonJS",
       "target": "es2022",
       "rootDir": ".",
       "outDir": "dist"
     }
   }
   ```

   These settings configure TypeScript to compile for CommonJS:

   * `module: "CommonJS"`. Compiles modules to CommonJS format (`require` or `module.exports`).
     For details on this compiler option, see [TypeScript - Module](https://www.typescriptlang.org/tsconfig/#module).

   * `target: "es2022"` (recommended). Emits modern JavaScript compatible with recent Node.js versions.
     For details on this compiler option, see [TypeScript - Target](https://www.typescriptlang.org/tsconfig/#target).

   * `rootDir: "."`. Treats the directory that contains `tsconfig.json` as the root of your input files. TypeScript uses this with `outDir` to mirror your source folder layout in the output.
     For details on this compiler option, see [TypeScript - Root Dir](https://www.typescriptlang.org/tsconfig/#rootDir).

   * `outDir: "dist"`. Writes emitted JavaScript and other compiler outputs into the `dist` folder.
     For details on this compiler option, see [TypeScript - Out Dir](https://www.typescriptlang.org/tsconfig/#outDir).

3. Install Weave and any other required libraries:

   ```bash theme={null}
   npm install weave
   ```

4. Compile your TypeScript file.

   For an example file `test.ts`:

   ```bash theme={null}
   npx tsc
   ```

   This compiles the file to `dist/test.js`.

5. Run the compiled file with Node.js:

   ```bash theme={null}
   node dist/test.js
   ```

Your CommonJS project is now configured so that Weave automatically instruments supported libraries when your code runs. Because CommonJS uses the Node.js `require` module loader, Weave can automatically instrument supported libraries without the `--import` flag used in ESM projects.

### Set up an ESM project

To use Weave with an ESM TypeScript project, configure your project for Node.js ESM, compile your code, and start Node.js with the `--import` flag so Weave can register its instrumentation before other modules load.

To configure your project for ESM:

1. Create or update your `package.json`:

   ```json theme={null}
   {
     "type": "module"
   }
   ```

2. Create a `tsconfig.json` with Node-compatible ESM settings:

   ```json theme={null}
   {
     "compilerOptions": {
       "module": "nodenext",
       "moduleResolution": "nodenext",
       "target": "es2022",
       "rootDir": ".",
       "outDir": "dist"
     }
   }
   ```

   These settings configure TypeScript to compile for modern Node.js ESM:

   * `module: "nodenext"`. Compiles modules using Node.js ESM semantics.\
     For details on this compiler option, see [TypeScript - Module](https://www.typescriptlang.org/tsconfig/#module).

   * `moduleResolution: "nodenext"`. Ensures module resolution follows Node.js ESM rules.\
     For details on this compiler option, see [TypeScript - Module Resolution](https://www.typescriptlang.org/tsconfig/#moduleResolution).

   * `target: "es2022"` (recommended). Emits modern JavaScript compatible with recent Node.js versions.\
     For details on this compiler option, see [TypeScript - Target](https://www.typescriptlang.org/tsconfig/#target).

   * `rootDir: "."`. Treats the directory that contains `tsconfig.json` as the root of your input files. TypeScript uses this with `outDir` to mirror your source folder layout in the output.\
     For details on this compiler option, see [TypeScript - Root Dir](https://www.typescriptlang.org/tsconfig/#rootDir).

   * `outDir: "dist"`. Writes emitted JavaScript and other compiler outputs into the `dist` folder.\
     For details on this compiler option, see [TypeScript - Out Dir](https://www.typescriptlang.org/tsconfig/#outDir).

3. Install Weave and any other required libraries:

   ```bash theme={null}
   npm install weave
   ```

4. Compile your TypeScript file.

   For an example file `test.ts`:

   ```bash theme={null}
   npx tsc
   ```

   This compiles the file to `dist/test.js`.

5. Run the compiled file with Node.js and preload the Weave instrumentation:

   ```bash theme={null}
   node --import=weave/instrument dist/test.js
   ```

The `--import` flag ensures that the `weave/instrument` module loads before other modules, so Weave can automatically instrument supported libraries and integrations.

Weave must be installed locally in the project you run.

Your ESM project is now configured so that Weave preloads its instrumentation before other modules and automatically traces calls to supported libraries.

## Advanced usage and troubleshooting

The following sections cover edge cases and workarounds for when the TypeScript SDK's automatic patching doesn't work as expected. For example, ESM-only environments, bundler setups such as Next.js, or constrained runtime environments may cause issues. If you see missing traces or integration issues, start here.

### Use `NODE_OPTIONS` (only for ESM)

<Warning>
  Use `NODE_OPTIONS` with caution, because it affects all Node.js processes in the environment and may introduce side effects.
</Warning>

If you use an ESM project and can't pass CLI flags (for example, due to constraints in CLI tools or frameworks), set the `NODE_OPTIONS` environment variable:

```bash theme={null}
export NODE_OPTIONS="--import=weave/instrument"
```

### Bundler compatibility

Some frameworks and bundlers, such as Next.js, may bundle third-party libraries in ways that prevent Node.js from patching them at runtime.

If this describes your setup, try the following steps:

1. Mark LLM libraries as external in your bundler configuration. This prevents the bundler from bundling them, so Weave can patch them correctly at runtime.

   The following example shows how to mark the `openai` package as external in a `next.config.js` configuration, which prevents the bundler from bundling it. The module loads at runtime, so Weave can automatically patch and track it. Use this setup when you work with frameworks such as Next.js to enable auto-instrumentation.

   ```js theme={null}
   externals: {
   'openai': 'commonjs openai'
   }
   ```

2. If patching still fails, fall back to [manual instrumentation](#manual-patching-fallback-option).

### Manual patching (fallback option)

<Warning>
  Manual patching is the legacy approach. Use it only when auto-patching doesn't work.
</Warning>

Sometimes, you may still need to use manual instrumentation:

```ts lines theme={null}
import { wrapOpenAI } from 'weave';
const client = wrapOpenAI(new OpenAI());
```


# LiteLLM
Source: https://docs.wandb.ai/weave/guides/integrations/litellm

Automatically track and log LLM calls made through LiteLLM

<a href="https://colab.research.google.com/github/wandb/examples/blob/master/weave/docs/quickstart_litellm.ipynb" aria-label="Open in Google Colab">
  <img alt="Open In Colab" />
</a>

Weave automatically tracks and logs LLM calls made through LiteLLM after you call `weave.init()`. This guide explains how to use Weave with LiteLLM to capture traces, wrap calls in versioned ops, organize experiments with a `Model`, and track function-calling behavior. Use it when you're building LLM applications and want observability across the multiple model providers that LiteLLM supports.

## Traces

It's important to store traces of LLM applications in a central database, both during development and in production. You use these traces for debugging and as a dataset that helps you improve your application.

> **Note:** When using LiteLLM, make sure to import the library using `import litellm` and call the completion function with `litellm.completion()` instead of `from litellm import completion`. This ensures that all functions and parameters are correctly referenced.

Weave automatically captures traces for LiteLLM. Use the library as usual, and start by calling `weave.init()`:

```python lines {4} theme={null}
import litellm
import weave

weave.init("weave_litellm_integration")

openai_response = litellm.completion(
    model="gpt-3.5-turbo", 
    messages=[{"role": "user", "content": "Translate 'Hello, how are you?' to French"}],
    max_tokens=1024
)
print(openai_response.choices[0].message.content)

claude_response = litellm.completion(
    model="claude-3-5-sonnet-20240620", 
    messages=[{"role": "user", "content": "Translate 'Hello, how are you?' to French"}],
    max_tokens=1024
)
print(claude_response.choices[0].message.content)
```

Weave now tracks and logs all LLM calls made through LiteLLM. You can view the traces in the Weave web interface. With basic tracing in place, the next section shows how to wrap LiteLLM calls in your own Weave ops to gain finer-grained, versioned tracking of your application logic.

## Wrap with your own ops

Weave ops make results reproducible by automatically versioning code as you experiment, and they capture their inputs and outputs. Create a function decorated with `@weave.op()` that calls into LiteLLM's completion function, and Weave tracks the inputs and outputs for you. Here's an example:

```python lines {4,6} theme={null}
import litellm
import weave

weave.init("weave_litellm_integration")

@weave.op()
def translate(text: str, target_language: str, model: str) -> str:
    response = litellm.completion(
        model=model,
        messages=[{"role": "user", "content": f"Translate '{text}' to {target_language}"}],
        max_tokens=1024
    )
    return response.choices[0].message.content

print(translate("Hello, how are you?", "French", "gpt-3.5-turbo"))
print(translate("Hello, how are you?", "Spanish", "claude-3-5-sonnet-20240620"))
```

## Create a `Model` for easier experimentation

Organizing experimentation is difficult when many pieces move at once. By using the `Model` class, you can capture and organize the experimental details of your app, such as your system prompt or the model you're using. This helps you organize and compare different iterations of your app.

In addition to versioning code and capturing inputs and outputs, Models capture structured parameters that control your application's behavior, helping you find which parameters work best. You can also use Weave Models with `serve` and Evaluations.

In the following example, you can experiment with different models and temperatures:

```python lines {4,6,10} theme={null}
import litellm
import weave

weave.init('weave_litellm_integration')

class TranslatorModel(weave.Model):
    model: str
    temperature: float
  
    @weave.op()
    def predict(self, text: str, target_language: str):
        response = litellm.completion(
            model=self.model,
            messages=[
                {"role": "system", "content": f"You are a translator. Translate the given text to {target_language}."},
                {"role": "user", "content": text}
            ],
            max_tokens=1024,
            temperature=self.temperature
        )
        return response.choices[0].message.content

# Create instances with different models
gpt_translator = TranslatorModel(model="gpt-3.5-turbo", temperature=0.3)
claude_translator = TranslatorModel(model="claude-3-5-sonnet-20240620", temperature=0.1)

# Use different models for translation
english_text = "Hello, how are you today?"

print("GPT-3.5 Translation to French:")
print(gpt_translator.predict(english_text, "French"))

print("\nClaude-3.5 Sonnet Translation to Spanish:")
print(claude_translator.predict(english_text, "Spanish"))
```

## Function calling

LiteLLM supports function calling for compatible models. Weave automatically tracks these function calls so you can inspect the functions, arguments, and responses alongside your other traces.

```python lines {4} theme={null}
import litellm
import weave

weave.init("weave_litellm_integration")

response = litellm.completion(
    model="gpt-3.5-turbo",
    messages=[{"role": "user", "content": "Translate 'Hello, how are you?' to French"}],
    functions=[
        {
            "name": "translate",
            "description": "Translate text to a specified language",
            "parameters": {
                "type": "object",
                "properties": {
                    "text": {
                        "type": "string",
                        "description": "The text to translate",
                    },
                    "target_language": {
                        "type": "string",
                        "description": "The language to translate to",
                    }
                },
                "required": ["text", "target_language"],
            },
        },
    ],
)

print(response)
```

Weave automatically captures the functions you use in the prompt and keeps them versioned.

[<img alt="litellm.gif" />](https://wandb.ai/a-sh0ts/weave_litellm_integration/weave/calls)


# MistralAI
Source: https://docs.wandb.ai/weave/guides/integrations/mistral

Track and monitor MistralAI model calls with Weave's automatic tracing for chat completions, function calling, and model interactions.

<a href="https://colab.research.google.com/github/wandb/examples/blob/master/weave/docs/quickstart_mistral.ipynb" aria-label="Open in Google Colab">
  <img alt="Open In Colab" />
</a>

Weave automatically tracks and logs LLM calls made through the [MistralAI Python library](https://github.com/mistralai/client-python). This guide shows you how to capture traces for MistralAI calls, wrap calls in your own ops for reproducibility, and organize experiments using the `Model` class.

> Weave supports the Mistral v1.0 SDK. For details on upgrading, see the [migration guide](https://github.com/mistralai/client-python/blob/main/MIGRATION.md).

## Traces

It's important to store traces of LLM applications in a central database, both during development and in production. You use these traces for debugging, and as a dataset that helps you improve your application.

Weave automatically captures traces for [mistralai](https://github.com/mistralai/client-python). To use the library as usual, start by calling `weave.init()`:

```python lines theme={null}
import weave
weave.init("cheese_recommender")

# then use mistralai library as usual
import os
from mistralai import Mistral

api_key = os.environ["MISTRAL_API_KEY"]
model = "mistral-large-latest"

client = Mistral(api_key=api_key)

messages = [
    {
        "role": "user",
        "content": "What is the best French cheese?",
    },
]

chat_response = client.chat.complete(
    model=model,
    messages=messages,
)
```

Weave now tracks and logs all LLM calls made through the MistralAI library. You can view the traces in the Weave web interface.

[<img alt="mistral_trace.png" />](https://wandb.ai/capecape/mistralai_project/weave/calls)

## Wrap calls with your own ops

Once basic tracing works, you can wrap your MistralAI calls in Weave Ops to make them reproducible across experiments. Weave ops make results *reproducible* by automatically versioning code as you experiment, and they capture their inputs and outputs. Create a function decorated with [`@weave.op()`](/weave/guides/tracking/ops) that calls into [`mistralai.client.MistralClient.chat()`](https://docs.mistral.ai/capabilities/completion), and Weave tracks the inputs and outputs for you. The following example shows how to do this for a cheese recommender:

```python lines {1} theme={null}
@weave.op()
def cheese_recommender(region:str, model:str) -> str:
    "Recommend the best cheese in a given region"
    
    messages = [
        {
            "role": "user",
            "content": f"What is the best cheese in {region}?",
        },
    ]

    chat_response = client.chat.complete(
        model=model,
        messages=messages,
    )
    return chat_response.choices[0].message.content

cheese_recommender(region="France", model="mistral-large-latest")
cheese_recommender(region="Spain", model="mistral-large-latest")
cheese_recommender(region="Netherlands", model="mistral-large-latest")
```

[<img alt="mistral_ops.png" />](https://wandb.ai/capecape/mistralai_project/weave/calls)

## Create a `Model` for easier experimentation

Organizing experimentation is difficult with many moving pieces. By using the [`Model`](/weave/guides/core-types/models) class, you can capture and organize the experimental details of your app like your system prompt or the model you're using. This helps organize and compare different iterations of your app.

Besides versioning code and capturing inputs and outputs, [`Model`](/weave/guides/core-types/models)s capture structured parameters that control your application's behavior, making it easy to find what parameters worked best. You can also use Weave Models with `serve`, and [`Evaluation`](/weave/guides/core-types/evaluations)s.

In the following example, you can experiment with `model` and `country`. Every time you change one of these, you get a new *version* of `CheeseRecommender`.

```python lines theme={null}
import weave
from mistralai import Mistral

weave.init("mistralai_project")

class CheeseRecommender(weave.Model): # Change to `weave.Model`
    model: str
    temperature: float

    @weave.op()
    def predict(self, region:str) -> str: # Change to `predict`
        "Recommend the best cheese in a given region"
        
        client = Mistral(api_key=api_key)

        messages = [
            {
                "role": "user",
                "content": f"What is the best cheese in {region}?",
            },
        ]

        chat_response = client.chat.complete(
            model=model,
            messages=messages,
            temperature=self.temperature
        )
        return chat_response.choices[0].message.content

cheese_model = CheeseRecommender(
    model="mistral-medium-latest",
    temperature=0.0
    )
result = cheese_model.predict(region="France")
print(result)
```

[<img alt="mistral_model.png" />](https://wandb.ai/capecape/mistralai_project/weave/models)


# NVIDIA NIM
Source: https://docs.wandb.ai/weave/guides/integrations/nvidia_nim

Use Weave to trace and log LLM calls made through the ChatNVIDIA library

Weave automatically tracks and logs LLM calls made through the [ChatNVIDIA](https://python.langchain.com/docs/integrations/chat/nvidia_ai_endpoints/) library after you call `weave.init()`. This guide shows Python developers using ChatNVIDIA how to capture traces, wrap their own functions as Ops, and organize experiments with Weave's `Model` class so they can debug, iterate on, and compare LLM applications more efficiently.

<Tip>
  For the latest tutorials, visit [Weights & Biases on NVIDIA](https://wandb.ai/site/partners/nvidia).
</Tip>

## Tracing

Storing traces of LLM applications in a central database, both during development and in production, helps you debug issues and build a dataset of tricky examples to evaluate against while you improve your application. The following section shows how to enable automatic tracing for ChatNVIDIA calls.

<Tabs>
  <Tab title="Python">
    Weave can automatically capture traces for the [ChatNVIDIA Python library](https://python.langchain.com/docs/integrations/chat/nvidia_ai_endpoints/).

    Start capturing by calling `weave.init([PROJECT-NAME])` with a project name of your choice.

    ```python lines {4} theme={null}
    from langchain_nvidia_ai_endpoints import ChatNVIDIA
    import weave
    client = ChatNVIDIA(model="mistralai/mixtral-8x7b-instruct-v0.1", temperature=0.8, max_tokens=64, top_p=1)
    weave.init('emoji-bot')

    messages=[
        {
          "role": "system",
          "content": "You are AGI. You will be provided with a message, and your task is to respond using emojis only."
        }]

    response = client.invoke(messages)
    ```

    After you run this code, Weave captures the ChatNVIDIA call under the project name you specified, where you can inspect the inputs, outputs, and metadata.
  </Tab>

  <Tab title="TypeScript">
    ```plaintext theme={null}
    This feature is not available in TypeScript yet since this library is only in Python.
    ```
  </Tab>
</Tabs>

<Frame>
  <img alt="chatnvidia_trace.png" />
</Frame>

## Track your own ops

<Tabs>
  <Tab title="Python">
    Wrapping a function with `@weave.op` starts capturing inputs, outputs, and app logic so you can debug how data flows through your app. You can deeply nest ops and build a tree of functions that you want to track. This also starts automatic code versioning as you experiment to capture ad-hoc details that you haven't committed to Git.

    Create a function decorated with [`@weave.op`](/weave/guides/tracking/ops) that calls into [ChatNVIDIA Python library](https://python.langchain.com/docs/integrations/chat/nvidia_ai_endpoints/).

    In the following example, two functions are wrapped with op. This shows you how intermediate steps, like the retrieval step in a RAG app, affect how your app behaves.

    ```python lines {1,9,11,29,31,33} theme={null}
    import weave
    from langchain_nvidia_ai_endpoints import ChatNVIDIA
    import requests, random
    PROMPT="""Emulate the Pokedex from early Pokémon episodes. State the name of the Pokemon and then describe it.
            Your tone is informative yet sassy, blending factual details with a touch of dry humor. Be concise, no more than 3 sentences. """
    POKEMON = ['pikachu', 'charmander', 'squirtle', 'bulbasaur', 'jigglypuff', 'meowth', 'eevee']
    client = ChatNVIDIA(model="mistralai/mixtral-8x7b-instruct-v0.1", temperature=0.7, max_tokens=100, top_p=1)

    @weave.op
    def get_pokemon_data(pokemon_name):
        # This is a step within your application, like the retrieval step within a RAG app
        url = f"https://pokeapi.co/api/v2/pokemon/{pokemon_name}"
        response = requests.get(url)
        if response.status_code == 200:
            data = response.json()
            name = data["name"]
            types = [t["type"]["name"] for t in data["types"]]
            species_url = data["species"]["url"]
            species_response = requests.get(species_url)
            evolved_from = "Unknown"
            if species_response.status_code == 200:
                species_data = species_response.json()
                if species_data["evolves_from_species"]:
                    evolved_from = species_data["evolves_from_species"]["name"]
            return {"name": name, "types": types, "evolved_from": evolved_from}
        else:
            return None

    @weave.op
    def pokedex(name: str, prompt: str) -> str:
        # This is your root op that calls out to other ops
        data = get_pokemon_data(name)
        if not data: return "Error: Unable to fetch data"

        messages=[
                {"role": "system","content": prompt},
                {"role": "user", "content": str(data)}
            ]

        response = client.invoke(messages)
        return response.content

    weave.init('pokedex-nvidia')
    # Get data for a specific Pokémon
    pokemon_data = pokedex(random.choice(POKEMON), PROMPT)
    ```

    Navigate to Weave and click `get_pokemon_data` in the UI to see the inputs and outputs of that step.
  </Tab>

  <Tab title="TypeScript">
    ```plaintext theme={null}
    This feature is not available in TypeScript yet since this library is only in Python.
    ```
  </Tab>
</Tabs>

<Frame>
  <img alt="nvidia_pokedex.png" />
</Frame>

## Create a `Model` for easier experimentation

<Tabs>
  <Tab title="Python">
    Organizing experimentation is difficult when there are many moving pieces. By using the [`Model`](/weave/guides/core-types/models) class, you can capture and organize the experimental details of your app like your system prompt or the model you're using. This helps you organize and compare different iterations of your app.

    In addition to versioning code and capturing inputs and outputs, [`Model`](/weave/guides/core-types/models)s capture structured parameters that control your application's behavior, making it straightforward to find what parameters worked best. You can also use Weave Models with `serve`, and [`Evaluation`](/weave/guides/core-types/evaluations)s.

    In the following example, you can experiment with `model` and `system_message`. Every time you change one of these, you get a new *version* of `GrammarCorrectorModel`.

    ```python lines theme={null}
    import weave
    from langchain_nvidia_ai_endpoints import ChatNVIDIA

    weave.init('grammar-nvidia')

    class GrammarCorrectorModel(weave.Model): # Change to `weave.Model`
      system_message: str

      @weave.op()
      def predict(self, user_input): # Change to `predict`
        client = ChatNVIDIA(model="mistralai/mixtral-8x7b-instruct-v0.1", temperature=0, max_tokens=100, top_p=1)

        messages=[
              {
                  "role": "system",
                  "content": self.system_message
              },
              {
                  "role": "user",
                  "content": user_input
              }
              ]

        response = client.invoke(messages)
        return response.content

    corrector = GrammarCorrectorModel(
        system_message = "You are a grammar checker, correct the following user input.")
    result = corrector.predict("That was so easy, it was a piece of pie!")
    print(result)
    ```
  </Tab>

  <Tab title="TypeScript">
    ```plaintext theme={null}
    This feature is not available in TypeScript yet since this library is only in Python.
    ```
  </Tab>
</Tabs>

<Frame>
  <img alt="chatnvidia_model.png" />
</Frame>

## Usage info

The following notes describe what the ChatNVIDIA integration supports.

The ChatNVIDIA integration supports `invoke`, `stream`, and their async variants. It also supports tool use.
Because ChatNVIDIA is meant for use with many types of models, it doesn't have function calling support.


# OpenAI
Source: https://docs.wandb.ai/weave/guides/integrations/openai

Integrate OpenAI with Weave for tracing, evaluation, and monitoring

<a href="https://colab.research.google.com/github/wandb/examples/blob/master/weave/docs/quickstart_openai.ipynb" aria-label="Open in Google Colab">
  <img alt="Open In Colab" />
</a>

This guide shows you how to integrate the OpenAI Python and TypeScript libraries with Weave so that you can trace, evaluate, and monitor your LLM application. It's intended for developers who already use OpenAI's SDKs and want visibility into their calls during development and in production.

<Note>
  Experiment with OpenAI models on Weave without any setup using the [LLM Playground](../tools/playground).
</Note>

## Tracing

Storing traces of LLM applications in a central database is valuable both during development and in production. Use these traces for debugging and to help build a dataset of tricky examples to evaluate against while you improve your application.

Weave can automatically capture traces for the [`openai` Python library](https://developers.openai.com/api/docs/libraries).

To start capturing, call `weave.init("[PROJECT_NAME]")` with a project name of your choice. Weave automatically patches OpenAI regardless of when you import it, so all subsequent OpenAI calls are traced.

If you don't specify a W\&B team when you call `weave.init()`, Weave uses your default entity. To find or update your default entity, refer to [User Settings](https://docs.wandb.ai/platform/app/settings-page/user-settings/#default-team) in the W\&B Models documentation.

### Automatic patching

Weave automatically patches OpenAI whether you import it before or after `weave.init()`. The following example shows the minimal setup you need to start tracing calls:

<CodeGroup>
  ```python Python lines {4} theme={null}
  from openai import OpenAI
  import weave

  weave.init('emoji-bot')  # OpenAI is automatically patched!

  client = OpenAI()
  response = client.chat.completions.create(
    model="gpt-4",
    messages=[
      {
        "role": "system",
        "content": "You are AGI. You will be provided with a message, and your task is to respond using emojis only."
      },
      {
        "role": "user",
        "content": "How are you?"
      }
    ]
  )
  ```

  ```typescript TypeScript theme={null}
  import { OpenAI } from 'openai';
  import { wrapOpenAI } from '@wandb/weave';

  const openai = wrapOpenAI(new OpenAI());

  // This will now trace all calls to OpenAI
  openai.chat.completions.create(
    {
      model: "gpt-4",
      messages: [
        {
          role: "system",
          content: "You are AGI. You will be provided with a message, and your task is to respond using emojis only."
        },
        {
          role: "user",
          content: "How are you?"
        }
      ]
    }
  );
  ```
</CodeGroup>

### Optional: Explicit patching

For fine-grained control over when patching takes effect, patch OpenAI explicitly instead of relying on the automatic behavior:

```python lines {3,4} theme={null}
import weave

weave.init('emoji-bot')
weave.integrations.patch_openai()  # Enable OpenAI tracing

from openai import OpenAI
client = OpenAI()
response = client.chat.completions.create(
  model="gpt-4",
  messages=[
    {"role": "user", "content": "Make me a emoji"}
  ]
)
```

[View a live trace](https://wandb.ai/capecape/emoji-bot/weave/calls/01928a78-6d8a-7e20-9b8c-0cbc8318a0c8)

<Tip>
  Weave also captures the function calling tools for [OpenAI Functions](https://platform.openai.com/docs/guides/function-calling) and [OpenAI Assistants](https://platform.openai.com/docs/assistants/overview).
</Tip>

## Structured outputs

Weave supports tracing OpenAI structured outputs, which are useful when you need to ensure your LLM responses follow a specific format. The following example traces a call that extracts a typed `UserDetail` object from a user message:

<CodeGroup>
  ```python Python theme={null}
  from openai import OpenAI
  from pydantic import BaseModel
  import weave

  class UserDetail(BaseModel):
      name: str
      age: int

  client = OpenAI()
  weave.init('extract-user-details')

  completion = client.beta.chat.completions.parse(
      model="gpt-4o-2024-08-06",
      messages=[
          {"role": "system", "content": "Extract the user details from the message."},
          {"role": "user", "content": "My name is David and I am 30 years old."},
      ],
      response_format=UserDetail,
  )

  user_detail = completion.choices[0].message.parsed
  print(user_detail)
  ```
</CodeGroup>

## Async support

Weave supports tracing async OpenAI calls, so applications that use `AsyncOpenAI` get the same visibility as synchronous applications.

<CodeGroup>
  ```python Python theme={null}
  from openai import AsyncOpenAI
  import weave

  client = AsyncOpenAI()
  weave.init('async-emoji-bot')

  async def call_openai():
      response = await client.chat.completions.create(
          model="gpt-4",
          messages=[
              {
                  "role": "system", 
                  "content": "You are AGI. You will be provided with a message, and your task is to respond using emojis only."
              },
              {
                  "role": "user",
                  "content": "How are you?"
              }
          ]
      )
      return response

  # Call the async function
  result = await call_openai()
  ```
</CodeGroup>

## Streaming support

Weave supports tracing streaming responses from OpenAI. The captured trace reflects the full streamed completion, so you can review the final output alongside the request parameters.

<CodeGroup>
  ```python Python theme={null}
  from openai import OpenAI
  import weave

  client = OpenAI()
  weave.init('streaming-emoji-bot')

  response = client.chat.completions.create(
      model="gpt-4",
      messages=[
          {
              "role": "system", 
              "content": "You are AGI. You will be provided with a message, and your task is to respond using emojis only."
          },
          {
              "role": "user",
              "content": "How are you?"
          }
      ],
      stream=True
  )

  for chunk in response:
      print(chunk.choices[0].delta.content or "", end="")
  ```
</CodeGroup>

## Tracing function calls

Weave traces function calls made by OpenAI when you use tools, which helps you understand how the model invoked each tool and with what arguments.

<CodeGroup>
  ```python Python theme={null}
  from openai import OpenAI
  import weave

  client = OpenAI()
  weave.init('function-calling-bot')

  tools = [
      {
          "type": "function",
          "function": {
              "name": "get_weather",
              "description": "Get the weather in a given location",
              "parameters": {
                  "type": "object",
                  "properties": {
                      "location": {
                          "type": "string",
                          "description": "The location to get the weather for"
                      },
                      "unit": {
                          "type": "string",
                          "enum": ["celsius", "fahrenheit"],
                          "description": "The unit to return the temperature in"
                      }
                  },
                  "required": ["location"]
              }
          }
      }
  ]

  response = client.chat.completions.create(
      model="gpt-4",
      messages=[
          {
              "role": "user",
              "content": "What's the weather like in New York?"
          }
      ],
      tools=tools
  )

  print(response.choices[0].message.tool_calls)
  ```
</CodeGroup>

## Batch API

Weave supports the OpenAI Batch API, which lets you process multiple requests asynchronously while Weave still captures each request in your traces.

<CodeGroup>
  ```python Python theme={null}
  from openai import OpenAI
  import weave

  client = OpenAI()
  weave.init('batch-processing')

  # Create a batch file
  batch_input = [
      {
          "custom_id": "request-1",
          "method": "POST",
          "url": "/v1/chat/completions",
          "body": {
              "model": "gpt-4",
              "messages": [{"role": "user", "content": "Hello, how are you?"}]
          }
      },
      {
          "custom_id": "request-2", 
          "method": "POST",
          "url": "/v1/chat/completions",
          "body": {
              "model": "gpt-4",
              "messages": [{"role": "user", "content": "What's the weather like?"}]
          }
      }
  ]

  # Submit the batch
  batch = client.batches.create(
      input_file_id="your-file-id",
      endpoint="/v1/chat/completions",
      completion_window="24h"
  )

  # Retrieve the batch results
  completed_batch = client.batches.retrieve(batch.id)
  ```
</CodeGroup>

## Assistants API

Weave supports the OpenAI Assistants API, so you can trace conversational AI applications built around assistants, threads, and runs.

<CodeGroup>
  ```python Python theme={null}
  from openai import OpenAI
  import weave

  client = OpenAI()
  weave.init('assistant-bot')

  # Create an assistant
  assistant = client.beta.assistants.create(
      name="Math Assistant",
      instructions="You are a personal math tutor. Answer questions about math.",
      model="gpt-4"
  )

  # Create a thread
  thread = client.beta.threads.create()

  # Add a message to the thread
  message = client.beta.threads.messages.create(
      thread_id=thread.id,
      role="user",
      content="What is 2+2?"
  )

  # Run the assistant
  run = client.beta.threads.runs.create(
      thread_id=thread.id,
      assistant_id=assistant.id
  )

  # Get the assistant's response
  messages = client.beta.threads.messages.list(thread_id=thread.id)
  ```
</CodeGroup>

## Cost tracking

Weave automatically tracks the cost of your OpenAI API calls so that you can monitor spend alongside performance. You can view the cost breakdown in the Weave UI.

<Note>
  Cost tracking is available for all OpenAI models, and Weave calculates costs based on OpenAI's published pricing.
</Note>

## Tracing custom functions

To group OpenAI calls under your own application logic, trace custom functions that use OpenAI by applying the `@weave.op` decorator. This produces a parent trace for the function with the underlying OpenAI calls nested inside it.

<CodeGroup>
  ```python Python theme={null}
  from openai import OpenAI
  import weave

  client = OpenAI()
  weave.init('custom-function-bot')

  @weave.op
  def generate_response(prompt: str) -> str:
      response = client.chat.completions.create(
          model="gpt-4",
          messages=[
              {
                  "role": "user",
                  "content": prompt
              }
          ]
      )
      return response.choices[0].message.content

  # This function call will be traced
  result = generate_response("Hello, how are you?")
  ```
</CodeGroup>

## Next steps

With tracing set up for OpenAI, your application's calls are now visible in Weave. From here, you can:

* **View traces in the Weave UI**: Go to your Weave project to see traces of your OpenAI calls.
* **Create evaluations**: Use your traces to build evaluation datasets.
* **Monitor performance**: Track latency, costs, and other metrics.
* **Debug issues**: Use traces to understand what's happening in your LLM application.

For more information about these topics, see the [evaluation guide](../evaluation/scorers) and [monitoring guide](../tracking).


# OpenAI Realtime API
Source: https://docs.wandb.ai/weave/guides/integrations/openai-realtime-audio

Use Weave to automatically trace your calls to the OpenAI Realtime API.

Weave's integration with [OpenAI's Realtime API](https://developers.openai.com/api/docs/guides/realtime/) lets you automatically trace your application's speech-to-speech interactions as they occur. You can use this to capture conversations between your agents and users to review and evaluate agent performance.

This guide shows you how to enable Realtime API tracing in your application and walks through two end-to-end voice assistant examples that you can run locally.

## Integrate realtime traces

This section shows the minimum code needed to enable Weave tracing for the OpenAI Realtime API in any application.

Weave automatically patches into the OpenAI Realtime API and only requires a few additional lines of code to start capturing audio interactions from your application. The following code imports Weave and the Realtime API integration:

```python lines {2,5} theme={null}
import weave
from weave.integrations import patch_openai_realtime

weave.init("your-team-name/your-project-name")
patch_openai_realtime()

# Your application's logic
```

Once imported into your code and executed, Weave automatically traces the interactions between the user and the OpenAI Realtime API.

### Run a realtime voice assistant with the OpenAI Agents SDK

This example runs a realtime voice assistant that streams microphone audio to OpenAI's Realtime API and plays back the AI's spoken responses through your local machine's speaker. The application uses the OpenAI Agents SDK with `RealtimeAgent` and `RealtimeRunner`, and enables Weave tracing by patching with `patch_openai_realtime()`. Use this approach when you want the higher-level Agents SDK to manage the Realtime session for you.

To run the example, complete the following steps:

1. Start your Python environment and install the following libraries:

   <Tabs>
     <Tab title="uv">
       ```bash theme={null}
       uv add weave openai-agents websockets pyaudio numpy
       ```
     </Tab>

     <Tab title="pip">
       ```bash theme={null}
       pip install weave openai-agents websockets pyaudio numpy
       ```
     </Tab>
   </Tabs>

2. Create a file titled `weave_voice_assistant.py` and add the following code to it.

   The highlighted lines indicate Weave's integration in the application. The rest of the code creates the basic voice assistant app.

   <Accordion title="weave_voice_assistant.py">
     ```python lines {8,11,14,51-55} theme={null}
     import argparse
     import asyncio
     import queue
     import sys
     import termios
     import threading
     import tty
     import weave
     import pyaudio
     import numpy as np
     from weave.integrations import patch_openai_realtime
     from agents.realtime import RealtimeAgent, RealtimeRunner

     DEFAULT_WEAVE_PROJECT = "<your-team-name/your-project-name>"

     FORMAT = pyaudio.paInt16
     RATE = 24000  # Required by the OpenAI Realtime API.
     CHUNK = 1024
     MAX_INPUT_CHANNELS = 1
     MAX_OUTPUT_CHANNELS = 1

     INP_DEV_IDX = None
     OUT_DEV_IDX = None

     # Parse CLI args for Weave project name and audio device selection.
     def parse_args():
         parser = argparse.ArgumentParser(description="Realtime agent with Weave logging")
         parser.add_argument(
             "--weave-project",
             default=DEFAULT_WEAVE_PROJECT,
             help=f"Weave project name (default: {DEFAULT_WEAVE_PROJECT})",
             dest="weave_project"
         )
         parser.add_argument(
             "--input-device",
             type=int,
             default=None,
             help="PyAudio input (mic) device index. Defaults to system default. Run mic_detect.py to list devices.",
             dest="input_device"
         )
         parser.add_argument(
             "--output-device",
             type=int,
             default=None,
             help="PyAudio output (speaker) device index. Defaults to system default. Run mic_detect.py to list devices.",
             dest="output_device"
         )
         return parser.parse_args()


     # Initialize Weave and patch the OpenAI Realtime API for tracing.
     def init_weave(project_name: str | None = None) -> None:
         name = project_name or DEFAULT_WEAVE_PROJECT
         weave.init(name)
         patch_openai_realtime()  # Enables automatic tracing of Realtime API sessions.


     mic_enabled = True

     # Listen for 't' key to toggle mic on/off. Runs in a daemon thread.
     def start_keylistener():
         global mic_enabled
         fd = sys.stdin.fileno()
         old_settings = termios.tcgetattr(fd)
         try:
             tty.setcbreak(fd)
             while True:
                 ch = sys.stdin.read(1)
                 if ch.lower() == 't':
                     mic_enabled = not mic_enabled
                     state = "ON" if mic_enabled else "OFF"
                     print(f"\n🎙  Mic {state} (press t to toggle)")
                 elif ch == '\x03':  # Ctrl-C
                     break
         finally:
             termios.tcsetattr(fd, termios.TCSADRAIN, old_settings)


     # Drain the audio queue and write to the speaker in a background thread.
     def play_audio(output_stream: pyaudio.Stream, audio_output_queue: queue.Queue):
         while True:
             data = audio_output_queue.get()
             if data is None:
                 break
             output_stream.write(data)


     # Open audio streams, start a Realtime session, and run the send/receive loop.
     async def main(*, input_device_index: int | None = None, output_device_index: int | None = None):
         p = pyaudio.PyAudio()

         if input_device_index is None:
             input_device_index = int(p.get_default_input_device_info()['index'])
         if output_device_index is None:
             output_device_index = int(p.get_default_output_device_info()['index'])

         # Clamp channel count to device capabilities to avoid pyaudio errors.
         input_info = p.get_device_info_by_index(input_device_index)
         output_info = p.get_device_info_by_index(output_device_index)

         input_channels = min(int(input_info['maxInputChannels']), MAX_INPUT_CHANNELS)
         output_channels = min(int(output_info['maxOutputChannels']), MAX_OUTPUT_CHANNELS)

         mic = p.open(
             format=FORMAT,
             channels=input_channels,
             rate=RATE,
             input=True,
             output=False,
             frames_per_buffer=CHUNK,
             input_device_index=input_device_index,
             start=False,
         )
         speaker = p.open(
             format=FORMAT,
             channels=output_channels,
             rate=RATE,
             input=False,
             output=True,
             frames_per_buffer=CHUNK,
             output_device_index=output_device_index,
             start=False,
         )
         mic.start_stream()
         speaker.start_stream()

         # Buffer audio through a queue so in-flight playback can be flushed on interrupt.
         audio_output_queue = queue.Queue()
         threading.Thread(
             target=play_audio, args=(speaker, audio_output_queue), daemon=True
         ).start()

         s_agent = RealtimeAgent(
             name="Speech Assistant",
             instructions="You are a tool using AI. Use tools to accomplish a task whenever possible"
         )

         s_runner = RealtimeRunner(s_agent, config={
             "model_settings": {
                 "model_name": "gpt-realtime",
                 "modalities": ["audio"],
                 "output_modalities": ["audio"],
                 "input_audio_format": "pcm16",
                 "output_audio_format": "pcm16",
                 "speed": 1.2,
                 "turn_detection": {
                     "prefix_padding_ms": 100,
                     "silence_duration_ms": 100,
                     "type": "server_vad",
                     "interrupt_response": True,
                     "create_response": True,
                 },
             }
         })
         print("--- Session Active (Speak into mic) ---")
         print("🎙  Mic ON (press t to toggle)")

         threading.Thread(target=start_keylistener, daemon=True).start()

         async with await s_runner.run() as session:
             # Stream mic input to the Realtime API, sending silence when muted.
             async def send_mic_audio():
                 silence = b'\x00' * CHUNK * 2  # 2 bytes per sample (16-bit PCM).
                 try:
                     while True:
                         raw_data = mic.read(CHUNK, exception_on_overflow=False)

                         if mic_enabled:
                             audio_data = np.frombuffer(raw_data, dtype=np.int16).astype(np.float64)
                             rms = np.sqrt(np.mean(audio_data**2))
                             meter = int(min(rms / 50, 50))
                             print(f"Mic Level: {'█' * meter}{' ' * (50-meter)} | 🎙 ON ", end="\r")
                             await session.send_audio(raw_data)
                         else:
                             print(f"Mic Level: {' ' * 50} | 🎙 OFF", end="\r")
                             await session.send_audio(silence)

                         await asyncio.sleep(0)  # Yield to the event loop between reads.
                 except Exception:
                     pass

             # Receive events from the session and route audio to the speaker.
             async def handle_events():
                 async for event in session:
                     if event.type == "audio":
                         audio_output_queue.put(event.audio.data)
                     elif event.type == "audio_interrupted":
                         # Flush queued AI audio so it doesn't talk over the user.
                         while not audio_output_queue.empty():
                             try:
                                 audio_output_queue.get_nowait()
                             except queue.Empty:
                                 break

             mic_task = asyncio.create_task(send_mic_audio())
             try:
                 await handle_events()
             finally:
                 mic_task.cancel()

         # Cleanup
         audio_output_queue.put(None)  # Signal playback thread to exit.
         mic.close()
         speaker.close()
         p.terminate()

     if __name__ == "__main__":
         args = parse_args()
         init_weave(args.weave_project)
         fd = sys.stdin.fileno()
         old_settings = termios.tcgetattr(fd)
         try:
             asyncio.run(main(input_device_index=args.input_device, output_device_index=args.output_device))
         finally:
             termios.tcsetattr(fd, termios.TCSADRAIN, old_settings)
     ```
   </Accordion>

3. Update the `DEFAULT_WEAVE_PROJECT` value with your team and project names.

4. Set your `OPENAI_API_KEY` environment variable.

5. Start the voice assistant:
   ```bash theme={null}
   python weave_voice_assistant.py
   ```

Once the assistant is running, press `T` on your keyboard to mute or unmute the mic. The assistant uses server-side voice activity detection to handle turn-taking and interruptions.

As you speak to the assistant, Weave captures traces that you can explore in the Weave UI, including the audio from the session.

### Run a realtime voice assistant with WebSockets

The following example connects directly to the OpenAI Realtime API over WebSockets. It streams microphone audio to the API, plays back spoken responses, and supports tool calling (weather lookups, math evaluation, code execution, and file writing). Weave traces the session using `weave.init()` and `patch_openai_realtime()`. Use this approach when you want full control over the Realtime session without depending on the Agents SDK.

To run the example, complete the following steps:

1. Start your Python environment and install the following libraries:

   <Tabs>
     <Tab title="uv">
       ```bash theme={null}
       uv add weave websockets pyaudio numpy
       ```
     </Tab>

     <Tab title="pip">
       ```bash theme={null}
       pip install weave websockets pyaudio numpy
       ```
     </Tab>
   </Tabs>

2. Create a file titled `tool_definitions.py` and add the following tool definitions to it. The main application imports from this module.

   <Accordion title="tool_definitions.py">
     ```python lines theme={null}
     import json
     import subprocess
     import tempfile
     from pathlib import Path
     import weave


     # @function_tool
     @weave.op
     def get_weather(city: str) -> str:
         """Get the current weather for a city.

         Args:
             city: The city name to get weather for.
         """
         return json.dumps({"city": city, "temperature": "72°F", "condition": "sunny"})


     @weave.op
     def calculate(expression: str) -> str:
         """Evaluate a math expression and return the result.

         Args:
             expression: A math expression to evaluate, e.g. '2 + 2'.
         """
         try:
             result = eval(expression)
             return str(result)
         except Exception as e:
             return f"Error: {e}"


     @weave.op
     def run_python_code(code: str) -> str:
         """Write and execute a Python script, returning its stdout/stderr.

         Args:
             code: The Python source code to execute.
         """
         with tempfile.NamedTemporaryFile(
             mode="w", suffix=".py", dir=tempfile.gettempdir(), delete=False
         ) as f:
             f.write(code)
             script_path = Path(f.name)

         try:
             result = subprocess.run(
                 ["python", str(script_path)],
                 capture_output=True,
                 text=True,
                 timeout=30,
             )
             output = result.stdout
             if result.stderr:
                 output += f"\nSTDERR:\n{result.stderr}"
             if result.returncode != 0:
                 output += f"\n(exit code {result.returncode})"
             return output or "(no output)"
         except subprocess.TimeoutExpired:
             return "Error: script timed out after 30 seconds."
         finally:
             script_path.unlink(missing_ok=True)


     @weave.op
     async def write_file(file_path: str, content: str) -> str:
         """Write content to a file on disk.

         Args:
             file_path: The path to write the file to.
             content: The content to write into the file.
         """
         try:
             path = Path(file_path)
             path.parent.mkdir(parents=True, exist_ok=True)
             path.write_text(content)
             return f"Wrote {len(content)} bytes to {file_path}"
         except Exception as e:
             return f"Error writing file: {e}"
     ```
   </Accordion>

3. Create a file titled `weave_ws_voice_assistant.py` in the same directory and add the following code to it.

   <Accordion title="weave_ws_voice_assistant.py">
     ```python theme={null}
     import asyncio
     import base64
     import json
     import os
     import queue
     import threading
     from typing import Any, Callable

     import numpy as np
     import pyaudio
     import websockets

     import weave
     weave.init("<your-team-name/your-project-name>")
     from weave.integrations import patch_openai_realtime
     patch_openai_realtime()

     from tool_definitions import (
         calculate,
         get_weather,
         run_python_code,
         write_file,
     )

     # Audio format (must be PCM16 for the Realtime API).
     FORMAT = pyaudio.paInt16
     RATE = 24000
     CHUNK = 1024
     MAX_INPUT_CHANNELS = 2
     MAX_OUTPUT_CHANNELS = 2

     OPENAI_API_KEY = os.environ.get("OPENAI_API_KEY", "")
     REALTIME_URL = "wss://api.openai.com/v1/realtime?model=gpt-realtime"

     DEBUG_WRITE_LOG = False

     # Map tool name -> callable for function call dispatch.
     TOOL_REGISTRY: dict[str, Callable[..., Any]] = {
         "get_weather": get_weather,
         "calculate": calculate,
         "run_python_code": run_python_code,
         "write_file": write_file,
     }

     # Raw tool definitions for the Realtime API session config.
     TOOL_DEFINITIONS = [
         {
             "type": "function",
             "name": "get_weather",
             "description": "Get the current weather for a city.",
             "parameters": {
                 "type": "object",
                 "properties": {
                     "city": {
                         "type": "string",
                         "description": "The city name to get weather for.",
                     }
                 },
                 "required": ["city"],
             },
         },
         {
             "type": "function",
             "name": "calculate",
             "description": "Evaluate a math expression and return the result.",
             "parameters": {
                 "type": "object",
                 "properties": {
                     "expression": {
                         "type": "string",
                         "description": "A math expression to evaluate, e.g. '2 + 2'.",
                     }
                 },
                 "required": ["expression"],
             },
         },
         {
             "type": "function",
             "name": "run_python_code",
             "description": "Write and execute a Python script, returning its stdout/stderr.",
             "parameters": {
                 "type": "object",
                 "properties": {
                     "code": {
                         "type": "string",
                         "description": "The Python source code to execute.",
                     }
                 },
                 "required": ["code"],
             },
         },
         {
             "type": "function",
             "name": "write_file",
             "description": "Write content to a file on disk.",
             "parameters": {
                 "type": "object",
                 "properties": {
                     "file_path": {
                         "type": "string",
                         "description": "The path to write the file to.",
                     },
                     "content": {
                         "type": "string",
                         "description": "The content to write into the file.",
                     },
                 },
                 "required": ["file_path", "content"],
             },
         },
     ]


     async def send_event(ws, event: dict) -> None:
         await ws.send(json.dumps(event))


     async def configure_session(ws) -> None:
         event = {
             "type": "session.update",
             "session": {
                 "type": "realtime",
                 "model": "gpt-realtime",
                 "output_modalities": ["audio"],
                 "instructions": (
                     "You are a helpful AI assistant with access to tools. "
                     "Use tools to accomplish tasks whenever possible. "
                     "Speak clearly and briefly."
                 ),
                 "tools": TOOL_DEFINITIONS,
                 "tool_choice": "auto",
                 "audio": {
                     "input": {
                         "format": {"type": "audio/pcm", "rate": 24000},
                         "transcription": {"model": "gpt-4o-transcribe"},
                         "turn_detection": {
                             "type": "server_vad",
                             "threshold": 0.5,
                             "prefix_padding_ms": 300,
                             "silence_duration_ms": 500,
                         },
                     },
                     "output": {
                         "format": {"type": "audio/pcm", "rate": 24000},
                     },
                 },
             },
         }
         await send_event(ws, event)
         print("Session configured.")


     async def handle_function_call(ws, call_id: str, name: str, arguments: str) -> None:
         if not name:
             raise Exception("Did not get a function name")

         print(f"\n[Function Call] {name}({arguments})")
         tool_fn = TOOL_REGISTRY.get(name)
         if tool_fn is None:
             result = json.dumps({"error": f"Unknown function: {name}"})
         else:
             try:
                 args = json.loads(arguments)
                 result = tool_fn(**args)
                 if asyncio.iscoroutine(result):
                     result = await result
             except Exception as e:
                 result = json.dumps({"error": str(e)})

         print(f"[Function Result] {result}")

         # Send the function call output back to the model.
         await send_event(ws, {
             "type": "conversation.item.create",
             "item": {
                 "type": "function_call_output",
                 "call_id": call_id,
                 "output": result if isinstance(result, str) else json.dumps(result),
             },
         })

         # Trigger a new response so the model incorporates the function result.
         await send_event(ws, {"type": "response.create"})


     def play_audio(output_stream: pyaudio.Stream, audio_output_queue: queue.Queue):
         """Runs in a separate thread because pyaudio's write() blocks until the
         sound card consumes the samples. Decoupling playback from the async event
         loop lets us flush the queue on interrupt without waiting for in-flight
         writes to finish."""
         while True:
             data = audio_output_queue.get()
             if data is None:
                 break
             output_stream.write(data)


     async def send_mic_audio(ws, mic) -> None:
         try:
             while True:
                 raw_data = mic.read(CHUNK, exception_on_overflow=False)

                 # Visual volume meter.
                 audio_data = np.frombuffer(raw_data, dtype=np.int16).astype(np.float64)
                 rms = np.sqrt(np.mean(audio_data**2))
                 meter = int(min(rms / 50, 50))
                 print(f"Mic Level: {'█' * meter}{' ' * (50 - meter)} |", end="\r")

                 # Base64-encode and send audio chunk.
                 b64_audio = base64.b64encode(raw_data).decode("utf-8")
                 await send_event(ws, {
                     "type": "input_audio_buffer.append",
                     "audio": b64_audio,
                 })

                 await asyncio.sleep(0)
         except asyncio.CancelledError:
             pass


     async def receive_events(ws, audio_output_queue: queue.Queue) -> None:
         # Accumulate function call arguments across delta events.
         pending_calls: dict[str, dict] = {}
         async for raw_message in ws:
             if DEBUG_WRITE_LOG:
                 with open("data.jsonl", "a", encoding="utf-8") as f:
                     f.write(json.dumps(raw_message) + "\n")

             event = json.loads(raw_message)
             event_type = event.get("type", "")

             if event_type == "session.created":
                 print(raw_message)

             elif event_type == "session.updated":
                 print(raw_message)

             elif event_type == "error":
                 print(f"\n[Error] {event}")

             elif event_type == "input_audio_buffer.speech_started":
                 # Flush queued AI audio so it doesn't talk over the user.
                 while not audio_output_queue.empty():
                     try:
                         audio_output_queue.get_nowait()
                     except queue.Empty:
                         break

             elif event_type == "input_audio_buffer.speech_stopped":
                 pass

             elif event_type == "input_audio_buffer.committed":
                 pass

             elif event_type == "response.created":
                 pass

             elif event_type == "response.output_text.delta":
                 pass

             elif event_type == "response.output_text.done":
                 pass

             # Audio output deltas - queue for playback.
             elif event_type == "response.output_audio.delta":
                 audio_bytes = base64.b64decode(event.get("delta", ""))
                 audio_output_queue.put(audio_bytes)

             elif event_type == "response.output_audio_transcript.delta":
                 pass

             elif event_type == "response.output_audio_transcript.done":
                 pass

             # Function call started - initialize pending call.
             elif event_type == "response.output_item.added":
                 item = event.get("item", {})
                 if item.get("type") == "function_call" and item.get("status") == "in_progress":
                     item_id = item.get("id", "")
                     pending_calls[item_id] = {
                         "call_id": item.get("call_id", ""),
                         "name": item.get("name", ""),
                         "arguments": "",
                     }
                     print(f"\n[Function Call Started] {item.get('name', '')}")

             # Function call argument deltas - accumulate.
             elif event_type == "response.function_call_arguments.delta":
                 item_id = event.get("item_id", "")
                 if item_id in pending_calls:
                     pending_calls[item_id]["arguments"] += event.get("delta", "")

             elif event_type == "response.function_call_arguments.done":
                 item_id = event.get("item_id", "")
                 call_info = pending_calls.pop(item_id, None)
                 if call_info is None:
                     # Fallback: use data directly from the done event.
                     call_info = {
                         "call_id": event.get("call_id"),
                         "name": event.get("name"),
                         "arguments": event.get("arguments"),
                     }
                 try:
                     await handle_function_call(
                         ws,
                         call_info["call_id"],
                         call_info["name"],
                         call_info["arguments"],
                     )
                 except Exception as e:
                     print(f"Failed to call function for message {call_info}: error - {e}")

             elif event_type == "response.done":
                 pass

             elif event_type == "rate_limits.updated":
                 pass

             else:
                 print(f"\n[Event: {event_type}]")


     async def main():
         if not OPENAI_API_KEY:
             print("Error: OPENAI_API_KEY environment variable not set")
             return

         p = pyaudio.PyAudio()

         input_device_index = int(p.get_default_input_device_info()['index'])
         output_device_index = int(p.get_default_output_device_info()['index'])

         # Channel count must match the device's capabilities or pyaudio errors upon open.
         input_info = p.get_device_info_by_index(input_device_index)
         output_info = p.get_device_info_by_index(output_device_index)
         input_channels = min(int(input_info['maxInputChannels']), 1)
         output_channels = min(int(output_info['maxOutputChannels']), 1)

         mic = p.open(
             format=FORMAT,
             channels=input_channels,
             rate=RATE,
             input=True,
             output=False,
             frames_per_buffer=CHUNK,
             input_device_index=input_device_index,
             start=False,
         )
         speaker = p.open(
             format=FORMAT,
             channels=output_channels,
             rate=RATE,
             input=False,
             output=True,
             frames_per_buffer=CHUNK,
             output_device_index=output_device_index,
             start=False,
         )
         mic.start_stream()
         speaker.start_stream()

         # Audio goes through a queue so we can flush it when the user interrupts.
         # Writing directly to the speaker makes it impossible to cancel in-flight audio.
         audio_output_queue = queue.Queue()
         threading.Thread(
             target=play_audio, args=(speaker, audio_output_queue), daemon=True
         ).start()

         headers = {
             "Authorization": f"Bearer {OPENAI_API_KEY}",
         }

         print("Connecting to OpenAI Realtime API...")

         async with websockets.connect(
             REALTIME_URL,
             additional_headers=headers,
         ) as ws:
             print("Connected! Configuring session...")
             await configure_session(ws)

             print("--- Session Active (Speak into mic) ---")

             mic_task = asyncio.create_task(send_mic_audio(ws, mic))
             try:
                 await receive_events(ws, audio_output_queue)
             finally:
                 mic_task.cancel()
                 try:
                     await mic_task
                 except asyncio.CancelledError:
                     pass

         # Cleanup
         audio_output_queue.put(None)  # Signal playback thread to exit.
         mic.close()
         speaker.close()
         p.terminate()
         print("\nSession ended.")


     if __name__ == "__main__":
         asyncio.run(main())
     ```
   </Accordion>

4. Update the `weave.init()` call with your team and project names.

5. Set your `OPENAI_API_KEY` environment variable.

6. Start the voice assistant:

   ```bash theme={null}
   python weave_ws_voice_assistant.py
   ```

Once the assistant is running, press `T` on your keyboard to mute or unmute the mic. The assistant uses server-side voice activity detection to handle turn-taking and interruptions.

As you speak to the assistant, Weave captures traces that you can explore in the Weave UI, including the audio from the session.


# OpenAI Agents SDK
Source: https://docs.wandb.ai/weave/guides/integrations/openai_agents

Use W&B Weave with the OpenAI Agents SDK to track and monitor your agentic applications

You can use W\&B Weave with the OpenAI Agents SDK to trace and monitor your agentic applications. This guide shows you how to install the integration, initialize Weave in your agent code, and view the resulting traces in the Weave dashboard so you can debug and evaluate multi-agent workflows.

<Tabs>
  <Tab title="Python">
    The [OpenAI Agents Python SDK](https://github.com/openai/openai-agents-python) is a lightweight framework for building multi-agent workflows.

    ## Installation

    Install the required dependencies using `pip`:

    ```bash theme={null}
    pip install weave openai-agents
    ```

    ## Get started

    To use the OpenAI Agents SDK with Weave:

    1. Initialize Weave with your project name.
    2. Add the Weave tracing processor to your agents.
    3. Create and run your agents as usual.

    The following code sample creates an OpenAI Agent and integrates it with Weave for traceability. The sample initializes a Weave project and sets up the `WeaveTracingProcessor` to capture execution traces. A `Weather` data model represents weather information. The `get_weather` function, decorated as a tool the agent can use, returns a sample weather report. An agent named `Hello world` uses basic instructions and access to the weather tool. The `main` function runs the agent asynchronously with a sample input (`What's the weather in Tokyo?`) and prints the final response.

    ```python lines theme={null}
    from pydantic import BaseModel
    from agents import Agent, Runner, function_tool
    import agents
    import weave
    import asyncio

    weave.init("openai-agents")

    class Weather(BaseModel):
        city: str
        temperature_range: str
        conditions: str

    @function_tool
    def get_weather(city: str) -> Weather:
        return Weather(city=city, temperature_range="14-20C", conditions="Sunny with wind.")

    agent = Agent(
        name="Hello world",
        instructions="You are a helpful agent.",
        tools=[get_weather]
    )

    async def main():
        result = await Runner.run(agent, input="What's the weather in Tokyo?")    
        print(result.final_output)

    if __name__ == "__main__":
        asyncio.run(main())
    ```

    ## View traces

    When you run the previous code sample, Weave generates a link to the Weave dashboard. Open the link to inspect traces from your agent run.
  </Tab>

  <Tab title="TypeScript">
    The [OpenAI Agents Node SDK](https://github.com/openai/openai-node) is a lightweight framework for building multi-agent workflows.

    ## Installation

    Install the required dependencies using `npm`. Weave's TypeScript integration requires `@openai/agents` version `0.4.15` or later:

    ```bash theme={null}
    npm install weave @openai/agents zod
    ```

    ## Get started

    In most Node.js environments, you don't need manual instrumentation. Weave automatically instruments `@openai/agents` when the module loads.

    Automatic instrumentation works by registering a module loader hook:

    * **CommonJS projects:** No additional configuration is required.
    * **ESM projects:** Start Node with the `--import=weave/instrument` flag so the instrumentation loads before other modules.

    If you need help setting up your project or determining which type you're using, see [TypeScript SDK: Third-Party Integration Guide](/weave/guides/integrations/js).

    The following code sample creates an OpenAI Agent and integrates it with Weave for traceability. Replace `your-team-name/your-project-name` in the Weave project initialization with your entity and project. The sample defines a `get_weather` tool that returns a sample weather report, then configures an agent named `Hello world` with basic instructions and access to the weather tool. The `main` function runs the agent with a sample input (`What's the weather in Tokyo?`) and logs the final response.

    ```typescript lines theme={null}
    import * as weave from "weave";
    import { Agent, run, tool } from "@openai/agents";
    import { z } from "zod";

    const getWeather = tool({
      name: "get_weather",
      description: "Get the current weather for a given city.",
      parameters: z.object({
        city: z.string().describe("The name of the city"),
      }),
      async execute({ city }) {
        return `${city}: 14-20C, Sunny with wind.`;
      },
    });

    async function main() {
      // UPDATE to your project info.
      await weave.init("your-team-name/your-project-name");

      const agent = new Agent({
        name: "Hello world",
        instructions: "You are a helpful agent.",
        tools: [getWeather],
      });

      const result = await run(agent, [
        { role: "user", content: "What's the weather in Tokyo?" },
      ]);
      console.log(result.finalOutput);
    }

    main();
    ```

    In CommonJS, the `import * as weave from "weave"` statement must appear before `import { Agent, run, tool } from "@openai/agents"` for automatic instrumentation. If your code cannot follow this ordering, use the manual instrumentation technique that follows.

    ### Manual instrumentation

    You only need manual instrumentation in cases where the module loader hook can't run, such as:

    * Bundlers that bundle dependencies into a single file.
    * Environments where Node CLI flags can't be passed.
    * Dynamic module loading patterns that bypass the loader hook.

    In these cases, you can explicitly register the instrumentation with `instrumentOpenAIAgents()`:

    ```typescript lines theme={null}
    import * as weave from "weave";

    await weave.init("openai-agents");
    await weave.instrumentOpenAIAgents();
    ```

    For full control over the tracing processor, such as for custom processor configuration or conditional registration, create and register one manually:

    ```typescript lines theme={null}
    import { addTraceProcessor } from "@openai/agents";
    import { createOpenAIAgentsTracingProcessor } from "weave";

    ...

    const processor = createOpenAIAgentsTracingProcessor();
    addTraceProcessor(processor);
    ```
  </Tab>
</Tabs>


# OpenRouter
Source: https://docs.wandb.ai/weave/guides/integrations/openrouter

Use OpenRouter's unified interface for many LLMs with automatic Weave integration

This guide shows you how to use Weave to automatically trace calls to LLMs accessed through OpenRouter, so you can monitor, debug, and evaluate your application's model usage from a single dashboard.

OpenRouter is a unified interface for many LLMs, supporting both foundational models like OpenAI GPT-4, Anthropic Claude, and Google Gemini, as well as open-source models like LLama-3, Mixtral, and [more](https://openrouter.ai/models). Some models are offered for free.

OpenRouter offers a REST API and [OpenAI SDK compatibility](https://openrouter.ai/docs/guides/community/openai-sdk) that Weave automatically detects and integrates with. For more details, refer to the OpenRouter [quick start guide](https://openrouter.ai/docs/quick-start). Because Weave detects the OpenAI SDK, you can reuse existing OpenAI code with a few configuration changes.

To switch your OpenAI SDK code to OpenRouter, switch out the API key to your [OpenRouter API](https://openrouter.ai/docs/api-keys) key, set `base_url` to `https://openrouter.ai/api/v1`, and set the model to one of the available [chat models](https://openrouter.ai/docs/models). When you call `weave.init()`, provide a project name for your traces. If you don't specify one, Weave uses your default entity. To find or update your default entity, refer to [User Settings](https://docs.wandb.ai/platform/app/settings-page/user-settings/#default-team) in the W\&B Models documentation. After you run the following example, Weave captures your call to OpenRouter as a trace in your Weave project.

```python lines {5,10-12} theme={null}
import os
import openai
import weave

weave.init('openrouter-weave')

system_content = "You are a travel agent. Be descriptive and helpful."
user_content = "Tell me about San Francisco"

client = openai.OpenAI(
    api_key=os.environ.get("OPENROUTER_API_KEY"),
    base_url="https://openrouter.ai/api/v1",
)
chat_completion = client.chat.completions.create(
    extra_headers={
    "HTTP-Referer": "[YOUR-SITE-URL]", # Optional, for including your app on openrouter.ai rankings.
    "X-Title": "[YOUR-APP-NAME]", # Optional. Shows in rankings on openrouter.ai.
    },
    model="meta-llama/llama-3.1-8b-instruct:free",
    messages=[
        {"role": "system", "content": system_content},
        {"role": "user", "content": user_content},
    ],
    temperature=0.7,
    max_tokens=1024,
)
response = chat_completion.choices[0].message.content
print("Model response:\n", response)
```

While this is a basic example to get started, see the [OpenAI](/weave/guides/integrations/openai#track-your-own-ops) guide for more details on how to integrate Weave with your own functions for more complex use cases.


# Together AI
Source: https://docs.wandb.ai/weave/guides/integrations/together_ai

Track Together AI's open-source LLMs with Weave using OpenAI SDK compatibility for seamless tracing and evaluation of model calls.

Together AI is a platform for building and fine-tuning generative AI models. It focuses on open-source LLMs and lets customers fine-tune and host their own models. This guide shows you how to trace and evaluate Together AI model calls in Weave by using Together's OpenAI SDK compatibility. You can monitor inputs, outputs, and performance of open-source LLMs alongside your other Weave-tracked work.

<Info>
  Full Weave support for the `together` Python package is in development.
</Info>

In the meantime, Together supports [OpenAI SDK compatibility](https://docs.together.ai/docs/openai-api-compatibility), which Weave automatically detects and integrates with. You can use the standard OpenAI client to call Together AI models and get automatic Weave tracing without any additional setup.

To switch to the Together API, set `api_key` to your [Together API](https://docs.together.ai/docs/quickstart) key, `base_url` to `https://api.together.xyz/v1`, and `model` to one of the [chat models](https://docs.together.ai/docs/inference-models#chat-models). The following example initializes Weave and then makes a chat completion call against a Together-hosted model. Once it runs, the call appears as a trace in your Weave project.

```python lines {5, 10-13} theme={null}
import os
import openai
import weave

weave.init('together-weave')

system_content = "You are a travel agent. Be descriptive and helpful."
user_content = "Tell me about San Francisco"

client = openai.OpenAI(
    api_key=os.environ.get("TOGETHER_API_KEY"),
    base_url="https://api.together.xyz/v1",
)
chat_completion = client.chat.completions.create(
    model="mistralai/Mixtral-8x7B-Instruct-v0.1",
    messages=[
        {"role": "system", "content": system_content},
        {"role": "user", "content": user_content},
    ],
    temperature=0.7,
    max_tokens=1024,
)
response = chat_completion.choices[0].message.content
print("Together response:\n", response)
```

This example gets you started. For more details on how to integrate Weave with your own functions for more complex use cases, see the [OpenAI](/weave/guides/integrations/openai#track-your-own-ops) guide.


# Define and log attributes
Source: https://docs.wandb.ai/weave/guides/tools/attributes

Use attributes to add metadata to your traces and evaluations.

Attributes in Weave let you attach custom metadata to your traces and evaluations. This metadata can include information like environment names, model versions, experiment IDs, user IDs, or other contextual information to help you organize, filter, and analyze your Weave data.

Attributes are especially helpful for grouping or filtering traces by:

* Deployments
* Tenants
* Experiments

Weave provides two ways to add attributes:

* **Per-call attributes**: Use the `weave.attributes()` context manager to add metadata to specific operations or code blocks.
* **Global attributes**: Use the `global_attributes` field to set attributes at initialization that apply to all traces and evaluations in your project.

You can view all attributes logged during traces and evaluations in the UI. You can then use them to filter and group data.

This guide is for Weave users who want to enrich their traces and evaluations with custom metadata. It covers how to attach attributes to individual calls, set project-wide defaults at initialization, combine the two approaches, and read the current attribute set during execution.

## Per-call attributes

The `weave.attributes()` context manager lets you add metadata to specific traced operations. Use it to tag particular function calls or evaluation runs with contextual information.

<Tip>
  You can't modify `call.attributes` once the call starts. Use this
  context manager to set any metadata before invoking the op.
</Tip>

<Tabs>
  <Tab title="Python">
    ```python lines theme={null}
    import weave

    weave.init("[TEAM-NAME]/[PROJECT-NAME]")

    @weave.op
    def my_function(name: str):
        return f"Hello, {name}!"

    # Add attributes to a specific call
    with weave.attributes({'env': 'production', 'user_id': '12345'}):
        result = my_function("World")
    ```
  </Tab>

  <Tab title="TypeScript">
    ```typescript lines theme={null}
    import {init, op, withAttributes} from 'weave';

    async function main() {
      await init('your-team/attribute-example');

      const myFunction = op(async function myFunction(name: string) {
        return `Hello, ${name}!`;
      });

      // Add attributes to a specific call
      const result = await withAttributes(
        {env: 'production', user_id: '12345'},
        async () => myFunction('World')
      );

      console.log('Result:', result);
    }

    main().catch(console.error);
    ```
  </Tab>
</Tabs>

The function attaches the attributes to all traced operations within the context manager block (Python) or callback function (TypeScript).

You can also nest `weave.attributes()` contexts. Inner contexts override outer contexts for the same keys:

<Tabs>
  <Tab title="Python">
    ```python lines theme={null}
    @weave.op
    def process_data(data: str):
        return data.upper()

    # Outer context
    with weave.attributes({
        "env": "production",
        "version": "1.0.0",
        "region": "us-west-2"
    }):
        process_data("hello")  # Has all three attributes
        
        # Inner context overrides 'version'
        with weave.attributes({
            "version": "1.1.0",
            "experiment": "exp-456"
        }):
            process_data("world")  # Has env='production', version='1.1.0', region='us-west-2', experiment='exp-456'
    ```
  </Tab>

  <Tab title="TypeScript">
    ```typescript lines theme={null}
    import {init, op, withAttributes} from 'weave';

    async function main() {
      await init('your-team/attribute-example');

      const processData = op(async function processData(data: string) {
        return data.toUpperCase();
      });
      
      // Outer context
      await withAttributes(
        {
          env: 'production',
          version: '1.0.0',
          region: 'us-west-2'
        },
        async () => {
          await processData('hello');  // Has all three attributes
          
          // Inner context overrides 'version'
          await withAttributes(
            {
              version: '1.1.0',
              experiment: 'exp-456'
            },
            async () => {
              await processData('world');  // Has env='production', version='1.1.0', region='us-west-2', experiment='exp-456'
            }
          );
        }
      );
    }

    main().catch(console.error);
    ```
  </Tab>
</Tabs>

## Global attributes

When you set global attributes during Weave initialization, they automatically apply to all traces and evaluations in your project. This is useful for propagating project-wide metadata like environment, deployment version, or team information.

<Tabs>
  <Tab title="Python">
    ```python lines theme={null}
    import weave

    weave.init(
        "my-project",
        global_attributes={
            "env": "production",
            "app_version": "2.1.0",
            "region": "us-west-2",
            "team": "ml-platform"
        }
    )

    # The global_attributes dictionary now applies these attributes to all subsequent operations
    @weave.op
    def my_function():
        return "Hello"

    my_function()  # Automatically has all global attributes

    # Evaluations also get global attributes
    evaluation = weave.Evaluation(dataset=examples, scorers=[scorer])
    asyncio.run(evaluation.evaluate(model))  # Has all global attributes
    ```
  </Tab>

  <Tab title="TypeScript">
    ```typescript lines theme={null}
    import {init, op, withAttributes} from 'weave';

    async function main() {
        await init('your-team/attribute-example', {
            globalAttributes: {
                env: 'production',
                app_version: '2.1.0',
                region: 'us-west-2',
                team: 'ml-platform'
            }
        });
        
        // The globalAttributes object now applies these attributes to all subsequent operations
        const myFunction = op(async function myFunction() {
            return 'Hello';
        });
        
        const result = await myFunction();  // Automatically has all global attributes
        console.log('Result:', result);
    }
    main().catch(console.error);
    ```
  </Tab>
</Tabs>

### Combine global and per-call attributes

You can use global attributes and per-call attributes together. Per-call attributes with the same key override global attributes:

<Tabs>
  <Tab title="Python">
    ```python lines theme={null}
    import weave

    # Set global attributes
    weave.init(
        "my-project",
        global_attributes={
            "env": "production",
            "app_version": "2.1.0"
        }
    )

    @weave.op
    def process(data: str):
        return data

    # This call has: env='production', app_version='2.1.0'
    process("test1")

    # This call has: env='staging', app_version='2.1.0', experiment='A'
    with weave.attributes({'env': 'staging', 'experiment': 'A'}):
        process("test2")
    ```
  </Tab>

  <Tab title="TypeScript">
    ```typescript lines theme={null}
    import {init, op, withAttributes} from 'weave';

    async function main() {
        // Set global attributes
        await init('your-team/attribute-example', {
        globalAttributes: {
            env: 'production',
            app_version: '2.1.0'
        }
        });

        const process = op(async function process(data: string) {
        return data;
        });

        // This call has: env='production', app_version='2.1.0'
        await process('test1');

        // This call has: env='staging', app_version='2.1.0', experiment='A'
        await withAttributes(
        {env: 'staging', experiment: 'A'},
        async () => process('test2')
        );
    }
    main().catch(console.error);
    ```
  </Tab>
</Tabs>

## Get attributes during execution

At runtime, you can inspect the attributes in effect for a Call. This is useful when you want to debug which metadata Weave attaches, or when your code needs to branch on contextual information set by an outer caller.

<Tabs>
  <Tab title="Python">
    The following example sets the Weave decorator to log the `process_data` function, configures attributes to log, and then returns them on execution.

    ```python lines  theme={null}
    import weave

    weave.init("your-team/your-project")

    @weave.op
    def process_data(data: str):
        # Get the current call inside the op
        call = weave.get_current_call()
        if call:
            print(f"Attributes: {call.attributes}")
        return data.upper()

    # Set attributes and execute the function
    with weave.attributes({
        "env": "production",
        "version": "1.0.0",
        "region": "us-west-2"
    }):
        process_data("hello")
        
        with weave.attributes({
            "version": "1.1.0",
            "experiment": "exp-456"
        }):
            process_data("world")
    ```

    This outputs:

    ```text theme={null}
    Attributes: {'env': 'production', 'version': '1.0.0', 'region': 'us-west-2'}
    Attributes: {'env': 'production', 'version': '1.1.0', 'region': 'us-west-2', 'experiment': 'exp-456'}
    ```

    <Note>
      `weave.get_current_call()` only works inside a `@weave.op` decorated function. Outside of an op, it returns `None`.
    </Note>
  </Tab>

  <Tab title="TypeScript">
    With the Weave TypeScript SDK, you can get the current attributes with `client.getCurrentAttributes()`. Unlike the Weave Python SDK, you can access TypeScript attributes within a `withAttributes()` context, not just inside an op.

    ```typescript lines theme={null}
    import * as weave from 'weave';
    import { withAttributes } from 'weave';

    async function main() {
      const client = await weave.init('your-team/your-project');

      const processData = weave.op(async function processData(data: string) {
        // You can also access attributes inside an op
        const attrs = client.getCurrentAttributes();
        console.log('Attributes inside op:', attrs);
        return data.toUpperCase();
      });

      // Set attributes and execute the function
      await withAttributes(
        {
          env: 'production',
          version: '1.0.0',
          region: 'us-west-2'
        },
        async () => {
          // Get attributes from anywhere in the context
          console.log('Current attributes:', client.getCurrentAttributes());
          await processData('hello');

          await withAttributes(
            {
              version: '1.1.0',
              experiment: 'exp-456'
            },
            async () => {
              console.log('Nested attributes:', client.getCurrentAttributes());
              await processData('world');
            }
          );
        }
      );
    }

    main().catch(console.error);
    ```

    This outputs:

    ```text theme={null}
    Current attributes: { env: 'production', version: '1.0.0', region: 'us-west-2' }
    Attributes inside op: { env: 'production', version: '1.0.0', region: 'us-west-2' }
    Nested attributes: { env: 'production', version: '1.1.0', region: 'us-west-2', experiment: 'exp-456' }
    Attributes inside op: { env: 'production', version: '1.1.0', region: 'us-west-2', experiment: 'exp-456' }
    ```
  </Tab>
</Tabs>


# Map columns in datasets
Source: https://docs.wandb.ai/weave/guides/tools/column-mapping

Map columns in datasets to different names. This helps you align the column names in your dataset with the column names expected by the scorer.

<Tabs>
  <Tab title="Python">
    In the Weave Python SDK, use the `column_map` attribute on a scorer to map its expected parameter names to your dataset's column names. The mapping format is `{scorer_parameter: dataset_column}`.

    The following example maps the `output` and `target` parameters to a dataset's `model_output` and `answer` columns:

    ```python theme={null}
    from weave.scorers import EmbeddingSimilarityScorer

    similarity_scorer = EmbeddingSimilarityScorer()

    similarity_scorer.column_map = {
        "output": "model_output",  # The model's generated text
        "target": "answer"         # The expected or reference response
    }
    ```

    For more information about scorer column mapping, see [Mapping Column Names with `column_map`](../evaluation/scorers#mapping-column-names-with-column_map).
  </Tab>

  <Tab title="TypeScript">
    In the Weave TypeScript SDK, you configure column mapping on the `Evaluation` object using the `columnMapping` option, not on individual scorers. The mapping format is `{scorer_key: dataset_column}`.

    The following example maps `expectedOutputTimesTwo` (used by the scorer) to the `expected` column in the dataset:

    ```typescript theme={null}
    import * as weave from 'weave';

    const myScorer = weave.op(
      ({modelOutput, datasetRow}) => {
        return modelOutput * 2 === datasetRow.expectedOutputTimesTwo;
      },
      {name: 'myScorer'}
    );

    const myEval = new weave.Evaluation({
      id: 'my-evaluation',
      dataset: [{expected: 2}],
      scorers: [myScorer],
      columnMapping: {expectedOutputTimesTwo: 'expected'},
    });
    ```

    For more information about TypeScript scorer arguments, see [Scorer Keyword Arguments](../evaluation/scorers#scorer-keyword-arguments).
  </Tab>
</Tabs>


# Compare traces and other logged information
Source: https://docs.wandb.ai/weave/guides/tools/comparison

Visually compare and diff code, traces, prompts, models, and configurations

The W\&B Weave Comparison feature lets you visually compare and diff code, traces, prompts, models, and model configurations. You can compare two objects side-by-side or analyze a larger set of objects to identify differences, patterns, and trends. This is useful when you iterate on prompts or models, debug behavior changes between runs, or audit how configurations have evolved over time.

This guide is for Weave users who want to inspect differences between logged objects in the UI. It covers how to start a comparison and the available actions to tailor your comparison view, including baseline comparisons and numeric diff formatting.

## Access the Comparison view

To open the Comparison view for two or more logged objects:

1. In the sidebar, select the type of object to compare (for example, **Traces** or **Models**).
2. Select the objects that you want to compare. The selection method varies depending on the type of object you're comparing:
   * For **Traces**, select traces to compare by checking the checkboxes for the appropriate rows in the Trace column.
   * For objects such as **Models**, navigate to the model Versions page and check the checkboxes next to the versions that you want to compare.
3. Select **Compare** to open the Comparison view. At the top of the page, the comparison bar contains interactive call ID tokens (for example, `f02b`, `4a98`) that you can drag, reorder, or remove to change the comparison.
4. You can refine your view using the available actions.

## Customize the Comparison view

In the Comparison view, you can adjust how objects are displayed and compared. The following sections describe the available actions:

* [Display summary](#display-summary)
* [Display a Calls view](#display-a-calls-view)
* [Change the diff display](#change-the-diff-display)
* [Display side-by-side](#display-side-by-side)
* [Display in a unified view](#display-in-a-unified-view)
* [Set a baseline](#set-a-baseline)
* [Remove a baseline](#remove-a-baseline)
* [Change the comparison order](#change-the-comparison-order)
* [Change numeric diff display format](#change-numeric-diff-display-format)
* [Compare with baseline or previous](#compare-with-baseline-or-previous)
* [Compare a pair from a multi-object comparison](#compare-a-pair-from-a-multi-object-comparison)
* [Remove an object from comparison](#remove-an-object-from-comparison)

### Display summary

The default view for a selected object comparison is a summary. This summary includes a preview of the LLM input and output, along with tokens, cost, and latency.

<Frame>
  <img alt="Comparison Summary view of two objects" />
</Frame>

For traces, the colored bar displayed at the top of the comparison table reflects any custom kinds and colors that you've applied to your Ops. For more information on using this enhanced visibility, see [Apply kinds and colors](/weave/guides/tracking/ops#apply-kinds-and-colors).

### Display a Calls view

To compare each object's Calls, select **Calls**. This view:

* Displays the complete trace tree for the Call.
* Provides basic in-memory text search.
* Lets you toggle the display of per-call latency, cost, tokens, Op kinds, and feedback.

<Frame>
  <img alt="Calls Comparison view of two objects" />
</Frame>

### Change the diff display

By default, **Diff only** is off. To filter the table rows so that only changed rows appear, toggle **Diff only** on.

**Diff only** applies to side-by-side and unified views and is disabled in other views.

### Display side-by-side

To compare each object side-by-side in separate columns, select **Side-by-side**.

<Frame>
  <img alt="Side-by-side Comparison view of two objects" />
</Frame>

### Display in a unified view

To compare two objects in a unified view, select **Unified**. This view is not available when comparing more than two objects.

<Frame>
  <img alt="Unified Comparison view of two objects" />
</Frame>

### Set a baseline

By default, each object in the Comparison view is compared to the object to its left. However, you can set an object as the *baseline*, which means that all objects are compared to the leftmost object in the view. Use a baseline when you want every object measured against a single reference point, such as a known-good run.

To set an object as the baseline:

1. In the comparison bar, hold the pointer over the object that you want to set as the baseline.
2. Click the **action (<Icon icon="ellipsis-vertical" />)** menu to the right of the ID.
   <img alt="Comparison view showing the Make baseline option for the selected object in the comparison bar." />
3. In the list, select **Make baseline**. The UI refreshes with the baseline object in the leftmost position in the comparison bar and `Baseline` next to the ID.
   <img alt="Comparison view showing the selected baseline object in the leftmost position of the comparison bar." />

### Remove a baseline

To remove an object as a baseline:

1. In the comparison bar, hold the pointer over the baseline object.
2. Click the **action (<Icon icon="ellipsis-vertical" />)** menu to the right of the ID.
3. In the list, select **Remove baseline**. `Baseline` no longer displays next to the call ID.

### Change the comparison order

Drag the objects in the comparison bar to reorder them. This also reorders the corresponding columns in the comparison table.

To change the comparison order:

1. In the comparison bar, hold the pointer over the ID that you want to reorder.
2. Click and hold the six dots to the left of the ID and drag it to the left or the right as needed.
3. Place the ID in the desired ordering. The comparison data refreshes with an updated comparison ordering.

### Change numeric diff display format

For numeric values such as `completion_tokens` and `total_tokens`, you can view the diff as either an integer or a percentage. You can also view positive numeric values as a multiplier.

To change a numeric diff's display format:

1. In the Comparison table, find the numeric value whose diff display format you want to update.
   <img alt="A numeric value displayed as an integer." />
2. Click the diff value. The format automatically updates to either an integer or a percentage.
   <img alt="A numeric value updated to a percentage." />

### Compare with baseline or previous

This option is only available when comparing three or more objects. You can also [set a baseline](#set-a-baseline) or [remove an existing baseline](#remove-a-baseline).

To perform a baseline comparison with three or more objects:

1. In the upper-right corner of either the Side-by-side or Unified Comparison view, click the list. Depending on your current view configuration, the list title is either **Compare with previous** or **Compare with baseline**.
2. Depending on your current view configuration, select either **Compare with previous** or **Compare with baseline**.
   * **Compare with baseline**: Sets the leftmost object as the baseline. The table updates so that the leftmost column is the baseline.
   * **Compare with previous**: No object is set as baseline.

### Compare a pair from a multi-object comparison

This option is only available when comparing three or more objects.

When comparing three or more objects, you can compare a single object to its adjacent object or a baseline. This changes the Comparison table view so that the view is identical to a two-object comparison.

To compare a pair of objects from a multi-object comparison:

1. In the comparison bar, select the ID you want to compare. If you've set a baseline, the view compares the selected object to the baseline. Otherwise, it compares the object to the ID immediately to its left in the comparison bar.
2. To select the item, click the ID. The UI refreshes with a two-way comparison table.
   <img alt="Comparing a pair from a multi-object comparison." />

To reset the view so that the first six objects selected for comparison are displayed in the table, click the ID again.

### Remove an object from comparison

This option is only available when comparing three or more objects.

To remove an object from comparison:

1. In the comparison bar, find the object that you want to remove from comparison.
2. Click the **action (<Icon icon="ellipsis-vertical" />)** menu to the right of the ID.
3. In the list, select **Remove object from comparison**. The UI refreshes with an updated table that no longer includes the removed object.

## Usage notes

The following notes describe limitations and behaviors of the Comparison feature:

* The Comparison feature is only available in the UI.
* You can compare as many objects as you'd like, but the UI displays a maximum of six. To view an object in the comparison table that isn't visible when comparing more than six objects, either [change the comparison order](#change-the-comparison-order) so the object is one of the first six from left to right, or [pair it from a multi-object comparison](#compare-a-pair-from-a-multi-object-comparison) to view it.


# Compare model performance using the Evaluation Playground
Source: https://docs.wandb.ai/weave/guides/tools/evaluation_playground

Compare and evaluate model performance without code using Weave's Evaluation Playground with custom datasets and LLM judges.

The Evaluation Playground lets you access existing models and compare their performance using evaluation datasets and LLM scoring judges. You can start experimenting with and comparing your models without setting up any code. You can also save the models, scorers, and datasets you develop in the playground for later development and deployment.

For example, you can open the Evaluation Playground, add two models you've previously saved, and then evaluate their performance based on a new or previously saved question-answer style evaluation dataset. You can then add a new model in the interface, add system prompts to it, and run a new evaluation on all three models to see how they perform against one another.

<Frame>
  <img alt="Evaluation Playground Interface" />
</Frame>

## Set up an evaluation in the playground

The following sections describe how to open the Evaluation Playground, choose a starting point for your evaluation, and configure the dataset, models, and scorers.

To set up an evaluation in the Evaluation Playground:

1. Open [the Weave UI](https://wandb.ai/wandb/quickstart_playground/weave/playground), and then open the project that you want to perform the evaluation in. The Traces page opens.
2. From the Traces page, click the **Playground** icon in the left menu, and then select the **Evaluate** tab on the Playground page. On the Evaluate page, you can either:
   * **Load a demo example**: Loads a predefined configuration that evaluates the MoonshotAI Kimi K2 model on expected output and uses an LLM judge to determine its correctness. You can use this configuration to experiment with the interface.
   * **Start from scratch**: Loads a blank configuration for you to build upon.
3. If you selected **Start from scratch**, add a descriptive title and description for your evaluation in the **Title** and **Description** fields.

Follow the instructions in the next sections to set up your datasets, models, and scorers.

### Add a dataset

[Datasets](../core-types/datasets) are organized collections of example user inputs and your expected responses from models. During evaluation, the playground feeds each test input into your model, collects the output, and then scores the output based on a metric you've selected, such as correctness. You can create a dataset in the UI, add an existing dataset already saved to your project, or upload a new dataset.

You can upload datasets in the following formats:

* `.csv`
* `.tsv`
* `.json`
* `.jsonl`

For more information about how to format and save datasets to Weave, see the Datasets documentation.

To add a dataset in the **Dataset** section:

1. Click the dropdown menu and then select either:
   * **Start from scratch** to create a new dataset in the UI.
   * **Upload a file** to upload a dataset from your local machine.
   * An existing dataset already saved to your project.
2. Optional: Click **Save** to save the dataset to your project for later use.

After you select an option, the dataset is visible in the right pane of the UI and you can edit each field as necessary by clicking it. You can also add new rows to the dataset by clicking **Add row**.

<Note>
  You can only use the UI to edit new datasets.

  It's also important to name the columns in your dataset `user_input` and `expected_output` so that the scorers can access the data.
</Note>

### Add a model

In the context of Weave, [models](/weave/guides/core-types/models) are a combination of an AI model (such as GPT) and the environment (in this case the system prompt) that defines how the model operates during the evaluation. You can select existing models in your project or create new ones to evaluate, and you can add multiple models at once to evaluate them simultaneously with the same dataset and scorer. **You can only use models created using the playground feature.**

To add a model in the **Models** section of the Evaluation Playground:

1. Click **Add Model** and either select **New Model** or an existing model from the dropdown menu.

2. If you selected **New Model**, configure the following fields:

   * **Name**: Add a descriptive name to your new model.
   * **LLM Model**: Select a foundation model to build your new model on, such as OpenAI's GPT-4. You can select from a list of foundation models you've already configured access to, or you can add access to a foundation model by selecting **Add AI provider** and selecting a model. Adding a provider prompts you to enter your access credentials to that provider. See your provider's documentation to learn how to locate your API key, endpoints, and any additional configuration information you need to access the model using Weave.
   * **System Prompt**: Provide the model instructions on how it should behave, for example, `You are a helpful assistant specializing in Python programming.` The `user_input` from your dataset is sent in a subsequent message and you don't need to include it in the system prompt.

   If you choose an existing model, a new field appears beside the model's name that lets you select a version of the existing model, and there are no other additional fields to configure. To make changes to your existing model before or after evaluation, use the [Prompt Playground](../tools/playground).

3. Optional: Click **Save** to save the model to your project for later use.

4. Optional: To evaluate multiple models simultaneously, click **Add Model** again and add other models as necessary.

### Add scorers

[Scorers](../evaluation/scorers) use LLM judges to measure and assess the quality of AI model outputs. You can select existing scorers in your project or create new ones to evaluate your models with.

To add a scorer in the Evaluation Playground:

1. Click **Add Scorer** and then configure the following fields:
   * **Name**: Add a descriptive name to your scorer.
   * **Type**: Select how scores are output, either a boolean or a number. Boolean scorers return a binary `True` or `False` depending on whether the model's output met the judging parameters you set for it. Number scorers output a score between `0` and `1`, providing a general grade of how well the model's output met your judging parameters.
   * **LLM-as-a-judge-model**: Select a foundation model to use as the scorer's judge. Similar to the LLM Model field in the **Models** section, you can select from foundation models you've already configured access to, or configure new access to foundation models.
   * **Scoring Prompt**: Provide the LLM judge parameters for scoring the output. For example, to check for hallucinations, enter a scoring prompt similar to this:

     ```text theme={null}
     Given the following context and answer, determine if the answer contains any information not supported by the context.

     User input: {user_input}
     Expected output: {expected_output}
     Model Output: {output}

     Is the model output correct?
     ```

     You can use the fields from your datasets and responses as variables in the scoring prompt, such as `{user_input}`, `{expected_output}`, and `{output}`. To see a list of available variables, click **Insert variable** in the UI.

2. Optional: Click **Save** to save the scorer to your project for later use.

## Run the evaluation

With your datasets, models, and scorers configured, you're ready to run the evaluation and generate results.

* To run the evaluation in the Evaluation Playground, click **Run eval**.

Weave runs an individual evaluation for each model you've added and collects metrics on each request made using the dataset. Weave saves each of these evaluations in the **Evals** section for later review.

## Review evaluation results

After the evaluation completes, you can review the results to compare how each model performed against your dataset and scorers.

When the evaluation completes, the playground opens a report that displays metrics collected on each request made to your models.

<Frame>
  <img alt="Evals hero" />
</Frame>

The **Dataset results** tab displays the input, expected output, the model's actual output, latency, token usage, and scoring results. You can click the IDs in the **Row** column to open a detailed view of the metrics for a specific set of requests. You can also use the display format buttons directly below the tabs to change the display format of the report's cells.

The **Summary** tab provides an overview of how each model performed with visual representations of the data.

For more information about how to open and compare evaluations, see [Evaluations](../core-types/evaluations).


# Use the Playground to experiment with prompts
Source: https://docs.wandb.ai/weave/guides/tools/playground

Simplify the process of iterating on LLM prompts and responses

<Tip>
  For a limited time, Serverless Inference is included in your free tier. Serverless Inference provides access to leading open source foundation models through API and the Weave Playground.

  * [Developer documentation](../integrations/inference)
  * [Product page](https://wandb.ai/site/inference)
</Tip>

Evaluating LLM prompts and responses is challenging. The Weave Playground simplifies the process of iterating on LLM prompts and responses, helping you experiment with different models and prompts. With features like prompt editing, message retrying, and model comparison, Playground helps you test and improve your LLM applications. Playground is intended for developers and prompt engineers who want to test, compare, and refine LLM prompts before integrating them into applications. Playground supports models from providers such as OpenAI, Anthropic, and Google, as well as [custom providers](#add-a-custom-provider).

* **Quick access:** Open the **Playground** from the Weave sidebar menu for a fresh session or from the Call page to test an existing project.
* **Message controls:** Edit, retry, or delete messages directly within the chat.
* **Flexible messaging:** Add new messages as either user or system inputs, and send them to the LLM.
* **Customizable settings:** Configure your preferred LLM provider and adjust model settings.
* **Multi-LLM support:** Switch between models, with team-level API key management.
* **Compare models:** Compare how different models respond to prompts.
* **Custom providers:** Test OpenAI-compatible API endpoints for custom models.
* **Saved models:** Create and configure a reusable model preset for your workflow.

Get started with the Playground to optimize your LLM interactions and streamline your prompt engineering process and LLM application development. The following sections describe how to configure and use the Playground:

* [Add provider credentials and information](#add-provider-credentials-and-information)
* [Access the Playground](#access-the-playground)
* [Select an LLM](#select-an-llm)
* [Customize Playground settings](#customize-playground-settings)
* [Message controls](#message-controls)
* [Compare LLMs](#compare-llms)
* [Custom providers](#custom-providers)
* [Saved models](#saved-models)

## Add provider credentials and information

Before you can use Playground, you must add provider credentials so that Playground can authenticate requests to each model provider on your behalf. Playground supports models from many providers. To use one of the available models, you must add the appropriate information to your team secrets in W\&B settings.

* Amazon Bedrock:
  * `AWS_ACCESS_KEY_ID`
  * `AWS_SECRET_ACCESS_KEY`
  * `AWS_REGION_NAME`
* Anthropic: `ANTHROPIC_API_KEY`
* Azure:
  * `AZURE_API_KEY`
  * `AZURE_API_BASE`
  * `AZURE_API_VERSION`
* Deepseek: `DEEPSEEK_API_KEY`
* Google: `GEMINI_API_KEY`
* Groq: `GROQ_API_KEY`
* Mistral: `MISTRAL_API_KEY`
* OpenAI: `OPENAI_API_KEY`
* X.AI: `XAI_API_KEY`

## Access the Playground

You can access the Playground in two ways:

1. *Open a fresh Playground page with a default system prompt*: In the sidebar of a Weave project, select **Playground**. Playground opens in the same tab.
2. *Open Playground for a specific call*:
   1. In the sidebar, select the **Traces** tab. A list of traces displays.
   2. In the list of traces, click the name of the call that you want to view. The call's details page opens.
   3. Click **Open chat in Playground**. Playground opens in a new tab.

<Frame>
  <img alt="Screenshot of Open in Playground button" />
</Frame>

## Select an LLM

You can switch the LLM using the **Select a model** dropdown in the prompt header (top of the main Playground panel). The following providers are available:

* Amazon Bedrock
* Anthropic
* Azure
* Deepseek
* Google
* Groq
* Mistral
* OpenAI
* X.AI

The available models depend on the providers configured for your team.

## Customize Playground settings

You can adjust the parameters of your selected model and add functions for the model to call. The following sections describe how to customize Playground settings.

### Adjust LLM parameters

You can experiment with different parameter values for your selected model. To adjust parameters in the Playground, do the following:

1. In the prompt header (top of main panel), click the **Chat settings (<Icon icon="sliders" />)** button to open the **Chat settings** panel.
2. In the **Chat settings** panel, adjust parameters as desired. You can also toggle Weave call tracking on or off, and [add a function](#add-a-function).
3. Changes apply automatically. Click **Chat settings** again, or the **x** in the upper-right corner, to close the panel. The hover text for the **Chat settings** button shows the settings you've changed.

<Note>
  If you leave the page, your settings are lost. To save your settings, [save your model](#save-a-model). If you want to discard your changed settings and start over, refresh the page.
</Note>

<Frame>
  <img alt="The Weave Playground showing the Select a model dropdown open and the Chat settings panel open on the right." />
</Frame>

Playground lets you generate multiple outputs for the same input by setting the number of trials. The default setting is `1`. To adjust the number of trials, open the **Chat settings** panel and adjust the **Number of trials** setting.

### Add a function

You can test how different models use functions based on input they receive from the user. To add a function for testing in Playground, in the **Chat settings** panel, click **+ Add function**. Follow the on-screen guidance to define the function and save your changes.

## Message controls

The Playground chat interface is organized into three areas: the prompt definition area, the messages panel, and the message composer. The following sections describe each area.

### Prompt definition area

The **Prompt definition area** lets you define the instructions that shape the model's behavior throughout an interaction.

Use this area to provide context that applies consistently before any messages are exchanged. This includes role definition, tone and style guidance, behavioral constraints, and output requirements. Changes made here affect all subsequent interactions unless modified.

It includes:

* **Prompt selector**: Select an existing saved prompt or create a new one.
* **Message role selector**: Specify the role of the message being defined (**System**, **Assistant**, or **User** role).
* **Prompt text**: Enter the instruction text that establishes how the model responds.
* **Add message** button: Lets you include additional messages in the prompt context before execution.

These messages are sent to the model together, and you can use them to:

* Add supplemental system-level instructions.
* Provide example assistant messages to guide responses (such as few-shot prompting).
* Predefine user messages for testing specific scenarios.

### Messages panel

The **Messages panel** displays the conversation generated during execution.

It includes:

* Any predefined messages included in the prompt setup.
* Messages sent from the message composer.
* Responses returned by the model.

You can also **Copy**, **Delete**, **Edit**, and **Retry** messages inside the panel.

### Message composer (input field)

Use the **Message composer** to send new messages to the model.

It supports selecting the message role and submitting messages for execution. Most interactions are authored as **User** messages. You can add **System** or **Assistant** messages when testing instruction changes.

<Frame>
  <img alt="Weave Playground showing the prompt definition area at the top, message history in the middle, and a message composer for sending new messages at the bottom." />
</Frame>

## View message history

To view message history, click the **History (<Icon icon="clock-rotate-left" />)** button in the right-side Playground toolbar. This opens a History panel showing all messages sent for the current project.

Selecting an item from the history loads it into an additional chat panel for comparison.

## Compare LLMs

Playground lets you compare LLMs side by side so you can evaluate how different models respond to the same prompts. To perform a comparison, click the **Add Chat (<Icon icon="plus" />)** button in the right-side Playground toolbar. A second chat opens next to the original chat. The second chat has the same functionality as the original chat, such as choosing the model, adjusting the settings, and adding functions.

## Custom providers

In addition to the built-in providers, the Playground supports custom providers that expose OpenAI-compatible API endpoints. The following sections describe how to add, edit, remove, and use custom providers.

### Add a custom provider

In addition to the built-in providers, you can use the Playground to test OpenAI-compatible API endpoints for custom models. Examples include:

* Older versions of supported model providers
* Local models

To add a custom provider to the Playground, do the following:

1. In the prompt header (top of main panel), click the **Select a model** dropdown.
2. Select **+ Add AI provider**.
3. Select **Custom Provider**.
4. In the pop-up modal, enter the provider information:
   * **Provider name**: A name for the provider, such as `openai` or `ollama`.
   * **API key**: The API key for the provider, such as an OpenAI API key.
   * **Base URL**: The base endpoint for the provider, such as `https://api.openai.com/v1/` or an ngrok URL like `https://e452-2600-1700-45f0-3e10-2d3f-796b-d6f2-8ba7.ngrok-free.app`.
   * **Headers** (Optional): One or more custom HTTP header key-value pairs.
   * **Models**: One or more models for the provider, such as `deepseek-r1` or `qwq`.
   * **Max tokens** (Optional): For each model, the maximum number of tokens the model can generate in a response.
5. After you've entered your provider information, click **Add provider**.
6. Select your new provider and available models from the **Select a model** dropdown.

<Warning>
  Because of CORS restrictions, you can't call localhost or 127.0.0.1 URLs directly from the Playground. If you're running a local model server (such as Ollama), use a tunneling service like ngrok to expose it securely. For details, see [Use ngrok with Ollama](#use-a-local-model-as-a-custom-provider).
</Warning>

After you add a custom provider, you can test the custom provider models using standard Playground features. You can also [edit](#edit-a-custom-provider) or [remove](#remove-a-custom-provider) the custom provider.

### Edit a custom provider

To edit information for a [previously created custom provider](#add-a-custom-provider), do the following:

1. In the prompt header, click the **Select a model** dropdown. Then select **+Configure providers**.

* Alternatively, in the sidebar menu, you can select **Project**, and then select the **AI Providers** tab.

2. In the **Custom providers** table, find the custom provider you want to update.
3. In the **Last Updated** column of the entry for your custom provider, click the edit button (the pencil icon).
4. In the pop-up modal, edit the provider information.
5. Click **Save**.

### Remove a custom provider

To remove a [previously created custom provider](#add-a-custom-provider), do the following:

1. In the prompt header, click the **Select a model** dropdown. Then select **+Configure providers**.

* Alternatively, in the sidebar menu, you can select **Project**, and then select the **AI Providers** tab.

2. In the **Custom providers** table, find the custom provider you want to update.
3. In the **Last Updated** column of the entry for your custom provider, click the delete button (the trashcan icon).
4. In the pop-up modal, confirm that you want to delete the provider. You can't undo this action.
5. Click **Delete**.

### Use a local model as a custom provider

To test a locally running model in the Playground, use ngrok and Ollama to create a temporary public URL that avoids CORS restrictions. This lets the Playground reach a model that is running on your own machine without violating browser security policies.

To set it up, do the following:

1. [Install ngrok](https://ngrok.com/docs/getting-started/#step-1-install) for your operating system.

2. Start your Ollama model. Replace `[MODEL]` with the name of the Ollama model you want to run.

   ```bash theme={null}
   ollama run [MODEL]
   ```

3. In a separate terminal, create an ngrok tunnel with the required CORS headers:

   ```bash theme={null}
   ngrok http 11434 --response-header-add "Access-Control-Allow-Origin: *" --host-header rewrite
   ```

4. After ngrok starts, it displays a public URL, such as `https://xxxx-xxxx.ngrok-free.app`. Use this URL as the **Base URL** when you [add a custom provider](#add-a-custom-provider) in the Playground.

The following diagram illustrates the data flow between your local environment, the ngrok proxy, and the W\&B cloud services:

```mermaid theme={null}
flowchart LR
    %% Style definitions
    classDef clientMachine fill:#FFD95CCC,stroke:#454B52,stroke-width:2px
    classDef proxy fill:#00CDDBCC,stroke:#454B52,stroke-width:2px
    classDef wandbCloud fill:#DE72FFCC,stroke:#454B52,stroke-width:2px
    classDef publicCloud fill:#FFCBADCC,stroke:#454B52,stroke-width:2px

    %% Subgraphs
    subgraph Client_Machine
        browser[Browser]
        llm_local[Local LLM Provider]
    end

    subgraph Proxy
        ngrok[Ngrok Proxy]
    end

    subgraph WandB_Cloud
        trace_server[Trace Server]
    end

    subgraph Public_Cloud
        llm_cloud[Public LLM Provider]
    end

    %% Apply styles to subgraphs
    class Client_Machine clientMachine
    class Proxy proxy
    class WandB_Cloud wandbCloud
    class Public_Cloud publicCloud

    %% Current Data Flow
    browser -->|Sends chat request| trace_server
    trace_server -->|Uses Public LLM| llm_cloud
    trace_server -->|Uses Local LLM| ngrok
    ngrok -->|Forwards to| llm_local
    llm_cloud -->|Returns response| trace_server
    llm_local -->|Returns response| ngrok
    ngrok -->|Forwards to| trace_server
    trace_server -->|Returns response| browser

    %% Future Possible Connection
    browser -.->|Future: Call local LLM directly| llm_local

    %% Link styles
    linkStyle default stroke:#454B52,stroke-width:2px
    linkStyle 8 stroke:#454B52,stroke-width:2px,stroke-dasharray:5
```

## Saved models

Saved models let you preserve a provider, model, parameters, and function hooks as a reusable preset. The following sections describe how to save, use, and update a model.

### Save a model

You can create and configure a reusable model preset for your workflow. Saving a model lets you quickly load it with your preferred settings, parameters, and function hooks.

1. In the prompt header (top of main panel), in the **Select a model** dropdown, select a provider and model.
2. In the prompt header, click the **Chat settings (<Icon icon="sliders" />)** button to open the **Chat settings** panel.
3. In the **Chat settings** panel:
   * **Model Name** (required): Enter a name for your saved model.
   * Adjust parameters as desired. You can also toggle Weave call tracking on or off, and [add a function](#add-a-function).
4. Click **Publish model**. The model is saved and accessible from **Saved Models** in the **Select a model** dropdown. You can then [use](#use-a-saved-model) and [update](#update-a-saved-model) the saved model.

### Use a saved model

Switch to a previously [saved model](#save-a-model) to maintain consistency across experiments or sessions. This way you can pick up where you left off.

1. In the prompt header, in the **Select a model** dropdown, select **Saved Models**.
2. From the list of saved models, select the saved model you want to load. The model loads and is ready to use in the Playground.

### Update a saved model

Edit an existing [saved model](#save-a-model) to fine-tune parameters or refresh its configuration. This way, your saved models evolve alongside your use cases.

1. In the prompt header, in the **Select a model** dropdown, select **Saved Models**.
2. From the list of saved models, select the saved model you want to update.
3. In the prompt header, click the **Chat settings (<Icon icon="sliders" />)** button to open the **Chat settings** panel.
4. In the **Chat settings** panel, adjust parameters as desired. You can also toggle Weave call tracking on or off, and [add a function](#add-a-function).
5. Click **Update model**. This updates the model and makes it accessible from **Saved Models** in the **Select a model** dropdown. The version of your saved model increments automatically.


# Create and manage saved views
Source: https://docs.wandb.ai/weave/guides/tools/saved-views

Customize how you interact with traced function calls and evaluations

In Weave, *saved views* let you customize how you interact with traced function calls and evaluations. By defining a saved view, you can configure filters, sorting, and column visibility to quickly access relevant data. Saved views help you and your team return to the same curated slice of your traces or evaluations without reapplying configuration each time.

You can create, modify, and save views directly in the Weave Python SDK or through the UI. The Python SDK provides fine-grained control for programmatic filtering and querying, while the UI lets you explore and save different table configurations in the **Traces** and **Evals** tabs.

This guide covers:

* [How to create and modify saved views in the Python SDK](#saved-views-in-the-python-sdk).
* [How to create and interact with saved views in the Weave UI](#saved-views-in-the-ui).

## Saved views in the Python SDK

The `SavedView` class in Weave lets you save, filter, sort, and customize views of trace and evals data. The following sections walk through how to initialize a view, configure its columns and filters, and persist it so you can retrieve matching calls later.

### Initialize a `SavedView`

Initialize a `SavedView` instance in your Weave project:

```python lines theme={null}
import weave
client = weave.init("[MY_PROJECT]")

view = weave.SavedView()
```

### Visualize the `SavedView` as a grid

Use `.to_grid()` to display the saved view as a grid. Specify the maximum number of rows to display with `limit`.

```python lines theme={null}
view.to_grid(limit=5)
```

Display the grid representation using `.show()`:

```python lines theme={null}
view.to_grid().show()
```

### Set displayed columns

Use `.set_columns()` to set the columns that display in the view. Specify one or more columns to display.

```python lines theme={null}
view.set_columns("id", "op_name")
```

### Add columns

Use `.add_column()` to add one or more new columns to the view. Specify one or more columns to add.

```python lines theme={null}
# Add a column with the field specifier and label "Created"
view.add_column("Created")
# Optionally, you can add a second argument to specify a different label name for the new column. By default, the field specifier is used for the label.
```

### Sort columns

Use `.sort_by()` to sort results based on a specific column. Specify the column name to sort and the sort order (`asc` or `desc`).

```python lines theme={null}
view.sort_by("started_at", "desc")
```

### Filter by operation name

In Weave, every trace or eval is associated with an operation name.
Use `.filter_op()` to filter the `SavedView` to only include calls where that specific operation ran.

```python lines theme={null}
view.filter_op("Evaluation.predict_and_score")
```

### Filter by operator and condition

Use `.add_filter()` to apply a custom filter to the view. Define the filter using one of the [supported filter operators](#filter-operators) and a condition.

```python lines theme={null}
view.add_filter("output.model_latency", ">=", 5)
```

#### Filter operators

| Operator         | Description                                                  | Example                                                 |
| ---------------- | ------------------------------------------------------------ | ------------------------------------------------------- |
| `"contains"`     | Checks whether a string contains a substring.                | `view.add_filter("output.status", "contains", "error")` |
| `"equals"`       | Checks whether a string is exactly equal to a given value.   | `view.add_filter("input.category", "equals", "Alice")`  |
| `"in"`           | Checks whether a string is in a list of values.              | `view.add_filter("category", "in", ["A", "B", "C"])`    |
| `"="`            | Checks whether a number is equal to a value.                 | `view.add_filter("output.score", "=", 80)`              |
| `"≠", "!="`      | Checks whether a number isn't equal to a value.              | `view.add_filter("metrics.loss", "!=", 0.5)`            |
| `"<"`            | Checks whether a number is less than a value.                | `view.add_filter("age", "<", 30)`                       |
| `"≤", "<="`      | Checks whether a number is less than or equal to a value.    | `view.add_filter("metric.value", "<=", 100)`            |
| `">"`            | Checks whether a number is greater than a value.             | `view.add_filter("output.score", ">", 90)`              |
| `"≥", ">="`      | Checks whether a number is greater than or equal to a value. | `view.add_filter("output.model_latency", ">=", 5)`      |
| `"is"`           | Checks whether a boolean field is `True` or `False`.         | `view.add_filter("is_active", "is", True)`              |
| `"after"`        | Checks whether a date is after a given timestamp.            | `view.add_filter("started_at", "after", "2024-01-01")`  |
| `"before"`       | Checks whether a date is before a given timestamp.           | `view.add_filter("ended_at", "before", "2024-12-31")`   |
| `"is empty"`     | Checks whether a field is empty (`None` or `""`).            | `view.add_filter("comments", "is empty", None)`         |
| `"is not empty"` | Checks whether a field isn't empty.                          | `view.add_filter("attachments", "is not empty", None)`  |

### Remove filters

Use `.remove_filter()` to remove a specific filter from the view by index or field name.

```python lines theme={null}
view.remove_filter("output.model_latency")
```

Use `.remove_filters()` to remove all filters.

```python lines theme={null}
view.remove_filters()
```

### Save the `SavedView`

Use `.save()` to publish the saved view to Weave.

```python lines theme={null}
view.save()
```

### Retrieve function calls

Use `.get_calls()` to retrieve function calls that match the filters in the saved view. You can specify optional parameters such as `limit` and `offset`.

```python lines theme={null}
calls = view.get_calls(limit=10)
```

## Saved views in the UI

You can create, load, rename, and edit saved views in the Weave UI. The UI is the quickest way to capture an ad hoc table configuration as a reusable view. For programmatic control, use the [Python SDK](#saved-views-in-the-python-sdk).

### Create a saved view

1. Navigate to your **Traces** or **Evals** tab.
2. Adjust any of the following variables in your table configuration:
   * Filters
   * Sort order
   * Page size
   * Column visibility
   * Column pinning
3. Save the view using one of two options:
   * In the upper-right corner, click **Save view**.
   * Click the **menu** (<Icon icon="bars" />) button to the left of **Save view**. In the dropdown menu, click **+ Save as new view**.

### Load a saved view

1. Navigate to your **Traces** or **Evals** tab.
2. Click the **menu** (<Icon icon="bars" />) button to the left of the tab title. A dropdown menu showing all saved views displays.
3. Click the view that you want to access. The saved view displays in the **Traces** or **Evals** tab.

### Rename a saved view

1. Follow the steps described in [Load a saved view](#load-a-saved-view).
2. In the upper-left corner of the **Traces** or **Evals** tab, click the view name.
3. Enter a new name for the view.
4. To save the new view name, press **Enter**.

### Edit a saved view

1. Follow the steps described in [Load a saved view](#load-a-saved-view).
2. Adjust your table configuration.
3. In the upper-right corner, click **Save view**.

### Delete a saved view

<Warning>
  You can delete a view if you believe it's no longer useful to you and your team. You can't undo this action.
</Warning>

1. Navigate to your **Traces** or **Evals** tab.
2. [Load the view](#load-a-saved-view) that you want to delete.
3. Click the **menu** (<Icon icon="bars" />) button to the left of **Save view**.
4. In the dropdown menu, click **Delete view**.
5. In the pop-up modal, confirm by clicking **Delete view**. Alternatively, click **Cancel** to stop deletion.

### Return to the default view

1. Navigate to your **Traces** or **Evals** tab.
2. Click the **menu** (<Icon icon="bars" />) button to the right of the **Traces** or **Evals** tab. A dropdown menu showing all saved views displays.
3. At the bottom of the menu, click **Traces** or **Evals**. The default view displays.


# Use Weave with W&B training runs
Source: https://docs.wandb.ai/weave/guides/tools/weave-in-workspaces

Integrate Weave traces with W&B training runs to view function execution details alongside ML metrics in workspace dashboards.

# Log traces during model training runs

You can log Weave traces during model training runs and view them in your personal workspace.

<Frame>
  <img alt="A W&B workspace showing a Weave trace panel alongside training run metrics." />
</Frame>

Personal workspaces are customizable dashboards in the UI that let you view in-depth analysis and data visualizations of your model training runs. When you add Weave panels to your workspace, you can view and access trace data logged during model runs to gain insight into how your models perform during training.

For example, if you're fine-tuning an LLM model and your W\&B workspace shows that accuracy drops at step 500, the integrated Weave traces can reveal exactly what happened. For instance, your model started generating overly verbose responses that failed the evaluation criteria.

## Use Weave panels

When you [decorate the functions with `@weave.op`](/weave/quickstart) in your ML pipelines, you automatically capture their execution information and access it in a personal workspace.

For example, the following script demonstrates how Weave traces integrate with W\&B training runs. It simulates a machine learning training loop where each training step logs metrics to W\&B while simultaneously creating detailed Weave traces. The `@weave.op` decorators on the functions automatically capture inputs, outputs, and execution details. When the training loop logs metrics with `wandb.log()`, it also logs Weave trace information to your project.

```python lines theme={null}
import wandb
import weave
import random

# Initialize both W&B and Weave with the same project
project = "my-workspace-project"
weave.init(project)

@weave.op
def evaluate_model(model_state, epoch):
    # Simulate evaluation metrics
    accuracy = 0.7 + epoch * 0.02 + random.uniform(-0.05, 0.05)
    
    # Track specific model behaviors
    test_responses = {
        "conciseness_check": "Model generated 500 words for simple question",
        "accuracy_check": "Model answered 8/10 questions correctly",
        "hallucination_check": "No factual errors detected"
    }
    
    return {
        "accuracy": accuracy,
        "diagnostic_results": test_responses
    }

@weave.op
def training_step(epoch, lr):
    # Simulate Training logic
    loss = 1.0 / (epoch + 1) + random.uniform(-0.1, 0.1)
    
    # Evaluation with traces
    eval_results = evaluate_model(f"checkpoint_{epoch}", epoch)
    
    return {
        "loss": loss,
        "accuracy": eval_results["accuracy"],
        "diagnostics": eval_results["diagnostic_results"]
    }

# Training loop
with wandb.init(project=project, config={"lr": 0.01}) as run:
    for epoch in range(5):
        # Execute training with Weave tracing
        results = training_step(epoch, run.config.lr)
        
        # Log to W&B - creates the integration point
        run.log({
            "epoch": epoch,
            "loss": results["loss"],
            "accuracy": results["accuracy"]
        })
```

You can run this example script to see how traces get logged with the rest of your run information.

After the run starts, you can view the trace information in your workspace either during or after the run. Open the link provided in the terminal at the start of a run (it looks like this: `wandb: 🚀 View run at https://wandb.ai/wandb/my-project/runs/[RUN-ID]`), or navigate to the workspace in the UI.

To navigate to a workspace from the UI:

1. Open the W\&B UI and click the **Projects** tab. This opens a list of your projects.
2. From the list of projects, click the project that you logged your run to. This opens the **Workspaces** page.
3. If you've set up your workspace as an automatic workspace, the workspace automatically populates with data visualizations and information about your run. The trace data for your run is under the **Weave** section of the workspace. If your workspace is a manual workspace, you can add Weave panels by clicking **Add panels** and selecting panels from the **Weave** section of the Add Panels menu.

For more information about workspaces, see [View experiments results](https://docs.wandb.ai/models/track/workspaces).

## Associate traces with a specific W\&B run

In some cases, you may want to attach traces to a run other than the one Weave detects automatically. By default, Weave automatically detects and associates traces with the active `wandb.run`. If you need to associate traces with a specific run that isn't the global `wandb.run`, use `set_wandb_run_context`. To associate traces with a specific run and run step, define values for the method's `run_id` and `step` arguments:

```python lines highlight={6} theme={null}
import weave

client = weave.init("[YOUR-TEAM-NAME]/[YOUR-PROJECT-NAME]")

# Associate all subsequent traces with a specific run.
client.set_wandb_run_context(run_id="[YOUR-RUN-ID]", step=[INTEGER])

@weave.op
def my_function(x):
    return x * 2

my_function(10)  # This trace is linked to "my-run-id" at step 5.
```

The `run_id` is the run's short ID (for example, `38m4t5ja`), not the full `entity/project/run_id` path. You can find a run's short ID in the run URL (`https://wandb.ai/team/project/runs/[RUN-ID]`) or by [accessing `wandb.run.id` in code](/models/runs/run-identifiers#find-and-customize-a-runs-id-or-name).

To stop the override and fall back to the global `wandb.run`, call `clear_wandb_run_context`:

```python lines theme={null}
client.clear_wandb_run_context()

my_function(20)  # This trace uses the global wandb.run (if any).
```

## Reference W\&B artifacts in a Weave trace

You can reference [W\&B artifacts](/models/artifacts) (such as models, datasets, and checkpoints) in your Weave traces. This creates a clickable link in the Weave UI that navigates directly to the artifact's details page, which helps you track which artifact versions you used during specific operations.

For example, if you've trained two versions of a model (`v1` and `v2`), you can add the `v1` artifact reference as an attribute of the trace when you query that model. This helps you trace exactly which artifact version produced a given output.

To associate an artifact with a trace, construct the artifact's URL by concatenating the `wandb-artifact:///` prefix with the artifact's full name using the following syntax: `wandb-artifact:///[FULL-NAME-OF-ARTIFACT]`. Then pass it to Weave in one of two ways:

* As an arbitrary attribute
* As an attribute of a `Model`

You can locate the artifact's full name on its details page in the W\&B UI. It looks like this: `[ENTITY]/[PROJECT]/[ARTIFACT-NAME]:[VERSION]`. You can copy the full name from the artifact's details page in the W\&B UI.

The following sections show each approach.

### Add an artifact as an attribute of a trace

To add an artifact as an attribute of a trace, pass it as a key-value pair to the `weave.attributes()` context manager:

<Tabs>
  <Tab title="Python">
    ```python lines {10} theme={null}
    import weave

    weave.init("[YOUR-TEAM-NAME]/[YOUR-PROJECT-NAME]")

    @weave.op
    def my_function(name: str):
        return f"Hello, {name}!"

    # Add an arbitrary key-name and the artifact's URL as the value
    with weave.attributes({'artifact_id': 'wandb-artifact:///team-name/project-name/run-38m4t5ja-history:v0'}):
        result = my_function("World")
    ```
  </Tab>

  <Tab title="TypeScript">
    ```typescript lines {12} theme={null}
    import {init, op, withAttributes} from 'weave';

    async function main() {
      await init('wandb/docs');

      const myFunction = op(async function myFunction(name: string) {
        return `Hello, ${name}!`;
      });

      // Add an arbitrary key-name and the artifact's URL as the value
      const result = await withAttributes(
        {artifact_id: "wandb-artifact:///team-name/project-name/run-38m4t5ja-history:v0"},
        async () => myFunction('World')
      );

      console.log('Result:', result);
    }

    main().catch(console.error);
    ```
  </Tab>
</Tabs>

### Add an artifact as an attribute of a `Model`

To add an artifact as an attribute of a `Model`, pass the artifact URL when you instantiate the model:

<Note>
  The `Model` class is only supported in the Weave Python SDK.
</Note>

```python lines {7,17} theme={null}
import weave

weave.init("[YOUR-TEAM-NAME]/[YOUR-PROJECT-NAME]")

class MyModel(weave.Model):
    # Store the artifact reference as a model attribute
    artifact_id: str
    temperature: float = 0.7
    
    @weave.op
    def predict(self, query: str) -> str:
        # Your model inference logic here
        return f"Response to: {query}"

# Pass the artifact's URL as an attribute of the model
model = MyModel(
    artifact_id="wandb-artifact:///team-name/project-name/run-38m4t5ja-history:v0"
)

result = model.predict("Hello, World!")
```


# Set up annotation queues
Source: https://docs.wandb.ai/weave/guides/tracking/annotation-queues

Create annotation queues, route traces to domain experts, and export structured feedback.

## Overview

This guide shows you how to set up annotation queues in Weave so you can collect structured feedback from domain experts on selected traces. Use annotation queues when you need expert-labeled data for evaluation, failure analysis, or dataset creation. Annotation queues also let reviewers focus on a curated set of traces rather than the full Weave UI.

Annotation queues let you route selected traces to domain experts for structured review without requiring them to navigate the full Weave UI. You define what feedback is collected, select which traces require review, and later can export completed annotations for analysis or dataset creation. Use cases can include:

* **Manual trace scoring**: Have SMEs rate model outputs on correctness, quality, or style.
* **Failure analysis**: Annotate failure modes (hallucinations, refusals, loops) to understand where your model breaks.
* **Domain expert review**: Enable medical, legal, or safety experts to review content with a task-focused interface.
* **Dataset creation**: Turn annotated traces into evaluation or training datasets.

## End-to-end workflow

The following workflow summarizes how to use annotation queues to obtain reviews. The following sections walk through each step in detail.

1. Define annotation fields.
2. Create an annotation queue.
3. Load traces into the queue for review.
4. Monitor progress while domain experts complete reviews.
5. Filter and export completed annotations.

## Define annotation fields

Before you create an annotation queue, define your annotation fields so they can be selected during queue setup. Annotation fields define the feedback that the annotator provides for each trace item. Fields are reusable across queues and projects.

Field types include:

* Boolean judgments such as correctness or acceptability.
* Numeric or integer values such as quality or confidence.
* Categorical labels such as failure mode or intent.
* Free-form text for qualitative feedback.

To create an annotation field:

1. Navigate to [wandb.ai](https://wandb.ai) and select your project.
2. In the Weave project sidebar, click **Annotate**. If you don't see **Annotate**, it might be nested in the menu under **More**.
3. In the tab bar, click the **Fields** tab.
4. In the Fields table toolbar, click **New Field**.
5. In the **Create annotation field** modal dialog, configure:
   * **Type**: Boolean, Integer, Number, String, or categorical options.
   * **Name**: Name of field to be displayed to the annotator.
   * **Description** (Optional): Details for evaluating this field to be displayed to the annotator.
6. Click **Create annotation field** to save the field.

<Frame>
  <img alt="Annotation field creation dialog with inputs for name, description, and field type, used to define the schema annotators complete when reviewing trace items." />
</Frame>

The new field appears in the **Fields** tab and is available to add to any annotation queue in the project.

<Note>
  Fields can't be edited after creation to ensure annotation consistency.
</Note>

## Create an annotation queue

With your annotation fields defined, the next step is to create the queue that annotators work from. An annotation queue consists of:

* A set of annotation fields.
* Guidelines that provide task instructions for annotators.
* A collection of trace items awaiting review.

To create an annotation queue:

1. In the Weave project sidebar, click **Annotate**.
2. In the tab bar, click the **Queues** tab.
3. In the Queues table toolbar, click **Create Queue**.
4. In the **Create Annotation Queue** modal dialog, configure:
   * **Queue name**: This is the queue name the annotator selects to complete their work.
   * **Guidelines** (Optional): Any additional instructions for the annotator.
5. Click **Next**.
6. Click **Manage fields** and choose what annotation fields to include in this review work. All existing annotation fields for the project are available for selection.
7. After you have selected all the fields for the queue, click **Create Queue** to save the queue.

All annotation queues for the project are listed in the Annotation Queues page.

<Frame>
  <img alt="Annotation Queues page displaying a table of queues in the project, including queue names, descriptions, and review progress to track annotation workflows." />
</Frame>

Creating an annotation queue defines the fields and guidelines for evaluation, but you still need to add traces to the queue to identify the data to evaluate.

### Add traces to a queue

Once a queue exists, you populate it with the specific traces you want reviewed. Add traces to an annotation queue directly from the **Traces** page.

To add traces to an annotation queue:

1. In the Weave project sidebar, click **Traces**.
2. In the Traces table toolbar, filter traces as needed (such as by hallucination scores, failure modes, or specific ops).
3. In the table, select the traces you want annotated.
4. In the table's action bar, click **Add to queue** to add the selected rows to an annotation queue.

<Frame>
  <img alt="Traces table with multiple rows selected and a bulk action bar showing actions that apply to the selected rows, such as Add to queue and Add to dataset." />
</Frame>

5. In the **Add to annotation queue** modal dialog, configure:
   * **Queue Name**: In the list, select the name of the existing queue to add these traces to.
   * **Select trace data to display**: Select the Weave trace data elements to display to the annotator as they evaluate the results.
     * **Inputs**: Select which trace input fields to show during annotation.
     * **Outputs**: Select which trace output fields to show during annotation.
6. Click **Add `[COUNT]` traces to annotation queue** to assign these traces as a part of the annotation queue review.

When you add traces, you control which trace inputs and which outputs or model responses are reviewed, so annotators see only the context needed to make accurate judgments.

## Monitor review progress

After traces are added, annotators can begin work and you can track how the review progresses. Share the queue name with your annotator so they can begin their review. See [Review items in an annotation queue](/weave/guides/tracking/annotation-review) for details on the review process.

To share a direct link to the annotation queue with an annotator:

1. In the Weave project sidebar, click **Annotate**.
2. In the tab bar, click the **Queues** tab.
3. In the Annotation Queues table, click the name of your queue to open the queue items.
4. In the Queue header bar, click the link button to copy a direct link to this queue. You can also copy the URL from the browser address bar.

<Frame>
  <img alt="Copy link to the annotation queue in the Annotation Queue header bar" />
</Frame>

In the Annotation Queues table, the **State** column indicates reviewing progress:

* **Not started**: Queue has items but no annotations have been submitted.
* **In progress**: At least one item has been reviewed.
* **Completed**: All items have been reviewed.

In the Annotation Queues table, the **Calls with responses** column indicates the percent of items (out of the total number of **Calls**) that have at least one submitted review.

## Filter and export annotations

After annotators have submitted reviews, you can use the resulting feedback for analysis, evaluation, or training. Weave stores completed annotations as structured metadata on traces.

You can:

* Filter traces by queue assignment and annotation completion.
* Save filtered views for reuse.
* Export annotated traces to datasets for evaluation or training workflows.

This connects expert human feedback directly to model evaluation and iteration.

### Filter annotated traces

You can use the filter controls of the **Traces** page to display only traces with annotations.

To view only traces with annotations:

1. In the Weave project sidebar, click **Traces**.
2. In the Traces table toolbar, click **Filter**.
3. Add three values to a filter row:
   * For **Column**, type "Queue", then press **Enter**.
   * For the second list, choose **Text: "is"**.
   * For **Select a queue**, choose your annotation queue name.
4. To also filter on 'completed' queue items only, click **+ Add Filter**:
   * For **Column**, type "feedback". A dialog populates with annotations and includes your annotation field names. Choose a required field from your queue.
   * For the second list, choose **Other: "is not empty"**.

<Frame>
  <img alt="Traces table configuring a filter to select traces belonging to specific annotation queue." />
</Frame>

5. Filter rows are automatically applied. Click elsewhere in the page for the filter entry to close.
6. Optional: Save as a view for quick access. In the Traces table header, click **Save View**.

### Export annotated traces to datasets

You can export annotated traces either through the UI or programmatically, depending on how you plan to use the data.

#### Add annotated traces to a dataset

Select annotated traces and click **Add to dataset** to include expert labels in your eval or training data.

To add annotated traces to a dataset:

1. In the Weave project sidebar, click **Traces**.
2. In the Traces table, select the traces that you want to export.
3. In the table toolbar, click **Add to dataset**. Follow the on-screen prompts to complete the addition.

To learn more about using datasets, see [Collect and track datasets](/weave/guides/core-types/datasets).

## Access annotations programmatically

In addition to the UI workflow, you can read annotation data programmatically. This is useful for automating analysis, building custom reports, or integrating annotations into downstream pipelines. To integrate annotations programmatically, you must know your project name and queue ID:

* **Project**: The W\&B project name (can be project or team/project). If you don't specify a W\&B team (such as "team/project"), your default team is used.
* **Queue ID**: The annotation queue's unique identifier.

To find the queue ID:

1. In the **Annotation Queues** table, select the name of the queue to open its items.
2. Copy the ID from the end of the page URL.

**Example**

```text theme={null}
https://wandb.ai/.../annotation-queues/019c0f63-7acb-7497-8f87-08873368fcd4
```

In this example, the queue ID is:

`019c0f63-7acb-7497-8f87-08873368fcd4`

You can iterate through the traces (calls) in your queue using the following code.

```python lines theme={null}
import weave
from weave.trace_server.trace_server_interface import AnnotationQueueItemsQueryReq

# Update project and queue identifiers to your own values.
PROJECT = "your-team-name/your-project-name"
QUEUE_ID = "019c0f63-7acb-7497-8f87-08873368fcd4"

# Initialize Weave.
client = weave.init(PROJECT)

# Get call IDs from calls in a queue.
calls = client.server.annotation_queue_items_query(
    AnnotationQueueItemsQueryReq(
        project_id=PROJECT,
        queue_id=QUEUE_ID,
    )
)

# Iterate through calls and get feedback.
for i, item in enumerate(calls.items):
    call = client.get_call(call_id=item.call_id, include_feedback=True)
    feedback = call.feedback or {}

    # Count total feedback items for call.
    total_feedback_items = len(feedback)

    print(f"\nItem {i}, call_id: {item.call_id}, total feedback items: {total_feedback_items}")

    if not feedback:
        print("No feedback for item")
        continue

    # Get the first annotation value. 
    # Field annotations are not added in repeatable order, thus the first field will vary.
    field_name = next(iter(feedback))
    field_value = feedback[0]

    print(f"  {field_name}: {field_value}")
```

The following Google Colab notebook provides an in-depth example of programmatically accessing annotation data and collecting an analytics report:

<div>
  <ColabLink />

  <GitHubLink />
</div>


# Review items in an annotation queue
Source: https://docs.wandb.ai/weave/guides/tracking/annotation-review

Evaluate trace items and submit structured feedback using a simplified review interface.

Annotation queues provide a focused review interface for domain experts. You review one item at a time, examine the provided context, and submit structured feedback using predefined fields. By completing a queue, you contribute expert judgments that your team can use to evaluate and improve LLM outputs.

You don't need to understand the underlying model or tracing system to complete reviews. This lets external experts review the data without needing broader project context.

## Annotation workflow

This section walks through the end-to-end process of reviewing items in a queue that has been shared with you.

As an annotator, your task is to review the accuracy of the LLM responses. Weave saves your feedback as annotations to the data.

To review an annotation queue:

1. Open the shared annotation queue using a queue link provided by your team.
2. Review each item. You can move forward and backward through items in the queue at any time.
3. Submit an annotation using the provided annotation fields.

Weave saves your progress automatically after you submit an item. If you need to pause and resume your work later, the review automatically resumes at the first incomplete item. After you review all items, the Annotation Queue table reflects completion of the work.

### Review a queue item

The following describes the review interface and how to submit an annotation for a single item in the queue.

For each item, the review interface shows two panes:

* A trace pane that shows selected input context (such as prompts, documents, or images) and the corresponding model response or decision.
* An item pane that contains an annotation entry form with annotation fields.

Complete each annotation field according to the instructions provided for the queue, then click **Submit and next** to save this item and continue to the next.

<Frame>
  <img alt="Annotation queue review interface for a single trace item, showing a trace information pane alongside an annotation form pane where reviewers submit structured feedback and move through the queue." />
</Frame>

<Tip>
  In the trace pane header, you can click the trace ID suffix (for example, "b3ea") to open a panel that shows the full trace details for the call, where you can inspect the trace and view or compare the annotation provided.
</Tip>

If any annotator has already reviewed an item in the queue, both pane headers of the review interface display **Has annotation** and **Response submitted** indicators.

You can still provide an annotation for an item that someone else has already reviewed. In this case, the annotation submission button still displays **Edit annotation**, but submitting creates a new annotation entry with your responses.

<Note> If you submit feedback for an item that someone else has already reviewed, your feedback is stored in addition to the original annotator's feedback. </Note>


# Call schema reference
Source: https://docs.wandb.ai/weave/guides/tracking/call-schema-reference

Reference for the Call object structure and properties

This reference describes the Call object schema in W\&B Weave, including its properties, how to write custom data to the `summary` dictionary during execution, and how to read summary data after a Call completes. Use this page when you need to inspect, filter, or extend the data Weave records for each Call. For information about querying calls, see [Query and export calls](/weave/guides/tracking/querying-calls).

## Call properties

The following table outlines the key properties of a Call in Weave. For the complete implementation, see the following:

* [class: CallSchema](/weave/reference/python-sdk/trace_server/trace_server_interface#class-callschema) in the Python SDK
* [Interface: CallSchema](/weave/reference/typescript-sdk/interfaces/callschema) in the TypeScript SDK

| Property       | Type                  | Description                                                                                        |
| -------------- | --------------------- | -------------------------------------------------------------------------------------------------- |
| `id`           | string (uuid)         | Unique identifier for the Call                                                                     |
| `project_id`   | string (optional)     | Associated project identifier                                                                      |
| `op_name`      | string                | Name of the operation (can be a reference)                                                         |
| `display_name` | string (optional)     | User-friendly name for the Call                                                                    |
| `trace_id`     | string (uuid)         | Identifier for the trace this Call belongs to                                                      |
| `parent_id`    | string (uuid)         | Identifier of the parent Call                                                                      |
| `started_at`   | datetime              | Timestamp when the Call started                                                                    |
| `attributes`   | Dict\[str, Any]       | User-defined metadata about the Call *(read-only during execution)*                                |
| `inputs`       | Dict\[str, Any]       | Input parameters for the Call                                                                      |
| `ended_at`     | datetime (optional)   | Timestamp when the Call ended                                                                      |
| `exception`    | string (optional)     | Error message if the Call failed                                                                   |
| `output`       | Any (optional)        | Result of the Call                                                                                 |
| `summary`      | Optional\[SummaryMap] | Post-execution summary information. You can modify this during execution to record custom metrics. |
| `wb_user_id`   | Optional\[str]        | Associated W\&B user ID                                                                            |
| `wb_run_id`    | Optional\[str]        | Associated W\&B run ID                                                                             |
| `deleted_at`   | datetime (optional)   | Timestamp of Call deletion, if applicable                                                          |

## Property details

`CallSchema` properties help you track and manage Calls:

* The `id`, `trace_id`, and `parent_id` properties help organize and relate Calls within the system.
* Timing information (`started_at`, `ended_at`) supports performance analysis.
* The `attributes` and `inputs` properties provide context for the Call. Attributes are frozen once the Call starts, so set them before invocation using the `weave.attributes()` context manager. `output` and `summary` capture the results.
* Use `wb_user_id` and `wb_run_id` to link the Call to a W\&B user and run.

Together, these properties support detailed tracking and analysis of Calls throughout your project.

## Use Call summary

Use the `summary` property to attach custom, post-execution data to a Call so you can analyze it later alongside Weave's built-in metrics. It's a dictionary you can write to during a Call's execution. When the Call finishes, Weave deep-merges your values with its own computed data and stores the result.

The dictionary has two zones:

* Your custom keys: anything you write to `call.summary` directly, such as `call.summary["accuracy"] = 0.95`. These sit at the top level of the summary dict.
* `summary["weave"]`: a reserved namespace Weave populates automatically at Call completion. Don't write to this key directly.

Weave also captures raw LLM token counts from the model response at `summary["usage"]` (keyed by model name). This is source data passed through from the provider, not a Weave computation. The `costs` field inside `summary["weave"]` is what Weave derives from that usage data using token pricing.

Weave's computed fields inside `summary["weave"]`:

| Field        | Description                                                                                                                        |
| ------------ | ---------------------------------------------------------------------------------------------------------------------------------- |
| `status`     | Execution status: `SUCCESS`, `ERROR`, `RUNNING`, or `DESCENDANT_ERROR` (the Call succeeded but a child Call errored).              |
| `latency_ms` | Duration in milliseconds between `started_at` and `ended_at`. `null` if `status` is `RUNNING`.                                     |
| `costs`      | Cost breakdown per model, derived from `summary["usage"]` and token pricing data. See [Track costs](/weave/guides/tracking/costs). |
| `trace_name` | Human-readable Op name, parsed from the internal Op reference URI. Used for display and filtering.                                 |

### Write during a Call

You can add custom data values during your Call using the `summary` dictionary.

<Tabs>
  <Tab title="Python">
    In Python, assign values to `call.summary` at any point during execution using `weave.get_current_call()`.

    ```python lines theme={null}
    import weave

    @weave.op()
    def my_op(x):
        result = do_work(x)
        call = weave.get_current_call()
        # Add custom data values to the summary.
        call.summary["accuracy"] = 0.95
        call.summary["num_retries"] = 2
        return result
    ```
  </Tab>

  <Tab title="TypeScript">
    In TypeScript, provide a `summarize` function as an option to `op()`. It receives the final result and returns summary data at Call completion.

    ```typescript lines theme={null}
    const myOp = weave.op(
      async (x: any) => {
        const result = await doWork(x);
        return result;
      },
      {
        name: 'my_op',
        summarize: (result) => ({
          accuracy: result.accuracy,
          numRetries: result.numRetries,
        }),
      }
    );
    ```
  </Tab>
</Tabs>

### Read summary data

Use `getCall` to fetch a single Call by ID, or `getCalls` to fetch multiple Calls. In both cases, `summary` is the same merged dictionary.

<Tabs>
  <Tab title="Python">
    ```python lines theme={null}
    import weave
    client = weave.init("my-team/my-project")

    # Fetch a single Call by ID.
    call = client.get_call("[CALL-ID]")
    weave_summary = (call.summary or {}).get("weave", {})

    print(weave_summary.get("status"))       # TraceStatus enum: SUCCESS, ERROR, RUNNING, or DESCENDANT_ERROR.
    print(weave_summary.get("latency_ms"))   # Null if the Call is still running.
    print(weave_summary.get("costs"))        # Cost breakdown per model.
    print(call.summary.get("usage"))         # Raw token counts from the LLM provider.
    print(call.summary.get("accuracy"))      # Your custom field.

    # Iterate over multiple Calls with server-side filtering.
    for call in client.get_calls(filter={"op_names": ["weave:///my-team/my-project/op/my_op:*"]}):
        s = call.summary or {}
        weave_s = s.get("weave", {})
        print(call.id, weave_s.get("status"), weave_s.get("latency_ms"), s.get("accuracy"))
    ```
  </Tab>

  <Tab title="TypeScript">
    ```typescript lines theme={null}
    import * as weave from 'weave';
    const client = await weave.init('my-team/my-project');

    // Fetch a single Call by ID.
    const call = await client.getCall('[CALL-ID]');
    const weaveSummary = call.summary?.weave;

    console.log(weaveSummary?.status);       // "success", "error", "running", or "descendant_error".
    console.log(weaveSummary?.latency_ms);   // Undefined if the Call is still running.
    console.log(weaveSummary?.costs);        // Cost breakdown per model.
    console.log(call.summary?.usage);        // Raw token counts from the LLM provider.
    console.log(call.summary?.accuracy);     // Your custom field.

    // Fetch multiple Calls with server-side filtering.
    const calls = await client.getCalls({ filter: { op_names: ['weave:///my-team/my-project/op/my_op:*'] } });
    for (const call of calls) {
      const weaveS = call.summary?.weave;
      console.log(call.id, weaveS?.status, weaveS?.latency_ms, call.summary?.accuracy);
    }
    ```
  </Tab>
</Tabs>


# Track costs
Source: https://docs.wandb.ai/weave/guides/tracking/costs

Understand automatic and custom cost tracking for LLM operations in Weave

Weave tracks the cost of LLM calls in two ways:

* **Automatic cost tracking**: For [supported integrations](/weave/guides/integrations), Weave captures token usage from the API response and applies built-in pricing for the model, with no extra code required.
* **Custom cost tracking**: For fine-tuned models, self-hosted models, or models that Weave doesn't automatically integrate with, configure Weave to track custom costs using available API methods.

<Note>
  The Weave TypeScript doesn't support cost tracking.
</Note>

## Use automatic cost tracking

When you call `weave.init()` and use a [supported LLM integration](/weave/guides/integrations) such as OpenAI, Anthropic, Cohere, or Mistral, Weave automatically records token usage and calculates cost for each call. Costs appear in the trace tree and in the calls table in the Weave UI. They're also available programmatically under `call.summary["weave"]["costs"]` when you query calls with the `include_costs` parameter set to `true`.

Automatic cost tracking requires two conditions:

* The LLM provider is a supported integration.
* The API response includes token usage (most providers return this by default).

If either condition isn't met, use custom cost tracking instead.

The following example shows how to retrieve automatic cost data programmatically:

```python lines theme={null}
import weave
from openai import OpenAI

weave.init("your-team/project-name")
client = OpenAI()

response = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": "What is 2 + 2?"}],
)

import time
time.sleep(2)

# Retrieve the current instance of the Weave client
weave_client = weave.get_client()
# Access the calls for the instance and their costs
calls = list(weave_client.get_calls(include_costs=True, limit=1))
call = calls[0]

# Access the call summary and retrieve the available costs fields
costs = call.summary.get("weave", {}).get("costs", {})
if not costs:
    print("No costs found in summary.weave.costs")
for model, cost in costs.items():
    print(f"Model: {model}")
    print(f"  Input cost:  ${cost['prompt_tokens_total_cost']:.6f}")
    print(f"  Output cost: ${cost['completion_tokens_total_cost']:.6f}")
```

## Add a custom cost

Use custom cost tracking when automatic cost tracking isn't available, such as with fine-tuned models, self-hosted models, or providers that Weave doesn't integrate with. The following sections describe how to add, query, purge, and aggregate custom costs.

Add a custom cost with the [`add_cost`](/weave/reference/python-sdk/trace/weave_client#method-add-cost) method.
The three required fields are `llm_id`, `prompt_token_cost`, and `completion_token_cost`.
`llm_id` is the name of the LLM (for example, `gpt-4o`). `prompt_token_cost` and `completion_token_cost` are the cost per token for the LLM. If the LLM prices are specified per million tokens, convert the value.
You can also set `effective_date` to a datetime to make the cost effective at a specific date. This defaults to the current date.

```python lines theme={null}
import weave
from datetime import datetime

client = weave.init("your-team/project-name")

client.add_cost(
    llm_id="your_model_name",
    prompt_token_cost=0.01,
    completion_token_cost=0.02
)

client.add_cost(
    llm_id="your_model_name",
    prompt_token_cost=10,
    completion_token_cost=20,
    effective_date=datetime(2025, 4, 22),
)
```

### Query custom costs

After adding custom costs, you can retrieve them to verify their values or to look up the cost ID needed for other operations such as purging.

Query for costs with the [`query_costs`](/weave/reference/python-sdk/trace/weave_client#method-query-costs) method.
You can query for costs in a few ways: pass in a single cost ID, or a list of LLM model names.

```python lines theme={null}
import weave

client = weave.init("your-team/project-name")

costs = client.query_costs(llm_ids=["your_model_name"])

cost = client.query_costs(costs[0].id)
```

### Purge a custom cost

Purge a custom cost with the [`purge_costs`](/weave/reference/python-sdk/trace/weave_client#method-purge-costs) method. Pass in a list of cost IDs, and Weave purges the costs with those IDs.

```python lines theme={null}
import weave

client = weave.init("your-team/project-name")

costs = client.query_costs(llm_ids=["your_model_name"])
client.purge_costs([cost.id for cost in costs])
```

### Calculate custom costs for a project

To understand the total spend across a project, aggregate the per-call costs returned by Weave. Calculate costs for a project with `get_calls()` and `include_costs=True`.

```python lines theme={null}
import weave

weave.init("your-team/project-name")

@weave.op()
def get_costs_for_project(project_name: str):
    total_cost = 0
    requests = 0

    client = weave.init(project_name)
    calls = list(
        client.get_calls(filter={"trace_roots_only": True}, include_costs=True)
    )

    for call in calls:
        if call.summary["weave"] is not None and call.summary["weave"].get("costs", None) is not None:
            for k, cost in call.summary["weave"]["costs"].items():
                requests += cost["requests"]
                total_cost += cost["prompt_tokens_total_cost"]
                total_cost += cost["completion_tokens_total_cost"]

    return {
        "total_cost": total_cost,
        "requests": requests,
        "calls": len(calls),
    }

get_costs_for_project("my_custom_cost_model")
```

### Set up a custom model with custom costs

For an end-to-end walkthrough that combines a custom model with custom cost tracking, try the cookbook for [setting up costs with a custom model](/weave/cookbooks/custom_model_cost).

<Card title="Try in Colab" href="https://colab.research.google.com/github/wandb/weave/blob/master/docs/./notebooks/custom_model_cost.ipynb" icon="python" />


# Trace your code
Source: https://docs.wandb.ai/weave/guides/tracking/create-call

Instrument your running code so its execution appears as detailed traces in W&B Weave.

This guide shows you how to instrument your application so that its execution appears as detailed traces in W\&B Weave. Traces help you debug, evaluate, and monitor LLM-powered code by capturing inputs, outputs, and the structure of each operation as a Call.

To see your running code as detailed traces in Weave, you create Calls. The following sections describe the three main ways to do this, from least to most manual:

* Automatic tracking of LLM library calls.
* Tracking of custom functions with `weave.op`.
* Manual Call tracking using the API directly.

Choose the approach that fits how much control you need over what gets traced.

## Automatic tracking of LLM library calls

Weave integrates automatically with many common libraries and frameworks, such as `openai`, `anthropic`, `cohere`, `mistral`, and `LangChain`. Import the LLM or framework library and initialize your Weave project. Weave then automatically traces all Calls made to the LLM or platform to your project without any additional code changes. For a complete list of supported library integrations, see [Integrations overview](/weave/guides/integrations/).

<Tabs>
  <Tab title="Python">
    ```python lines theme={null}
    import weave

    from openai import OpenAI
    client = OpenAI()

    # Initialize Weave Tracing
    weave.init('intro-example')

    response = client.chat.completions.create(
        model="gpt-4",
        messages=[
            {
                "role": "user",
                "content": "How are you?"
            }
        ],
        temperature=0.8,
        max_tokens=64,
        top_p=1,
    )
    ```
  </Tab>

  <Tab title="TypeScript">
    ```typescript lines theme={null}
    import OpenAI from 'openai'
    import * as weave from 'weave'

    const client = new OpenAI()

    // Initialize Weave Tracing
    await weave.init('intro-example')

    const response = await client.chat.completions.create({
      model: 'gpt-4',
      messages: [
        {
          role: 'user',
          content: 'How are you?',
        },
      ],
      temperature: 0.8,
      max_tokens: 64,
      top_p: 1,
    });
    ```

    For a complete setup guide for JavaScript and TypeScript projects, see the [TypeScript SDK: Third-Party Integration Guide](/weave/guides/integrations/js).
  </Tab>
</Tabs>

If you want more control over automatic behavior, see [Configure automatic LLM call tracking](/weave/guides/integrations/autopatching).

## Track custom functions

LLM applications often have additional logic, such as pre-processing and post-processing, prompts, and more, that you want to track. Use this approach when you want Weave to capture your own functions alongside the LLM calls it traces automatically.

<Tabs>
  <Tab title="Python">
    Weave lets you manually track these Calls using the [`@weave.op`](/weave/reference/python-sdk/#function-op) decorator. For example:

    ```python lines theme={null}
    import weave

    # Initialize Weave Tracing
    weave.init('intro-example')

    # Decorate your function
    @weave.op
    def my_function(name: str):
        return f"Hello, {name}!"

    # Call your function -- Weave will automatically track inputs and outputs
    print(my_function("World"))
    ```

    You can also track [methods on classes](#track-class-and-object-methods).
  </Tab>

  <Tab title="TypeScript">
    Weave lets you manually track these Calls by wrapping your function with [`weave.op`](/weave/reference/typescript-sdk/functions/op). For example:

    ```typescript lines theme={null}
    import * as weave from 'weave'

    await weave.init('intro-example')

    function myFunction(name: string) {
        return `Hello, ${name}!`
    }

    const myFunctionOp = weave.op(myFunction)
    ```

    You can also define the wrapping inline:

    ```typescript lines theme={null}
    const myFunctionOp = weave.op((name: string) => `Hello, ${name}!`)
    ```

    This works for both functions and methods on classes:

    ```typescript lines theme={null}
    class MyClass {
        constructor() {
            this.myMethod = weave.op(this.myMethod)
        }

        myMethod(name: string) {
            return `Hello, ${name}!`
        }
    }
    ```
  </Tab>
</Tabs>

### Track class and object methods

In addition to standalone functions, you can track class and object methods. You can track any method in a class by decorating the method with `weave.op`.

<Tabs>
  <Tab title="Python">
    ```python lines theme={null}
    import weave

    # Initialize Weave Tracing
    weave.init("intro-example")

    class MyClass:
        # Decorate your method
        @weave.op
        def my_method(self, name: str):
            return f"Hello, {name}!"

    instance = MyClass()

    # Call your method -- Weave will automatically track inputs and outputs
    print(instance.my_method("World"))
    ```
  </Tab>

  <Tab title="TypeScript">
    <Important>
      **Using decorators in TypeScript**

      To use the `@weave.op` decorator with your TypeScript code, make sure your environment is properly configured:

      * **TypeScript v5.0 or newer**: Decorators are supported out of the box and no additional configuration is required.
      * **TypeScript older than v5.0**: Enable experimental support for decorators. For more details, see the [official TypeScript documentation on decorators](https://www.typescriptlang.org/docs/handbook/decorators.html).
    </Important>

    You can apply `@weave.op` to instance methods for tracing.

    ```typescript lines theme={null}
    class Foo {
        @weave.op
        async predict(prompt: string) {
            return "bar"
        }
    }
    ```

    You can also apply `@weave.op` to static methods to monitor utility functions within a class.

    ```typescript lines theme={null}
    class MathOps {
        @weave.op
        static square(n: number): number {
            return n * n;
        }
    }
    ```
  </Tab>
</Tabs>

### Trace parallel (multi-threaded) function calls

By default, parallel Calls all appear in Weave as separate root Calls, which makes the trace hierarchy hard to follow. To get correct nesting under the same parent Op, use a `ThreadPoolExecutor`.

<Tabs>
  <Tab title="Python">
    The following code sample demonstrates the use of [`ThreadPoolExecutor`](/weave/reference/python-sdk/trace/util#class-contextawarethreadpoolexecutor).
    The first function, `func`, is a simple Op that takes `x` and returns `x+1`. The second function, `outer`, is another Op that accepts a list of inputs.
    Inside `outer`, the use of `ThreadPoolExecutor` and `exc.map(func, inputs)` means that each call to `func` still carries the same parent trace context.

    ```python lines theme={null}
    import weave

    @weave.op
    def func(x):
        return x+1

    @weave.op
    def outer(inputs):
        with weave.ThreadPoolExecutor() as exc:
            exc.map(func, inputs)

    # Update your Weave project name
    client = weave.init('my-weave-project')
    outer([1,2,3,4,5])
    ```
  </Tab>

  <Tab title="TypeScript">
    ```text theme={null}
    This feature is not available in the TypeScript SDK yet.
    ```
  </Tab>
</Tabs>

In the Weave UI, this produces a single parent Call with five nested child Calls. You get a fully hierarchical trace even though the increments run in parallel.

<img alt="The Trace UI, showing a single parent Call for outer, with five nested child Calls." />

## Manual Call tracking

If neither automatic integration nor the `weave.op` decorator fits your workflow, you can manually create Calls using the API directly. This approach gives you full control over when a Call starts and ends, at the cost of more boilerplate.

<Tabs>
  <Tab title="Python">
    ```python lines theme={null}
    import weave

    # Initialize Weave Tracing
    client = weave.init('intro-example')

    def my_function(name: str):
        # Start a Call
        call = client.create_call(op="my_function", inputs={"name": name})

        # ... your function code ...

        # End a Call
        client.finish_call(call, output="Hello, World!")

        # Call your function
        print(my_function("World"))
    ```
  </Tab>

  <Tab title="TypeScript">
    ```text theme={null}
    This feature is not available in the TypeScript SDK yet.
    ```
  </Tab>

  <Tab title="HTTP API">
    * Start a Call: [POST `/call/start`](https://docs.wandb.ai/weave/reference/service-api/calls/call-start).
    * End a Call: [POST `/call/end`](https://docs.wandb.ai/weave/reference/service-api/calls/call-end).

    ```bash lines theme={null}
    curl -L 'https://trace.wandb.ai/call/start' \
    -H 'Content-Type: application/json' \
    -H 'Accept: application/json' \
    -d '{
        "start": {
            "project_id": "string",
            "id": "string",
            "op_name": "string",
            "display_name": "string",
            "trace_id": "string",
            "parent_id": "string",
            "started_at": "2024-09-08T20:07:34.849Z",
            "attributes": {},
            "inputs": {},
            "wb_run_id": "string"
        }
    }
    ```
  </Tab>
</Tabs>


# Collect feedback and use annotations
Source: https://docs.wandb.ai/weave/guides/tracking/feedback

Collect and analyze feedback for LLM applications through UI and SDK

Evaluating LLM applications requires tooling to collect and analyze feedback. W\&B Weave provides an integrated feedback system that lets you provide Call feedback directly through the UI or programmatically through the SDK. Weave supports several feedback types, including emoji reactions, textual comments, and structured data, so your team can:

* Build evaluation datasets for performance monitoring.
* Identify and resolve LLM content issues.
* Gather examples for advanced tasks like fine-tuning.

This guide is for developers and reviewers working with LLM applications in Weave. It covers how to use Weave's feedback functionality in both the UI and SDK, query and manage feedback, and use human annotations for detailed evaluations.

* [Provide feedback in the UI](#provide-feedback-in-the-ui)
* [Provide feedback through the SDK](#provide-feedback-through-the-sdk)
* [Add human annotations](#add-human-annotations)

## Provide feedback in the UI

The following sections describe how to provide feedback in the Weave UI, either from the Call details panel or using the feedback icons.

### Use the Call details panel

1. In the Weave project sidebar, navigate to **Traces**.
2. Find the row for the Call that you want to add feedback to.
3. Click the linked Trace name to open the trace tree and Call details panel.
4. In the Call details tab bar, select **Feedback**.
5. Add, view, or delete feedback:
   * *Add and view feedback using the icons* located in the upper right corner of the Call details feedback view.
   * *View and delete feedback from the Call details feedback table.* Delete feedback by clicking the trashcan icon in the rightmost column of the appropriate feedback row.

<Frame>
  <img alt="Feedback tab in Call details" />
</Frame>

### Use the feedback icons

You can add or remove a reaction, and add a note using the icons that are located in both the Traces table and individual Call details panel.

* *Traces table*: Located in **Feedback** column in the appropriate row in the **Traces** table.
* *Call details panel*: Located in the upper right corner of each Call details panel.

To add a reaction:

1. Click the emoji icon.
2. Add a thumbs up, thumbs down, or click the **+** icon for more emojis.

To remove a reaction:

1. Hover over the emoji reaction you want to remove.
2. Click the reaction to remove it.

> You can also delete feedback from the [**Feedback** column on the Call details panel](#use-the-call-details-panel).

To add a comment:

1. Click the comment bubble icon.
2. In the text box, add your note. The maximum number of characters in a feedback note is 1024.
3. To save the note, press the **Enter** key. You can add more notes.

<Warning>
  The maximum number of characters in a feedback note is 1024. If a note exceeds this limit, Weave doesn't create it.
</Warning>

<Frame>
  <img alt="Calls grid with feedback column" />
</Frame>

## Provide feedback through the SDK

Use the SDK when you want to automate feedback collection or integrate it into evaluation pipelines, rather than entering feedback by hand in the UI.

You can find SDK usage examples for feedback in the UI under the **Use** tab in the Call details panel.

You can use the Weave Python SDK to programmatically [add](/weave/reference/python-sdk/trace/feedback#method-add), [remove](/weave/reference/python-sdk/trace/feedback#method-purge), and [query feedback](/weave/reference/python-sdk/trace/weave_client#method-get_feedback) on calls. The TypeScript SDK does not support feedback functionality.

### Query a project's feedback

You can query the feedback for your Weave project using the SDK. The SDK supports the following feedback query operations:

* `client.get_feedback()`: Returns all feedback in a project.
* `client.get_feedback("[FEEDBACK-UUID]")`: Returns a specific feedback object specified by `[FEEDBACK-UUID]` as a collection.
* `client.get_feedback(reaction="[REACTION-TYPE]")`: Returns all feedback objects for a specific reaction type.

You can also get more information for each feedback object in `client.get_feedback()`:

* `id`: The feedback object ID.
* `created_at`: The creation time information for the feedback object.
* `feedback_type`: The type of feedback (reaction, note, custom).
* `payload`: The feedback payload.

<Tabs>
  <Tab title="Python">
    ```python lines theme={null}
    import weave
    client = weave.init('intro-example')

    # Get all feedback in a project
    all_feedback = client.get_feedback()

    # Fetch a specific feedback object by id.
    # The API returns a collection, which is expected to contain at most one item.
    one_feedback = client.get_feedback("[FEEDBACK-UUID]")[0]

    # Find all feedback objects with a specific reaction. You can specify offset and limit.
    thumbs_up = client.get_feedback(reaction="👍", limit=10)

    # After retrieval, view the details of individual feedback objects.
    for f in client.get_feedback():
        print(f.id)
        print(f.created_at)
        print(f.feedback_type)
        print(f.payload)
    ```
  </Tab>

  <Tab title="TypeScript">
    ```plaintext theme={null}
    This feature is not available in TypeScript yet.
    ```
  </Tab>
</Tabs>

### Add feedback to a Call

You can add feedback to a Call using the Call's UUID. To use the UUID to get a particular Call, [retrieve it during or after Call execution](#retrieve-the-call-uuid). The SDK supports the following operations for adding feedback to a Call:

* `call.feedback.add_reaction("[REACTION-TYPE]")`: Add one of the supported `[REACTION-TYPE]` values (emojis), such as 👍.
* `call.feedback.add_note("[NOTE]")`: Add a note.
* `call.feedback.add("[LABEL]", [OBJECT])`: Add a custom feedback `[OBJECT]` specified by `[LABEL]`.

<Warning>
  The maximum number of characters in a feedback note is 1024. If a note exceeds this limit, Weave doesn't create it.
</Warning>

<Tabs>
  <Tab title="Python">
    ```python lines theme={null}
    import weave
    client = weave.init('intro-example')

    call = client.get_call("[CALL-UUID]")

    # Adding an emoji reaction
    call.feedback.add_reaction("👍")

    # Adding a note
    call.feedback.add_note("this is a note")

    # Adding custom key/value pairs.
    # The first argument is a user-defined "type" string.
    # Feedback must be JSON serializable and less than 1 KB when serialized.
    call.feedback.add("correctness", { "value": 5 })
    ```
  </Tab>

  <Tab title="TypeScript">
    ```plaintext theme={null}
    This feature is not available in TypeScript yet.
    ```
  </Tab>
</Tabs>

#### Retrieve the Call UUID

For scenarios where you must add feedback immediately after a Call, you can retrieve the Call UUID programmatically during or after the Call execution.

##### During Call execution

To retrieve the UUID during Call execution, get the current Call, and return the ID.

<Tabs>
  <Tab title="Python">
    ```python lines theme={null}

    import weave
    weave.init("uuid")

    @weave.op()
    def simple_operation(input_value):
        # Perform some simple operation
        output = f"Processed {input_value}"
        # Get the current call ID
        current_call = weave.require_current_call()
        call_id = current_call.id
        return output, call_id
    ```
  </Tab>

  <Tab title="TypeScript">
    ```plaintext theme={null}
    This feature is not available in TypeScript yet.
    ```
  </Tab>
</Tabs>

##### After Call execution

Alternatively, you can use the `call()` method to execute the operation and retrieve the ID after Call execution:

<Tabs>
  <Tab title="Python">
    ```python lines theme={null}
    import weave
    weave.init("uuid")

    @weave.op()
    def simple_operation(input_value):
        return f"Processed {input_value}"

    # Execute the operation and retrieve the result and call ID
    result, call = simple_operation.call("example input")
    call_id = call.id
    ```
  </Tab>

  <Tab title="TypeScript">
    ```plaintext theme={null}
    This feature is not available in TypeScript yet.
    ```
  </Tab>
</Tabs>

### Delete feedback from a Call

You can delete feedback from a particular call by specifying a UUID.

<Tabs>
  <Tab title="Python">
    ```python lines theme={null}
    call.feedback.purge("[FEEDBACK-UUID]")
    ```
  </Tab>

  <Tab title="TypeScript">
    ```plaintext theme={null}
    This feature is not available in TypeScript yet.
    ```
  </Tab>
</Tabs>

## Add human annotations

Human annotations let you capture structured, human-reviewed judgments about Calls so that reviewers can score model output against your own criteria.

Human annotations are supported in the Weave UI. This functionality lets you create custom fields to add human-entered data to your Traces as feedback. To make human annotations, you must first create a Human Annotation scorer using either the [UI](#create-a-human-annotation-scorer-in-the-ui) or the [API](#create-a-human-annotation-scorer-using-the-api). Then, you can [use the scorer in the UI to make annotations](#use-the-human-annotation-scorer-in-the-ui), and [modify your annotation scorers using the API](#modify-a-human-annotation-scorer-using-the-api).

### Create a human annotation scorer in the UI

To create a human annotation scorer in the UI, do the following:

1. In the project sidebar, navigate to **Assets**.
2. In the Assets navigation panel, click **Scorers**.
3. In the **Scorers** panel header, click **New scorer**.
4. In the **Create Scorer** modal dialog, set:
   * `Scorer type` to `Human annotation`
   * `Name`
   * `Description`
   * `Type`, which determines the type of feedback to collect, such as `boolean` or `integer`.
5. Click **Create scorer**. Now, you can use your scorer to make annotations.

In the following example, a human annotator selects which type of document the LLM loaded. The `Type` for the score configuration is an `enum` that contains the possible document types.

<Frame>
  <img alt="Create Scorer modal dialog" />
</Frame>

### Use the human annotation scorer in the UI

After you create a human annotation scorer, it becomes available to use on the Traces page.

To use the scorer, do the following:

1. In the project sidebar, navigate to **Traces**.

2. Find the row for the Call that you want to add a human annotation to.

3. Click the linked Trace name to open the trace tree and Call details panel.

4. In the upper right corner of the Call details tab bar, click the **Show feedback** button.

   <img alt="Marker icon in Call header" />

   Your available human annotation scorers display in an **Annotate** panel.

   <img alt="Human Annotation scorer feedback panel" />

5. Make an annotation.

6. Click **Save**.

7. In the Call details panel tab bar, click the **Feedback** tab to view the Feedback table. The new annotation displays in the table. You can also view the annotations in the **Annotations** column in the main Traces table.

   > Refresh the Traces table to view the most up-to-date information.

<Frame>
  <img alt="Human Annotation scorer feedback in Traces table" />
</Frame>

### Create a human annotation scorer using the API

You can also create human annotation scorers through the API. Each scorer is its own object, which you create and update independently. To create a human annotation scorer programmatically, do the following:

1. Import the `AnnotationSpec` class from `weave.flow.annotation_spec`.
2. Use the `publish` method from `weave` to create the scorer.

The following example creates two scorers. The first scorer, `Temperature`, scores the perceived temperature of the LLM call. The second scorer, `Tone`, scores the tone of the LLM response. Each scorer uses `save` with an associated object ID (`temperature-scorer` and `tone-scorer`).

<Tabs>
  <Tab title="Python">
    ```python lines theme={null}
    import weave
    from weave.flow.annotation_spec import AnnotationSpec

    client = weave.init("feedback-example")

    spec1 = AnnotationSpec(
      name="Temperature",
      description="The perceived temperature of the llm call",
      field_schema={
        "type": "number",
        "minimum": -1,
        "maximum": 1,
      }
    )
    spec2 = AnnotationSpec(
      name="Tone",
      description="The tone of the llm response",
      field_schema={
        "type": "string",
        "enum": ["Aggressive", "Neutral", "Polite", "N/A"],
      },
    )
    weave.publish(spec1, "temperature-scorer")
    weave.publish(spec2, "tone-scorer")
    ```
  </Tab>

  <Tab title="TypeScript">
    ```plaintext theme={null}
    This feature is not available in TypeScript yet.
    ```
  </Tab>
</Tabs>

### Modify a human annotation scorer using the API

Expanding on [creating a human annotation scorer using the API](#create-a-human-annotation-scorer-using-the-api), the following example creates an updated version of the `Temperature` scorer, by using the original object ID (`temperature-scorer`) on `publish`. The result is an updated object, with a history of all versions.

> You can view human annotation scorer object history in the **Scorers** tab under **Human annotations**.

<Tabs>
  <Tab title="Python">
    ```python lines theme={null}
    import weave
    from weave.flow.annotation_spec import AnnotationSpec

    client = weave.init("feedback-example")

    # create a new version of the scorer
    spec1 = AnnotationSpec(
      name="Temperature",
      description="The perceived temperature of the llm call",
      field_schema={
        "type": "integer",  # <<- change type to integer
        "minimum": -1,
        "maximum": 1,
      }
    )
    weave.publish(spec1, "temperature-scorer")
    ```
  </Tab>

  <Tab title="TypeScript">
    ```plaintext theme={null}
    This feature is not available in TypeScript yet.
    ```
  </Tab>
</Tabs>

<Frame>
  <img alt="Human Annotation scorer history" />
</Frame>

### Use a human annotation scorer using the API

The feedback API lets you use a human annotation scorer by specifying a specially constructed name and an `annotation_ref` field. You can obtain the `annotation_spec_ref` from the UI by selecting the appropriate tab, or during the creation of the `AnnotationSpec`.

<Tabs>
  <Tab title="Python">
    ```python lines theme={null}
    import weave

    client = weave.init("feedback-example")

    call = client.get_call("[CALL-ID]")
    annotation_spec = weave.ref("[ANNOTATION-SPEC-REF-URI]")

    call.feedback.add(
      feedback_type="wandb.annotation." + annotation_spec.name,
      payload={"value": 1},
      annotation_ref=annotation_spec.uri(),
    )
    ```
  </Tab>
</Tabs>


# Get a handle to the Call object during execution
Source: https://docs.wandb.ai/weave/guides/tracking/get-call-object

Access the W&B Weave `Call` object at runtime for feedback, display names, and other metadata

This guide shows how to get a handle to the Weave `Call` object while your code runs, so you can inspect inputs and outputs, update metadata, or attach feedback at runtime. It's intended for developers who already use Weave Ops and want to interact with the underlying `Call` object programmatically.

When you use an Op in Weave, you can call it directly as you would any function:

<CodeGroup>
  ```python Python lines theme={null}
  @weave.op
  def my_op():
      ...

  my_op()
  ```

  ```typescript Typescript lines theme={null}
  function myFunction() {
      ...
  }

  const myFunctionOp = weave.op(myFunction)
  ```
</CodeGroup>

To access the `Call` object directly, invoke the `op.call` method, which returns both the result and the `Call` object:

<Tabs>
  <Tab title="Python">
    ```python lines theme={null}
    @weave.op
    def my_op():
    ...

    output, call = my_op.call()
    ```

    The `call` object contains all the information about the `Call`, including the inputs, outputs, and other metadata. Use `call` to set, update, fetch additional properties, or add feedback.

    If your Op is a method on a class, pass the class instance as the first argument to `call`:

    ```python lines theme={null}
    import weave

    # Initialize Weave Tracing
    weave.init("intro-example")

    class MyClass:
        # Decorate your method
        @weave.op
        def my_method(self, name: str):
            return f"Hello, {name}!"

    instance = MyClass()

    # Pass `instance` as the first argument to `call`.
    result, call = instance.my_method.call(instance, "World")
    ```
  </Tab>

  <Tab title="TypeScript">
    ```text theme={null}
    This feature is not available in the TypeScript SDK yet.
    ```
  </Tab>
</Tabs>


# Track and version objects
Source: https://docs.wandb.ai/weave/guides/tracking/objects

Track and version any JSON-serializable object in W&B Weave

This page explains how to publish, retrieve, delete, and reference versioned objects in W\&B Weave. Use it when you need to track structured data such as datasets, models, or prompts across runs and over time.

## Objects

An **Object** is versioned, serializable data. Weave automatically versions objects when they change and creates an immutable history. Objects include:

* **Datasets**: Collections of examples for evaluation
* **Models**: Configurations and parameters for your LLM logic
* **Prompts**: Versioned prompt templates

```python lines theme={null}
dataset = weave.Dataset(
    name="test-cases",
    rows=[
        {"input": "What is 2+2?", "expected": "4"},
        {"input": "What is the capital of France?", "expected": "Paris"},
    ]
)
weave.publish(dataset)
```

## Publish an object

Weave's serialization layer saves and versions objects.

<Tabs>
  <Tab title="Python">
    ```python lines theme={null}
    import weave
    weave.init("your-team-name/your-project-name")
    # Save a list, giving it the name 'cat-names'
    weave.publish(['felix', 'jimbo', 'billie'], 'cat-names')
    ```
  </Tab>

  <Tab title="TypeScript">
    TypeScript publishing support is limited. Not all objects are supported.

    ```typescript lines theme={null}
    import * as weave from 'weave'

    const client = await weave.init("your-team-name/your-project-name")

    // Save an array, giving it the name 'cat-names'
    client.publish(['felix', 'jimbo', 'billie'], 'cat-names')
    ```
  </Tab>
</Tabs>

When you save an object with a name, Weave creates the first version of that object if it doesn't exist.

## Get an object back

After publishing, you can fetch a stored object using its reference.

<Tabs>
  <Tab title="Python">
    `weave.publish()` returns a Ref. Call `.get()` on any Ref to get the object back.

    You can construct a ref and then fetch the object back.

    ```python lines theme={null}
    weave.init("your-team-name/your-project-name")
    cat_names = weave.ref('cat-names').get()
    ```
  </Tab>

  <Tab title="TypeScript">
    ```plaintext lines theme={null}
    This feature isn't available in TypeScript yet.
    ```
  </Tab>
</Tabs>

## Delete an object

If you no longer need a specific version of an object, you can remove it using its ref.

<Tabs>
  <Tab title="Python">
    To delete a version of an object, call `.delete()` on the object's ref.

    ```python lines theme={null}
    weave.init("your-team-name/your-project-name")
    cat_names_ref = weave.ref('cat-names:v1')
    cat_names_ref.delete()
    ```

    Accessing a deleted object returns an error. Resolving an object that references a deleted object returns a `DeletedRef` in place of the deleted object.
  </Tab>

  <Tab title="TypeScript">
    ```plaintext theme={null}
    This feature isn't available in TypeScript yet.
    ```
  </Tab>
</Tabs>

## Construct object refs

Refs uniquely identify a stored object and version. The following sections describe the URI structure and the ways you can construct a ref.

In Weave, a fully qualified object ref URI looks like this:

```text theme={null}
weave:///[YOUR-TEAM-NAME]/[YOUR-PROJECT-NAME]/object/[OBJECT-NAME]:[OBJECT-VERSION]
```

* `[YOUR-TEAM-NAME]`: W\&B entity (username or team name)
* `[YOUR-PROJECT-NAME]`: W\&B project
* `[OBJECT-NAME]`: object name
* `[OBJECT-VERSION]`: either a version hash, a string like `v0` or `v1`, or an alias like `:latest`. All objects have the `:latest` alias.

You can construct refs with a few different styles:

* `weave.ref([NAME])`: retrieves the `:latest` version of an object. Requires calling `weave.init(...)`.
* `weave.ref([NAME]:[VERSION])`: retrieves the specified version of an object. Requires calling `weave.init(...)`.
* `weave.ref([FULLY-QUALIFIED-REF-URI])`: retrieves the object located at the specified fully qualified object ref URI. Doesn't require calling `weave.init()`.


# Customize Ops
Source: https://docs.wandb.ai/weave/guides/tracking/ops

Learn how to color your Ops for better visibility, how to modify what's logged, and how to control the sampling rate

A Weave Op is a versioned function that automatically logs all Calls. This guide is for developers who use Weave to instrument their code, and shows you how to create Ops, customize how they appear in the Weave UI, control what data Weave logs, and manage sampling and deletion.

<Tabs>
  <Tab title="Python">
    To create an Op, decorate a Python function with `weave.op()`.

    ```python lines theme={null}
    import weave

    @weave.op()
    def track_me(v):
        return v + 5

    weave.init('intro-example')
    track_me(15)
    ```

    Calling an Op creates a new Op version if the code changed from the last call, and logs the function's inputs and outputs.

    Functions that you decorate with `@weave.op()` behave normally (without code versioning and tracking) if you don't call `weave.init('your-project-name')` before calling them.

    You can [serve](/weave/guides/tools/serve) or [deploy](/weave/guides/tools/deploy) Ops using the Weave toolbelt.
  </Tab>

  <Tab title="TypeScript">
    To create an Op, wrap a TypeScript function with `weave.op`.

    ```typescript lines theme={null}
    import * as weave from 'weave'

    function trackMe(v: number) {
        return v + 5
    }

    const trackMeOp = weave.op(trackMe)
    trackMeOp(15)

    // You can also do this inline, which may be more convenient
    const trackMeInline = weave.op((v: number) => v + 5)
    trackMeInline(15)
    ```
  </Tab>
</Tabs>

## Customize display names

A custom display name makes it easier to identify an Op in the Weave UI, especially when the function name is generic or auto-generated.

<Tabs>
  <Tab title="Python">
    To customize the Op's display name, set the `name` parameter in the `@weave.op` decorator:

    ```python lines theme={null}
    @weave.op(name="custom_name")
    def func():
        ...
    ```
  </Tab>

  <Tab title="TypeScript">
    ```text theme={null}
    This feature is not available in TypeScript yet.
    ```
  </Tab>
</Tabs>

## Apply kinds and colors

To better organize your Ops in the Weave UI, apply custom kinds and colors by adding the `kind` and `color` arguments to the `@weave.op` decorators in your code. For example, the following code applies an `LLM` `kind` and a `blue` `color` to the parent function, and a `tool` `kind` and a `red` `color` to a nested function:

<Tabs>
  <Tab title="Python">
    ```python lines theme={null}
    import weave

    weave.init("[YOUR-TEAM-NAME]/[YOUR-PROJECT-NAME]")

    @weave.op(kind="LLM", color="blue")
    def llm_func():
        @weave.op(kind="tool", color="red")
        def tool_func():
            return "tool result"

        tool_result = tool_func()
        
        return f"llm result with {tool_result}"

    llm_func()
    ```
  </Tab>

  <Tab title="TypeScript">
    ```text theme={null}
    This feature is not available in TypeScript yet.
    ```
  </Tab>
</Tabs>

This applies the colors and kinds to your Ops in the Weave UI, like this:

<Frame>
  <img alt="The Weave UI, showing a parent call with a LLM kind and a blue color, and a nested call with a tool kind and a red color." />
</Frame>

The available `kind` values are:

* `agent`
* `llm`
* `tool`
* `search`

The available `color` values are:

* `red`
* `orange`
* `yellow`
* `green`
* `blue`
* `purple`

## Customize logged inputs and outputs

<Tabs>
  <Tab title="Python">
    To change the data that Weave logs without modifying the original function (for example, to hide sensitive data), pass `postprocess_inputs` and `postprocess_output` to the Op decorator.

    `postprocess_inputs` takes a dict where the keys are the argument names and the values are the argument values, and returns a dict with the transformed inputs.

    `postprocess_output` takes any value that the function would normally return and returns the transformed output.

    ```python lines theme={null}
    from dataclasses import dataclass
    from typing import Any
    import weave

    @dataclass
    class CustomObject:
        x: int
        secret_password: str

    def postprocess_inputs(inputs: dict[str, Any]) -> dict[str, Any]:
        return {k:v for k,v in inputs.items() if k != "hide_me"}

    def postprocess_output(output: CustomObject) -> CustomObject:
        return CustomObject(x=output.x, secret_password="REDACTED")

    @weave.op(
        postprocess_inputs=postprocess_inputs,
        postprocess_output=postprocess_output,
    )
    def func(a: int, hide_me: str) -> CustomObject:
        return CustomObject(x=a, secret_password=hide_me)

    weave.init('hide-data-example')
    func(a=1, hide_me="password123")
    ```
  </Tab>

  <Tab title="TypeScript">
    ```text theme={null}
    This feature is not available in TypeScript yet.
    ```
  </Tab>
</Tabs>

## Control sampling rate

<Tabs>
  <Tab title="Python">
    To control how frequently Weave traces an Op's calls, set the `tracing_sample_rate` parameter in the `@weave.op` decorator. Use this for high-frequency Ops where you only need to trace a subset of calls.

    Collect all traces during agent development to help you shape and understand its behavior. In production, configure trace sampling to lower costs while maintaining observability into your agent's behavior.

    Weave applies sampling rates only to outer-most Ops. If a nested Op has a sample rate but a parent Op calls it first, Weave ignores the sampling rate of the nested Op.

    ```python lines theme={null}
    @weave.op(tracing_sample_rate=0.1)  # Only trace ~10% of calls
    def high_frequency_op(x: int) -> int:
        return x + 1

    @weave.op(tracing_sample_rate=1.0)  # Always trace (default)
    def always_traced_op(x: int) -> int:
        return x + 1
    ```

    When Weave doesn't sample an Op's call:

    * The function executes normally.
    * Weave receives no trace data.
    * Weave doesn't trace child Ops for that call.

    The sampling rate must be between 0.0 and 1.0 inclusive.
  </Tab>

  <Tab title="TypeScript">
    ```text theme={null}
    This feature is not available in TypeScript yet.
    ```
  </Tab>
</Tabs>

## Control Call link output

By default, Weave prints a link to each Call as it logs the Call. To suppress the printing of Call links during logging, set the `WEAVE_PRINT_CALL_LINK` environment variable to `false`. Use this to reduce output verbosity and clutter in your logs.

```bash lines theme={null}
export WEAVE_PRINT_CALL_LINK=false
```

## Delete an Op

Deleting an Op version removes it from your project. Use this to clean up obsolete or unwanted versions.

<Tabs>
  <Tab title="Python">
    To delete a version of an Op, call `.delete()` on the Op ref.

    ```python lines theme={null}
    weave.init('intro-example')
    my_op_ref = weave.ref('track_me:v1')
    my_op_ref.delete()
    ```

    Accessing a deleted Op returns an error.
  </Tab>

  <Tab title="TypeScript">
    ```text theme={null}
    This feature is not available in TypeScript yet.
    ```
  </Tab>
</Tabs>


# Send OpenTelemetry Traces to Weave
Source: https://docs.wandb.ai/weave/guides/tracking/otel

Import OpenTelemetry compatible trace data through a dedicated endpoint

Weave supports import of OpenTelemetry compatible trace data through a dedicated endpoint. This endpoint lets you send OTLP (OpenTelemetry Protocol) formatted trace data directly to your Weave project. Use this integration when you want to instrument your application with the OpenTelemetry standard and have those traces appear alongside your other Weave data, without replacing your existing OTel-based observability pipeline.

This page covers the endpoint details, authentication, end-to-end examples in Python and TypeScript, how to forward traces through an OpenTelemetry Collector, how to organize traces into Weave threads, and the attribute mappings Weave applies to incoming spans.

## Endpoint details

**Path**: `/otel/v1/traces`
**Method**: `POST`
**Content-Type**: `application/x-protobuf`
**Base URL**: The base URL for the OTel trace endpoint depends on your W\&B deployment type:

* Multi-tenant Cloud: `https://trace.wandb.ai/otel/v1/traces`.
* Dedicated Cloud and Self-Managed instances: `https://<your-subdomain>.wandb.io/traces/otel/v1/traces`.

Replace `<your-subdomain>` with your organization's unique W\&B domain, for example, `acme.wandb.io`.

## Authentication and routing

Weave uses the `wandb-api-key` header to authenticate requests and resource attributes on your `TracerProvider` to route spans to the correct entity and project. Pass your W\&B API key in the `wandb-api-key` header, then specify the following keys as OpenTelemetry Resource attributes in your `TracerProvider` class:

* `wandb.entity`: Your W\&B team or user name.
* `wandb.project`: The project name to send traces to.

The following example shows how to configure authentication and project routing:

<CodeGroup>
  ```python Python lines {7,8} theme={null}
  import os
  from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
  from opentelemetry.sdk import trace as trace_sdk
  from opentelemetry.sdk.resources import Resource

  WANDB_BASE_URL = "https://trace.wandb.ai"
  ENTITY = "<your-team-name>"
  PROJECT = "<your-project-name>"

  OTEL_EXPORTER_OTLP_ENDPOINT = f"{WANDB_BASE_URL}/otel/v1/traces"

  # Create an API key at https://wandb.ai/settings
  WANDB_API_KEY = os.environ["WANDB_API_KEY"]

  exporter = OTLPSpanExporter(
      endpoint=OTEL_EXPORTER_OTLP_ENDPOINT,
      headers={"wandb-api-key": WANDB_API_KEY},
  )

  tracer_provider = trace_sdk.TracerProvider(resource=Resource({
      "wandb.entity": ENTITY,
      "wandb.project": PROJECT,
  }))
  ```

  ```typescript TypeScript lines theme={null}
  import { NodeTracerProvider } from "@opentelemetry/sdk-trace-node";
  import { OTLPTraceExporter } from "@opentelemetry/exporter-trace-otlp-proto";
  import { Resource } from "@opentelemetry/resources";

  const WANDB_BASE_URL = "https://trace.wandb.ai";
  const ENTITY = "<your-team-name>";
  const PROJECT = "<your-project-name>";

  const OTEL_EXPORTER_OTLP_ENDPOINT = `${WANDB_BASE_URL}/otel/v1/traces`;

  // Create an API key at https://wandb.ai/settings
  const WANDB_API_KEY = process.env.WANDB_API_KEY!;

  const exporter = new OTLPTraceExporter({
    url: OTEL_EXPORTER_OTLP_ENDPOINT,
    headers: { "wandb-api-key": WANDB_API_KEY },
  });

  const provider = new NodeTracerProvider({
    resource: new Resource({
      "wandb.entity": ENTITY,
      "wandb.project": PROJECT,
    }),
  });
  ```
</CodeGroup>

## Examples

The following examples show how to send OpenTelemetry traces to Weave using Python and TypeScript. Each example covers a different approach: using the OpenInference instrumentation library, using OpenLLMetry instrumentation, or using the OpenTelemetry SDK directly without an instrumentation package.

Before running the following code samples, set the following fields:

* `WANDB_API_KEY`: You can get this from [User Settings](https://wandb.ai/settings).
* Entity: You can only log traces to the project under a team/entity that you have access to. To find your entity name, visit your [W\&B dashboard](https://wandb.ai/home) and check the **Teams** field in the left sidebar.
* Project name: Choose a name.
* `OPENAI_API_KEY`: You can obtain this from the [OpenAI dashboard](https://platform.openai.com/api-keys).

### OpenInference instrumentation

[OpenInference](https://github.com/Arize-ai/openinference) is an open source instrumentation library from Arize AI that captures LLM calls as OpenTelemetry spans. This example shows how to use the OpenAI instrumentation. Additional instrumentations are available in the [official repository](https://github.com/Arize-ai/openinference).

First, install the required dependencies:

<Tabs>
  <Tab title="Python">
    ```bash theme={null}
    pip install openai openinference-instrumentation-openai opentelemetry-exporter-otlp-proto-http
    ```
  </Tab>

  <Tab title="TypeScript">
    ```bash theme={null}
    npm install openai @opentelemetry/sdk-trace-node @opentelemetry/sdk-trace-base @opentelemetry/resources @opentelemetry/exporter-trace-otlp-proto @arizeai/openinference-instrumentation-openai @opentelemetry/api
    ```
  </Tab>
</Tabs>

<Warning>
  **Performance recommendation**: Always use `BatchSpanProcessor` instead of `SimpleSpanProcessor` when sending traces to Weave. `SimpleSpanProcessor` exports spans synchronously, which can impact the performance of other workloads. These examples illustrate `BatchSpanProcessor`, which is recommended in production because it batches spans asynchronously and efficiently.
</Warning>

<Tabs>
  <Tab title="Python">
    Paste the following code into a Python file such as `openinference_example.py`:

    ```python lines theme={null}
    import os
    import openai
    from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
    from opentelemetry.sdk import trace as trace_sdk
    from opentelemetry.sdk.resources import Resource
    from opentelemetry.sdk.trace.export import ConsoleSpanExporter, BatchSpanProcessor
    from openinference.instrumentation.openai import OpenAIInstrumentor

    OPENAI_API_KEY = "YOUR_OPENAI_API_KEY"
    WANDB_BASE_URL = "https://trace.wandb.ai"
    ENTITY = "<your-team-name>"
    PROJECT = "<your-project-name>"

    OTEL_EXPORTER_OTLP_ENDPOINT = f"{WANDB_BASE_URL}/otel/v1/traces"

    # Create an API key at https://wandb.ai/settings
    WANDB_API_KEY = os.environ["WANDB_API_KEY"]

    exporter = OTLPSpanExporter(
        endpoint=OTEL_EXPORTER_OTLP_ENDPOINT,
        headers={"wandb-api-key": WANDB_API_KEY},
    )

    tracer_provider = trace_sdk.TracerProvider(resource=Resource({
        "wandb.entity": ENTITY,
        "wandb.project": PROJECT,
    }))
    tracer_provider.add_span_processor(BatchSpanProcessor(exporter))

    # Optionally, print the spans to the console.
    tracer_provider.add_span_processor(BatchSpanProcessor(ConsoleSpanExporter()))

    OpenAIInstrumentor().instrument(tracer_provider=tracer_provider)

    def main():
        client = openai.OpenAI(api_key=OPENAI_API_KEY)
        response = client.chat.completions.create(
            model="gpt-3.5-turbo",
            messages=[{"role": "user", "content": "Describe OTel in a single sentence."}],
            max_tokens=20,
            stream=True,
            stream_options={"include_usage": True},
        )
        for chunk in response:
            if chunk.choices and (content := chunk.choices[0].delta.content):
                print(content, end="")

    if __name__ == "__main__":
        main()
    ```

    Run the code:

    ```bash theme={null}
    python openinference_example.py
    ```
  </Tab>

  <Tab title="TypeScript">
    The TypeScript implementation of this example contains the following key differences from the Python implementation:

    * OpenAI must be imported before registering instrumentation (ESM modules require this).
    * Uses `@opentelemetry/exporter-trace-otlp-proto` (protobuf format) instead of the HTTP exporter, since W\&B's endpoint only accepts protobuf.
    * Requires explicit `provider.shutdown()` with a delay before shutdown to ensure spans are flushed, since `BatchSpanProcessor` flushes asynchronously.

    Paste the following code into a TypeScript file such as `openinference_example.ts`:

    ```typescript lines {11,12} theme={null}
    // IMPORTANT: Import OpenAI FIRST so instrumentation can patch it
    import OpenAI from "openai";
    import { NodeTracerProvider } from "@opentelemetry/sdk-trace-node";
    import { BatchSpanProcessor } from "@opentelemetry/sdk-trace-base";
    import { OTLPTraceExporter } from "@opentelemetry/exporter-trace-otlp-proto";
    import { resourceFromAttributes } from "@opentelemetry/resources";
    import { OpenAIInstrumentation, isPatched } from "@arizeai/openinference-instrumentation-openai";

    const OPENAI_API_KEY = process.env.OPENAI_API_KEY;
    const WANDB_BASE_URL = "https://trace.wandb.ai";
    const ENTITY = "<your-team-name>";
    const PROJECT = "<your-project-name>";

    const OTEL_EXPORTER_OTLP_ENDPOINT = `${WANDB_BASE_URL}/otel/v1/traces`;

    // Create an API key at https://wandb.ai/settings
    const WANDB_API_KEY = process.env.WANDB_API_KEY!;

    const exporter = new OTLPTraceExporter({
      url: OTEL_EXPORTER_OTLP_ENDPOINT,
      headers: { "wandb-api-key": WANDB_API_KEY },
    });

    const provider = new NodeTracerProvider({
      resource: resourceFromAttributes({
        "wandb.entity": ENTITY,
        "wandb.project": PROJECT,
      }),
      spanProcessors: [
        new BatchSpanProcessor(exporter)
      ],
    });

    provider.register();

    // Register the OpenAI instrumentation with the tracer provider
    const openAIInstrumentation = new OpenAIInstrumentation();
    openAIInstrumentation.setTracerProvider(provider);

    // Manually instrument OpenAI since we're using ESM
    openAIInstrumentation.manuallyInstrument(OpenAI);

    async function main() {
      console.log("OpenAI is patched?", isPatched());

      const client = new OpenAI({ apiKey: OPENAI_API_KEY });

      console.log("Making OpenAI API call...");
      const response = await client.chat.completions.create({
        model: "gpt-3.5-turbo",
        messages: [{ role: "user", content: "Describe OTel in a single sentence." }],
        max_tokens: 50,
      });

      console.log("Response:", response.choices[0]?.message?.content);
      console.log("Waiting for spans to flush...");
    }

    (async () => {
      await main();

      // Give spans time to flush
      console.log("Waiting 2 seconds for spans to flush...");
      await new Promise(resolve => setTimeout(resolve, 2000));

      await provider.shutdown(); // flush all pending spans before exit
      console.log("Shutdown complete");
    })();
    ```

    Run the code:

    ```bash theme={null}
    npx ts-node openinference_example.ts
    ```
  </Tab>
</Tabs>

### OpenLLMetry instrumentation

[OpenLLMetry](https://github.com/traceloop/openllmetry) is an open source observability library from Traceloop that provides OpenTelemetry instrumentation for popular LLM providers and frameworks. The following example shows how to use its OpenAI instrumentation. Additional examples are available in the [OpenLLMetry repository](https://github.com/traceloop/openllmetry).

First, install the required dependencies:

<Tabs>
  <Tab title="Python">
    ```bash theme={null}
    pip install openai opentelemetry-instrumentation-openai opentelemetry-exporter-otlp-proto-http
    ```
  </Tab>

  <Tab title="TypeScript">
    ```bash theme={null}
    npm install openai @traceloop/instrumentation-openai @opentelemetry/sdk-trace-node @opentelemetry/resources @opentelemetry/exporter-trace-otlp-http
    ```
  </Tab>
</Tabs>

<Tabs>
  <Tab title="Python">
    Paste the following code into a Python file such as `openllmetry_example.py`. This is the same code as the preceding example, except the `OpenAIInstrumentor` is imported from `opentelemetry.instrumentation.openai` instead of `openinference.instrumentation.openai`:

    ```python lines theme={null}
    import os
    import openai
    from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
    from opentelemetry.sdk import trace as trace_sdk
    from opentelemetry.sdk.resources import Resource
    from opentelemetry.sdk.trace.export import ConsoleSpanExporter, BatchSpanProcessor
    from opentelemetry.instrumentation.openai import OpenAIInstrumentor

    OPENAI_API_KEY = "YOUR_OPENAI_API_KEY"
    WANDB_BASE_URL = "https://trace.wandb.ai"
    ENTITY = "<your-team-name>"
    PROJECT = "<your-project-name>"

    OTEL_EXPORTER_OTLP_ENDPOINT = f"{WANDB_BASE_URL}/otel/v1/traces"

    # Create an API key at https://wandb.ai/settings
    WANDB_API_KEY = os.environ["WANDB_API_KEY"]

    exporter = OTLPSpanExporter(
        endpoint=OTEL_EXPORTER_OTLP_ENDPOINT,
        headers={"wandb-api-key": WANDB_API_KEY},
    )

    tracer_provider = trace_sdk.TracerProvider(resource=Resource({
        "wandb.entity": ENTITY,
        "wandb.project": PROJECT,
    }))
    tracer_provider.add_span_processor(BatchSpanProcessor(exporter))

    # Optionally, print the spans to the console.
    tracer_provider.add_span_processor(BatchSpanProcessor(ConsoleSpanExporter()))

    OpenAIInstrumentor().instrument(tracer_provider=tracer_provider)

    def main():
        client = openai.OpenAI(api_key=OPENAI_API_KEY)
        response = client.chat.completions.create(
            model="gpt-3.5-turbo",
            messages=[{"role": "user", "content": "Describe OTel in a single sentence."}],
            max_tokens=20,
            stream=True,
            stream_options={"include_usage": True},
        )
        for chunk in response:
            if chunk.choices and (content := chunk.choices[0].delta.content):
                print(content, end="")

    if __name__ == "__main__":
        main()
    ```

    Run the code:

    ```bash theme={null}
    python openllmetry_example.py
    ```
  </Tab>

  <Tab title="TypeScript">
    Paste the following code into a TypeScript file such as `openllmetry_example.ts`. This uses the Traceloop OpenAI instrumentation package:

    ```typescript lines theme={null}
    import OpenAI from "openai";
    import { NodeTracerProvider } from "@opentelemetry/sdk-trace-node";
    import { BatchSpanProcessor, ConsoleSpanExporter } from "@opentelemetry/sdk-trace-base";
    import { OTLPTraceExporter } from "@opentelemetry/exporter-trace-otlp-proto";
    import { Resource } from "@opentelemetry/resources";
    import { OpenAIInstrumentation } from "@traceloop/instrumentation-openai";
    import { registerInstrumentations } from "@opentelemetry/instrumentation";

    const OPENAI_API_KEY = process.env.OPENAI_API_KEY;
    const WANDB_BASE_URL = "https://trace.wandb.ai";
    const ENTITY = "<your-team-name>";
    const PROJECT = "<your-project-name>";

    const OTEL_EXPORTER_OTLP_ENDPOINT = `${WANDB_BASE_URL}/otel/v1/traces`;

    // Create an API key at https://wandb.ai/settings
    const WANDB_API_KEY = process.env.WANDB_API_KEY!;

    const exporter = new OTLPTraceExporter({
      url: OTEL_EXPORTER_OTLP_ENDPOINT,
      headers: { "wandb-api-key": WANDB_API_KEY },
    });

    const provider = new NodeTracerProvider({
      resource: new Resource({
        "wandb.entity": ENTITY,
        "wandb.project": PROJECT,
      }),
      spanProcessors: [
        new BatchSpanProcessor(exporter),
        // Optionally, print the spans to the console.
        new BatchSpanProcessor(new ConsoleSpanExporter()),
      ],
    });

    provider.register();

    // Register the OpenAI instrumentation with the tracer provider
    const openAIInstrumentation = new OpenAIInstrumentation();
    registerInstrumentations({
      tracerProvider: provider,
      instrumentations: [openAIInstrumentation],
    });

    // Manually instrument OpenAI since we're using ESM
    openAIInstrumentation.manuallyInstrument(OpenAI);

    async function main() {
      const client = new OpenAI({ apiKey: OPENAI_API_KEY });
      const stream = await client.chat.completions.create({
        model: "gpt-3.5-turbo",
        messages: [{ role: "user", content: "Describe OTel in a single sentence." }],
        max_tokens: 20,
        stream: true,
      });

      for await (const chunk of stream) {
        const content = chunk.choices[0]?.delta?.content;
        if (content) {
          process.stdout.write(content);
        }
      }
      console.log(); // newline after streaming
    }

    (async () => {
      await main();

      // Give spans time to flush
      await new Promise(resolve => setTimeout(resolve, 2000));

      await provider.shutdown(); // flush all pending spans before exit
    })();
    ```

    Run the code:

    ```bash theme={null}
    npx ts-node openllmetry_example.ts
    ```
  </Tab>
</Tabs>

### Without instrumentation

If you prefer to use OTel directly instead of an instrumentation package, you can do so. This approach gives you full control over which attributes are set on each span. Weave parses span attributes according to the OpenTelemetry semantic conventions described at [https://opentelemetry.io/docs/specs/semconv/gen-ai/gen-ai-spans/](https://opentelemetry.io/docs/specs/semconv/gen-ai/gen-ai-spans/).

First, install the required dependencies:

<Tabs>
  <Tab title="Python">
    ```bash theme={null}
    pip install openai opentelemetry-sdk opentelemetry-api opentelemetry-exporter-otlp-proto-http
    ```
  </Tab>

  <Tab title="TypeScript">
    ```bash theme={null}
    npm install openai @opentelemetry/api @opentelemetry/sdk-trace-node @opentelemetry/resources @opentelemetry/exporter-trace-otlp-http
    ```
  </Tab>
</Tabs>

<Tabs>
  <Tab title="Python">
    Paste the following code into a Python file such as `opentelemetry_example.py`:

    ```python lines theme={null}
    import json
    import os
    import openai
    from opentelemetry import trace
    from opentelemetry.sdk import trace as trace_sdk
    from opentelemetry.sdk.resources import Resource
    from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
    from opentelemetry.sdk.trace.export import ConsoleSpanExporter, BatchSpanProcessor

    OPENAI_API_KEY = "YOUR_OPENAI_API_KEY"
    WANDB_BASE_URL = "https://trace.wandb.ai"
    ENTITY = "<your-team-name>"
    PROJECT = "<your-project-name>"

    OTEL_EXPORTER_OTLP_ENDPOINT = f"{WANDB_BASE_URL}/otel/v1/traces"

    # Create an API key at https://wandb.ai/settings
    WANDB_API_KEY = os.environ["WANDB_API_KEY"]

    # Configure the OTLP exporter
    exporter = OTLPSpanExporter(
        endpoint=OTEL_EXPORTER_OTLP_ENDPOINT,
        headers={"wandb-api-key": WANDB_API_KEY},
    )

    tracer_provider = trace_sdk.TracerProvider(resource=Resource({
        "wandb.entity": ENTITY,
        "wandb.project": PROJECT,
    }))
    tracer_provider.add_span_processor(BatchSpanProcessor(exporter))

    # Optionally, print the spans to the console.
    tracer_provider.add_span_processor(BatchSpanProcessor(ConsoleSpanExporter()))

    # Set the tracer provider
    trace.set_tracer_provider(tracer_provider)

    # Create a tracer from the global tracer provider
    tracer = trace.get_tracer(__name__)

    def my_function():
        with tracer.start_as_current_span("outer_span") as outer_span:
            client = openai.OpenAI()
            input_messages = [{"role": "user", "content": "Describe OTel in a single sentence."}]
            outer_span.set_attribute("input.value", json.dumps(input_messages))
            outer_span.set_attribute("gen_ai.system", "openai")
            response = client.chat.completions.create(
                model="gpt-3.5-turbo",
                messages=input_messages,
                max_tokens=20,
                stream=True,
                stream_options={"include_usage": True},
            )
            out = ""
            for chunk in response:
                if chunk.choices and (content := chunk.choices[0].delta.content):
                    out += content
            outer_span.set_attribute("output.value", json.dumps({"content": out}))

    if __name__ == "__main__":
        my_function()
    ```

    Run the code:

    ```bash theme={null}
    python opentelemetry_example.py
    ```
  </Tab>

  <Tab title="TypeScript">
    Paste the following code into a TypeScript file such as `opentelemetry_example.ts`:

    ```typescript lines theme={null}
    import OpenAI from "openai";
    import { trace } from "@opentelemetry/api";
    import { NodeTracerProvider } from "@opentelemetry/sdk-trace-node";
    import { BatchSpanProcessor, ConsoleSpanExporter } from "@opentelemetry/sdk-trace-base";
    import { OTLPTraceExporter } from "@opentelemetry/exporter-trace-otlp-http";
    import { Resource } from "@opentelemetry/resources";

    const OPENAI_API_KEY = "YOUR_OPENAI_API_KEY";
    const WANDB_BASE_URL = "https://trace.wandb.ai";
    const ENTITY = "<your-team-name>";
    const PROJECT = "<your-project-name>";

    const OTEL_EXPORTER_OTLP_ENDPOINT = `${WANDB_BASE_URL}/otel/v1/traces`;

    // Create an API key at https://wandb.ai/settings
    const WANDB_API_KEY = process.env.WANDB_API_KEY!;

    const exporter = new OTLPTraceExporter({
      url: OTEL_EXPORTER_OTLP_ENDPOINT,
      headers: { "wandb-api-key": WANDB_API_KEY },
    });

    const provider = new NodeTracerProvider({
      resource: new Resource({
        "wandb.entity": ENTITY,
        "wandb.project": PROJECT,
      }),
      spanProcessors: [
        new BatchSpanProcessor(exporter),
        // Optionally, print the spans to the console.
        new BatchSpanProcessor(new ConsoleSpanExporter()),
      ],
    });

    provider.register();

    // Creates a tracer from the global tracer provider
    const tracer = trace.getTracer("my-app");

    async function myFunction() {
      const span = tracer.startSpan("outer_span");

      try {
        const client = new OpenAI({ apiKey: OPENAI_API_KEY });
        const inputMessages = [
          { role: "user" as const, content: "Describe OTel in a single sentence." },
        ];

        // This will only appear in the side panel
        span.setAttribute("input.value", JSON.stringify(inputMessages));
        
        // This follows conventions and will appear in the dashboard
        span.setAttribute("gen_ai.system", "openai");

        const stream = await client.chat.completions.create({
          model: "gpt-3.5-turbo",
          messages: inputMessages,
          max_tokens: 20,
          stream: true,
        });

        let output = "";
        for await (const chunk of stream) {
          const content = chunk.choices[0]?.delta?.content;
          if (content) {
            output += content;
          }
          }

        // This will only appear in the side panel
        span.setAttribute("output.value", JSON.stringify({ content: output }));
      } finally {
        span.end();
      }
    }

    myFunction();
    ```

    Run the code:

    ```bash theme={null}
    npx ts-node opentelemetry_example.ts
    ```
  </Tab>
</Tabs>

Weave uses the span attribute prefixes `gen_ai` and `openinference` to determine which convention to apply, if any, when interpreting the trace. If neither key is detected, then all span attributes are visible in the trace view. The full span is available in the side panel when you select a trace.

## Use an OpenTelemetry Collector

The previous examples export traces directly from your application to Weave. In production, you can use an [OpenTelemetry Collector](https://opentelemetry.io/docs/collector/) as an intermediary between your application and Weave. The collector receives traces from your app, then forwards them to one or more backends. This pattern centralizes authentication, batching, and routing logic outside of your application code, and lets you fan traces out to multiple observability backends from a single pipeline.

### Set up a collector

This section walks through running a local OpenTelemetry Collector in Docker and configuring an application to send traces to it. The following example shows how to:

* Set up a Docker configuration file that deploys a local server (collector) that listens for OTLP traces, batches them, and forwards them to Weave.
* Locally run the collector using Docker.
* Send a basic call to OpenAI that forwards traces to the collector running in the Docker container.

To use a collector, first create a `collector-config.yaml` file that configures the collector to receive OTLP traces and export them to Weave:

```yaml lines {23,26} collector-config.yaml title="collector-config.yaml"  theme={null}
receivers:
  otlp:
    protocols:
      http:
        endpoint: 0.0.0.0:4318

exporters:
  otlphttp/weave:
    endpoint: ${env:WANDB_OTLP_ENDPOINT}
    headers:
      wandb-api-key: ${env:WANDB_API_KEY}
    sending_queue:
      batch:

processors:
  resource:
    attributes:
      - key: wandb.entity # Resource attributes field
        value: ${env:DEFAULT_WANDB_ENTITY}  # Value to inject
        action: insert # Inject only if not already present
      - key: wandb.project
        value: ${env:DEFAULT_WANDB_PROJECT}
        action: insert 

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [resource]
      exporters: [otlphttp/weave]
```

This configuration file:

* Listens for OTLP traces on port `4318` (HTTP).
* Exports traces to Weave's OTLP endpoint using the `wandb-api-key` header, reading the endpoint URL from `WANDB_OTLP_ENDPOINT` and the API key from `WANDB_API_KEY`.
* Sets `wandb.entity` and `wandb.project` as resource attributes using the `resource` processor, reading values from `DEFAULT_WANDB_ENTITY` and `DEFAULT_WANDB_PROJECT`. The `insert` action injects these attributes only if your application code does not already set them.
* Enables the exporter's built-in `sending_queue` with batching to reduce network overhead.

After configuring the collector's settings, update the API and entity values in the following Docker command and run it:

```bash lines {3,5} theme={null}
docker run \
  -v ./config.yaml:/etc/otelcol-contrib/config.yaml \
  -e WANDB_API_KEY="<your-wandb-api-key>" \
  -e WANDB_OTLP_ENDPOINT="https://trace.wandb.ai/otel" \
  -e DEFAULT_WANDB_ENTITY="<your-team-name>" \
  -e DEFAULT_WANDB_PROJECT="YOUR_PROJECT" \
  -p 4318:4318 \
  otel/opentelemetry-collector-contrib:latest
```

Once the collector is running, configure your application to export traces to it by setting the `OTEL_EXPORTER_OTLP_ENDPOINT` environment variable. The OTel SDK reads this variable automatically, so you don't need to pass the endpoint to the exporter.

If you set `wandb.entity` or `wandb.project` as resource attributes in your application's `TracerProvider`, they take precedence over the defaults defined in the collector config.

<CodeGroup>
  ```python Python lines theme={null}
  import os
  import openai
  from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
  from opentelemetry.sdk import trace as trace_sdk
  from opentelemetry.sdk.trace.export import BatchSpanProcessor
  from openinference.instrumentation.openai import OpenAIInstrumentor

  os.environ["OTEL_EXPORTER_OTLP_ENDPOINT"] = "http://localhost:4318"

  OPENAI_API_KEY = "YOUR_OPENAI_API_KEY"

  tracer_provider = trace_sdk.TracerProvider()
  tracer_provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter()))

  OpenAIInstrumentor().instrument(tracer_provider=tracer_provider)

  def main():
      client = openai.OpenAI(api_key=OPENAI_API_KEY)
      response = client.chat.completions.create(
          model="gpt-3.5-turbo",
          messages=[{"role": "user", "content": "Describe OTel in a single sentence."}],
          max_tokens=20,
      )
      print(response.choices[0].message.content)

  if __name__ == "__main__":
      main()
  ```

  ```typescript TypeScript lines theme={null}
  import OpenAI from "openai";
  import { NodeTracerProvider } from "@opentelemetry/sdk-trace-node";
  import { BatchSpanProcessor } from "@opentelemetry/sdk-trace-base";
  import { OTLPTraceExporter } from "@opentelemetry/exporter-trace-otlp-proto";
  import { OpenAIInstrumentation, isPatched } from "@arizeai/openinference-instrumentation-openai";

  process.env.OTEL_EXPORTER_OTLP_ENDPOINT = "http://localhost:4318";

  const OPENAI_API_KEY = process.env.OPENAI_API_KEY;

  const provider = new NodeTracerProvider({
    spanProcessors: [new BatchSpanProcessor(new OTLPTraceExporter())],
  });

  provider.register();

  const openAIInstrumentation = new OpenAIInstrumentation();
  openAIInstrumentation.setTracerProvider(provider);
  openAIInstrumentation.manuallyInstrument(OpenAI);

  async function main() {
    console.log("OpenAI is patched?", isPatched());

    const client = new OpenAI({ apiKey: OPENAI_API_KEY });
    const response = await client.chat.completions.create({
      model: "gpt-3.5-turbo",
      messages: [{ role: "user", content: "Describe OTel in a single sentence." }],
      max_tokens: 20,
    });

    console.log("Response:", response.choices[0]?.message?.content);
  }

  (async () => {
    await main();
    await new Promise(resolve => setTimeout(resolve, 2000));
    await provider.shutdown();
  })();
  ```
</CodeGroup>

The `OpenAIInstrumentor` wraps OpenAI calls, creates traces, and exports them to the collector. The collector handles authentication and routing to Weave.

After running the script, you can [view the traces](/weave/guides/tracking/trace-tree) in the Weave UI.

To send traces to additional backends, add more exporters and include them in the `service.pipelines.traces.exporters` list. For example, you can export to both Weave and Jaeger from the same Collector instance.

## Organize OTel traces into threads

[Weave threads](/weave/guides/tracking/threads) let you group related traces so you can analyze multi-turn conversations or user sessions as a single unit. Add specific span attributes to organize your OpenTelemetry traces into threads, then use Weave's Thread UI to analyze related operations such as multi-turn conversations or user sessions.

Add the following attributes to your OTel spans to enable thread grouping:

* `wandb.thread_id`: Groups spans into a specific thread.
* `wandb.is_turn`: Marks a span as a conversation turn (appears as a row in the thread view).

The following examples show how to organize OTel traces into Weave threads. They use `wandb.thread_id` to group related operations and `wandb.is_turn` to mark high-level operations that appear as rows in the thread view.

<Accordion title="Initial set up">
  Use this configuration to run these examples:

  <Tabs>
    <Tab title="Python">
      ```python lines theme={null}
      import json
      import os
      from opentelemetry import trace
      from opentelemetry.sdk import trace as trace_sdk
      from opentelemetry.sdk.resources import Resource
      from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
      from opentelemetry.sdk.trace.export import ConsoleSpanExporter, BatchSpanProcessor

      # Configuration
      ENTITY = "<your-team-name>"
      PROJECT = "<your-project-name>"
      WANDB_API_KEY = os.environ["WANDB_API_KEY"]

      OTEL_EXPORTER_OTLP_ENDPOINT = "https://trace.wandb.ai/otel/v1/traces"

      exporter = OTLPSpanExporter(
          endpoint=OTEL_EXPORTER_OTLP_ENDPOINT,
          headers={"wandb-api-key": WANDB_API_KEY},
      )

      tracer_provider = trace_sdk.TracerProvider(resource=Resource({
          "wandb.entity": ENTITY,
          "wandb.project": PROJECT,
      }))
      tracer_provider.add_span_processor(BatchSpanProcessor(exporter))

      # Optionally, print the spans to the console
      tracer_provider.add_span_processor(BatchSpanProcessor(ConsoleSpanExporter()))

      trace.set_tracer_provider(tracer_provider)

      # Creates a tracer from the global tracer provider
      tracer = trace.get_tracer(__name__)
      ```
    </Tab>

    <Tab title="TypeScript">
      ```typescript lines theme={null}
      import { trace, context } from "@opentelemetry/api";
      import { NodeTracerProvider } from "@opentelemetry/sdk-trace-node";
      import {
        BatchSpanProcessor,
        ConsoleSpanExporter,
      } from "@opentelemetry/sdk-trace-base";
      import { OTLPTraceExporter } from "@opentelemetry/exporter-trace-otlp-proto";
      import { Resource } from "@opentelemetry/resources";

      // Configuration
      const ENTITY = "<your-team-name>";
      const PROJECT = "<your-project-name>";
      const WANDB_API_KEY = process.env.WANDB_API_KEY;

      if (!WANDB_API_KEY) {
        console.error("Error: WANDB_API_KEY environment variable is not set");
        console.error("Run: export WANDB_API_KEY=your_api_key_here");
        process.exit(1);
      }

      // OTel Setup
      const OTEL_EXPORTER_OTLP_ENDPOINT = "https://trace.wandb.ai/otel/v1/traces";

      const exporter = new OTLPTraceExporter({
        url: OTEL_EXPORTER_OTLP_ENDPOINT,
        headers: { "wandb-api-key": WANDB_API_KEY },
      });

      // Initialize tracer provider with span processors
      const provider = new NodeTracerProvider({
        resource: new Resource({
          "wandb.entity": ENTITY,
          "wandb.project": PROJECT,
        }),
        spanProcessors: [
          new BatchSpanProcessor(exporter),
          new BatchSpanProcessor(new ConsoleSpanExporter()),
        ],
      });

      // Register the tracer provider
      provider.register();

      // Create a tracer from the global tracer provider
      const tracer = trace.getTracer("threads-examples");
      ```
    </Tab>
  </Tabs>
</Accordion>

<Accordion title="Trace a basic single-turn thread">
  <Tabs>
    <Tab title="Python">
      ```python lines theme={null}
      def example_1_basic_thread_and_turn():
          """Example 1: Basic thread with a single turn"""
          print("\n=== Example 1: Basic Thread and Turn ===")

          # Create a thread context
          thread_id = "thread_example_1"

          # This span represents a turn (direct child of thread)
          with tracer.start_as_current_span("process_user_message") as turn_span:
              # Set thread attributes
              turn_span.set_attribute("wandb.thread_id", thread_id)
              turn_span.set_attribute("wandb.is_turn", True)

              # Add some example attributes
              turn_span.set_attribute("input.value", "Hello, help me with setup")

              # Simulate some work with nested spans
              with tracer.start_as_current_span("generate_response") as nested_span:
                  # This is a nested call within the turn, so is_turn should be false or unset
                  nested_span.set_attribute("wandb.thread_id", thread_id)
                  # wandb.is_turn is not set or set to False for nested calls

                  response = "I'll help you get started with the setup process."
                  nested_span.set_attribute("output.value", response)

              turn_span.set_attribute("output.value", response)
              print(f"Turn completed in thread: {thread_id}")

      def main():
          example_1_basic_thread_and_turn()

      if __name__ == "__main__":
          main()
      ```
    </Tab>

    <Tab title="TypeScript">
      ```typescript lines theme={null}
      function example_1_basic_thread_and_turn() {
        console.log("\n=== Example 1: Basic Thread and Turn ===");

        // Create a thread context
        const threadId = "thread_example_1";

        // This span represents a turn (direct child of thread)
        tracer.startActiveSpan("process_user_message", (turnSpan) => {
          // Set thread attributes
          turnSpan.setAttribute("wandb.thread_id", threadId);
          turnSpan.setAttribute("wandb.is_turn", true);

          // Add some example attributes
          turnSpan.setAttribute("input.value", "Hello, help me with setup");

          let response: string;
          
          // Simulate some work with nested spans
          tracer.startActiveSpan("generate_response", (nestedSpan) => {
            // This is a nested call within the turn, so is_turn should be false or unset
            nestedSpan.setAttribute("wandb.thread_id", threadId);
            // wandb.is_turn is not set or set to false for nested calls

            response = "I'll help you get started with the setup process.";
            nestedSpan.setAttribute("output.value", response);
            nestedSpan.end();
          });
          
          turnSpan.setAttribute("output.value", response!);
          console.log(`Turn completed in thread: ${threadId}`);
          turnSpan.end();
        });
      }

      function main() {
        example_1_basic_thread_and_turn();
      }

      main();
      ```
    </Tab>
  </Tabs>
</Accordion>

<Accordion title="Trace a multi-turn conversation sharing one thread ID">
  <Tabs>
    <Tab title="Python">
      ```python lines theme={null}
      def example_2_multiple_turns():
          """Example 2: Multiple turns in a single thread"""
          print("\n=== Example 2: Multiple Turns in Thread ===")

          thread_id = "thread_conversation_123"

          # Turn 1
          with tracer.start_as_current_span("process_message_turn1") as turn1_span:
              turn1_span.set_attribute("wandb.thread_id", thread_id)
              turn1_span.set_attribute("wandb.is_turn", True)
              turn1_span.set_attribute("input.value", "What programming languages do you recommend?")

              # Nested operations
              with tracer.start_as_current_span("analyze_query") as analyze_span:
                  analyze_span.set_attribute("wandb.thread_id", thread_id)
                  # No is_turn attribute or set to False for nested spans

              response1 = "I recommend Python for beginners and JavaScript for web development."
              turn1_span.set_attribute("output.value", response1)
              print(f"Turn 1 completed in thread: {thread_id}")

          # Turn 2
          with tracer.start_as_current_span("process_message_turn2") as turn2_span:
              turn2_span.set_attribute("wandb.thread_id", thread_id)
              turn2_span.set_attribute("wandb.is_turn", True)
              turn2_span.set_attribute("input.value", "Can you explain Python vs JavaScript?")

              # Nested operations
              with tracer.start_as_current_span("comparison_analysis") as compare_span:
                  compare_span.set_attribute("wandb.thread_id", thread_id)
                  compare_span.set_attribute("wandb.is_turn", False)  # Explicitly false for nested

              response2 = "Python excels at data science while JavaScript dominates web development."
              turn2_span.set_attribute("output.value", response2)
              print(f"Turn 2 completed in thread: {thread_id}")

      def main():
          example_2_multiple_turns()

      if __name__ == "__main__":
          main()
      ```
    </Tab>

    <Tab title="TypeScript">
      ```typescript lines theme={null}
      function example_2_multiple_turns() {
        console.log("\n=== Example 2: Multiple Turns in Thread ===");

        const threadId = "thread_conversation_123";

        // Turn 1
        tracer.startActiveSpan("process_message_turn1", (turn1Span) => {
          turn1Span.setAttribute("wandb.thread_id", threadId);
          turn1Span.setAttribute("wandb.is_turn", true);
          turn1Span.setAttribute(
            "input.value",
            "What programming languages do you recommend?"
          );

          // Nested operations
          tracer.startActiveSpan("analyze_query", (analyzeSpan) => {
            analyzeSpan.setAttribute("wandb.thread_id", threadId);
            // No is_turn attribute or set to false for nested spans
            analyzeSpan.end();
          });

          const response1 =
            "I recommend Python for beginners and JavaScript for web development.";
          turn1Span.setAttribute("output.value", response1);
          console.log(`Turn 1 completed in thread: ${threadId}`);
          turn1Span.end();
        });

        // Turn 2
        tracer.startActiveSpan("process_message_turn2", (turn2Span) => {
          turn2Span.setAttribute("wandb.thread_id", threadId);
          turn2Span.setAttribute("wandb.is_turn", true);
          turn2Span.setAttribute("input.value", "Can you explain Python vs JavaScript?");

          // Nested operations
          tracer.startActiveSpan("comparison_analysis", (compareSpan) => {
            compareSpan.setAttribute("wandb.thread_id", threadId);
            compareSpan.setAttribute("wandb.is_turn", false); // Explicitly false for nested
            compareSpan.end();
          });

          const response2 =
            "Python excels at data science while JavaScript dominates web development.";
          turn2Span.setAttribute("output.value", response2);
          console.log(`Turn 2 completed in thread: ${threadId}`);
          turn2Span.end();
        });
      }

      function main() {
        example_2_multiple_turns();
      }

      main();
      ```
    </Tab>
  </Tabs>
</Accordion>

<Accordion title="Trace deeply nested operations and mark only the outermost span as a turn">
  <Tabs>
    <Tab title="Python">
      ```python lines theme={null}
      def example_3_complex_nested_structure():
          """Example 3: Complex nested structure with multiple levels"""
          print("\n=== Example 3: Complex Nested Structure ===")

          thread_id = "thread_complex_456"

          # Turn with multiple levels of nesting
          with tracer.start_as_current_span("handle_complex_request") as turn_span:
              turn_span.set_attribute("wandb.thread_id", thread_id)
              turn_span.set_attribute("wandb.is_turn", True)
              turn_span.set_attribute("input.value", "Analyze this code and suggest improvements")

              # Level 1 nested operation
              with tracer.start_as_current_span("code_analysis") as analysis_span:
                  analysis_span.set_attribute("wandb.thread_id", thread_id)
                  # No is_turn for nested operations

                  # Level 2 nested operation
                  with tracer.start_as_current_span("syntax_check") as syntax_span:
                      syntax_span.set_attribute("wandb.thread_id", thread_id)
                      syntax_span.set_attribute("result", "No syntax errors found")

                  # Another Level 2 nested operation
                  with tracer.start_as_current_span("performance_check") as perf_span:
                      perf_span.set_attribute("wandb.thread_id", thread_id)
                      perf_span.set_attribute("result", "Found 2 optimization opportunities")

              # Another Level 1 nested operation
              with tracer.start_as_current_span("generate_suggestions") as suggest_span:
                  suggest_span.set_attribute("wandb.thread_id", thread_id)
                  suggestions = ["Use list comprehension", "Consider caching results"]
                  suggest_span.set_attribute("suggestions", json.dumps(suggestions))

              turn_span.set_attribute("output.value", "Analysis complete with 2 improvement suggestions")
              print(f"Complex turn completed in thread: {thread_id}")

      def main():
          example_3_complex_nested_structure()

      if __name__ == "__main__":
          main()
      ```
    </Tab>

    <Tab title="TypeScript">
      ```typescript lines theme={null}
      function example_3_complex_nested_structure() {
        console.log("\n=== Example 3: Complex Nested Structure ===");

        const threadId = "thread_complex_456";

        // Turn with multiple levels of nesting
        tracer.startActiveSpan("handle_complex_request", (turnSpan) => {
          turnSpan.setAttribute("wandb.thread_id", threadId);
          turnSpan.setAttribute("wandb.is_turn", true);
          turnSpan.setAttribute(
            "input.value",
            "Analyze this code and suggest improvements"
          );

          // Level 1 nested operation
          tracer.startActiveSpan("code_analysis", (analysisSpan) => {
            analysisSpan.setAttribute("wandb.thread_id", threadId);
            // No is_turn for nested operations

            // Level 2 nested operation
            tracer.startActiveSpan("syntax_check", (syntaxSpan) => {
              syntaxSpan.setAttribute("wandb.thread_id", threadId);
              syntaxSpan.setAttribute("result", "No syntax errors found");
              syntaxSpan.end();
            });

            // Another Level 2 nested operation
            tracer.startActiveSpan("performance_check", (perfSpan) => {
              perfSpan.setAttribute("wandb.thread_id", threadId);
              perfSpan.setAttribute("result", "Found 2 optimization opportunities");
              perfSpan.end();
            });

            analysisSpan.end();
          });

          // Another Level 1 nested operation
          tracer.startActiveSpan("generate_suggestions", (suggestSpan) => {
            suggestSpan.setAttribute("wandb.thread_id", threadId);
            const suggestions = ["Use list comprehension", "Consider caching results"];
            suggestSpan.setAttribute("suggestions", JSON.stringify(suggestions));
            suggestSpan.end();
          });

          turnSpan.setAttribute(
            "output.value",
            "Analysis complete with 2 improvement suggestions"
          );
          console.log(`Complex turn completed in thread: ${threadId}`);
          turnSpan.end();
        });
      }

      function main() {
        example_3_complex_nested_structure();
      }

      main();
      ```
    </Tab>
  </Tabs>
</Accordion>

<Accordion title="Trace background operations that belong to a thread but aren't turns">
  <Tabs>
    <Tab title="Python">
      ```python lines theme={null}
      def example_4_non_turn_operations():
          """Example 4: Operations that are part of a thread but not turns"""
          print("\n=== Example 4: Non-Turn Thread Operations ===")

          thread_id = "thread_background_789"

          # Background operation that's part of thread but not a turn
          with tracer.start_as_current_span("background_indexing") as bg_span:
              bg_span.set_attribute("wandb.thread_id", thread_id)
              # wandb.is_turn is unset or false - this is not a turn
              bg_span.set_attribute("wandb.is_turn", False)
              bg_span.set_attribute("operation", "Indexing conversation history")
              print(f"Background operation in thread: {thread_id}")

          # Actual turn in the same thread
          with tracer.start_as_current_span("user_query") as turn_span:
              turn_span.set_attribute("wandb.thread_id", thread_id)
              turn_span.set_attribute("wandb.is_turn", True)
              turn_span.set_attribute("input.value", "Search my previous conversations")
              turn_span.set_attribute("output.value", "Found 5 relevant conversations")
              print(f"Turn completed in thread: {thread_id}")

      def main():
          example_4_non_turn_operations()

      if __name__ == "__main__":
          main()
      ```
    </Tab>

    <Tab title="TypeScript">
      ```typescript lines theme={null}
      function example_4_non_turn_operations() {
        console.log("\n=== Example 4: Non-Turn Thread Operations ===");

        const threadId = "thread_background_789";

        // Background operation that's part of thread but not a turn
        tracer.startActiveSpan("background_indexing", (bgSpan) => {
          bgSpan.setAttribute("wandb.thread_id", threadId);
          // wandb.is_turn is unset or false - this is not a turn
          bgSpan.setAttribute("wandb.is_turn", false);
          bgSpan.setAttribute("operation", "Indexing conversation history");
          console.log(`Background operation in thread: ${threadId}`);
          bgSpan.end();
        });

        // Actual turn in the same thread
        tracer.startActiveSpan("user_query", (turnSpan) => {
          turnSpan.setAttribute("wandb.thread_id", threadId);
          turnSpan.setAttribute("wandb.is_turn", true);
          turnSpan.setAttribute("input.value", "Search my previous conversations");
          turnSpan.setAttribute("output.value", "Found 5 relevant conversations");
          console.log(`Turn completed in thread: ${threadId}`);
          turnSpan.end();
        });
      }

      function main() {
        example_4_non_turn_operations();
      }

      main();
      ```
    </Tab>
  </Tabs>
</Accordion>

After sending these traces, you can view them in the Weave UI under the **Threads** tab, where they're grouped by `thread_id` and each turn appears as a separate row.

## Attribute mappings

Weave maps OpenTelemetry span attributes from various instrumentation frameworks to its internal data model. This mapping means you don't need to rename or transform attributes from your existing instrumentation to get a rich view in Weave. When multiple attribute names map to the same field, Weave applies them in priority order, which lets frameworks coexist in the same traces.

### Supported frameworks

Weave supports attribute conventions from the following observability frameworks and SDKs:

* **OpenTelemetry GenAI**: Standard semantic conventions for generative AI (`gen_ai.*`).
* **OpenInference**: Arize AI's instrumentation library (`input.value`, `output.value`, `llm.*`, `openinference.*`).
* **Vercel AI SDK**: Vercel's AI SDK attributes (`ai.prompt`, `ai.response`, `ai.model.*`, `ai.usage.*`).
* **MLflow**: MLflow tracking attributes (`mlflow.spanInputs`, `mlflow.spanOutputs`).
* **Traceloop**: OpenLLMetry instrumentation (`traceloop.entity.*`, `traceloop.span.kind`).
* **Google Vertex AI**: Vertex AI agent attributes (`gcp.vertex.agent.*`).
* **OpenLit**: OpenLit observability attributes (`gen_ai.content.completion`).
* **Langfuse**: Langfuse tracing attributes (`langfuse.startTime`, `langfuse.endTime`).

### Attribute reference

| Attribute Field Name              | W\&B Mapping                  | Description                            | Type                        | Example                                        |
| :-------------------------------- | :---------------------------- | :------------------------------------- | :-------------------------- | :--------------------------------------------- |
| `ai.prompt`                       | `inputs`                      | User prompt text or messages.          | String, list, dict          | `"Write a short haiku about summer."`          |
| `gen_ai.prompt`                   | `inputs`                      | AI model prompt or message array.      | List, dict, string          | `[{"role":"user","content":"abc"}]`            |
| `input.value`                     | `inputs`                      | Input value for model invocation.      | String, list, dict          | `{"text":"Tell a joke"}`                       |
| `mlflow.spanInputs`               | `inputs`                      | Span input data.                       | String, list, dict          | `["prompt text"]`                              |
| `traceloop.entity.input`          | `inputs`                      | Entity input data.                     | String, list, dict          | `"Translate this to French"`                   |
| `gcp.vertex.agent.tool_call_args` | `inputs`                      | Tool call arguments.                   | Dict                        | `{"args":{"query":"weather in SF"}}`           |
| `gcp.vertex.agent.llm_request`    | `inputs`                      | LLM request payload.                   | Dict                        | `{"contents":[{"role":"user","parts":[...]}]}` |
| `input`                           | `inputs`                      | Generic input value.                   | String, list, dict          | `"Summarize this text"`                        |
| `inputs`                          | `inputs`                      | Generic input array.                   | List, dict, string          | `["Summarize this text"]`                      |
| `ai.response`                     | `outputs`                     | Model response text or data.           | String, list, dict          | `"Here is a haiku..."`                         |
| `gen_ai.completion`               | `outputs`                     | AI completion result.                  | String, list, dict          | `"Completion text"`                            |
| `output.value`                    | `outputs`                     | Output value from model.               | String, list, dict          | `{"text":"Answer text"}`                       |
| `mlflow.spanOutputs`              | `outputs`                     | Span output data.                      | String, list, dict          | `["answer"]`                                   |
| `gen_ai.content.completion`       | `outputs`                     | Content completion result.             | String                      | `"Answer text"`                                |
| `traceloop.entity.output`         | `outputs`                     | Entity output data.                    | String, list, dict          | `"Answer text"`                                |
| `gcp.vertex.agent.tool_response`  | `outputs`                     | Tool execution response.               | Dict, string                | `{"toolResponse":"ok"}`                        |
| `gcp.vertex.agent.llm_response`   | `outputs`                     | LLM response payload.                  | Dict, string                | `{"candidates":[...]}`                         |
| `output`                          | `outputs`                     | Generic output value.                  | String, list, dict          | `"Answer text"`                                |
| `outputs`                         | `outputs`                     | Generic output array.                  | List, dict, string          | `["Answer text"]`                              |
| `gen_ai.usage.input_tokens`       | `usage.input_tokens`          | Number of input tokens consumed.       | Int                         | `42`                                           |
| `gen_ai.usage.prompt_tokens`      | `usage.prompt_tokens`         | Number of prompt tokens consumed.      | Int                         | `30`                                           |
| `llm.token_count.prompt`          | `usage.prompt_tokens`         | Prompt token count.                    | Int                         | `30`                                           |
| `ai.usage.promptTokens`           | `usage.prompt_tokens`         | Prompt tokens consumed.                | Int                         | `30`                                           |
| `gen_ai.usage.completion_tokens`  | `usage.completion_tokens`     | Number of completion tokens generated. | Int                         | `40`                                           |
| `llm.token_count.completion`      | `usage.completion_tokens`     | Completion token count.                | Int                         | `40`                                           |
| `ai.usage.completionTokens`       | `usage.completion_tokens`     | Completion tokens generated.           | Int                         | `40`                                           |
| `llm.usage.total_tokens`          | `usage.total_tokens`          | Total tokens used in request.          | Int                         | `70`                                           |
| `llm.token_count.total`           | `usage.total_tokens`          | Total token count.                     | Int                         | `70`                                           |
| `gen_ai.system`                   | `attributes.system`           | System prompt or instructions.         | String                      | `"You are a helpful assistant."`               |
| `llm.system`                      | `attributes.system`           | System prompt or instructions.         | String                      | `"You are a helpful assistant."`               |
| `weave.span.kind`                 | `attributes.kind`             | Span type or category.                 | String                      | `"llm"`                                        |
| `traceloop.span.kind`             | `attributes.kind`             | Span type or category.                 | String                      | `"llm"`                                        |
| `openinference.span.kind`         | `attributes.kind`             | Span type or category.                 | String                      | `"llm"`                                        |
| `gen_ai.response.model`           | `attributes.model`            | Model identifier.                      | String                      | `"gpt-4o"`                                     |
| `llm.model_name`                  | `attributes.model`            | Model identifier.                      | String                      | `"gpt-4o-mini"`                                |
| `ai.model.id`                     | `attributes.model`            | Model identifier.                      | String                      | `"gpt-4o"`                                     |
| `llm.provider`                    | `attributes.provider`         | Model provider name.                   | String                      | `"openai"`                                     |
| `ai.model.provider`               | `attributes.provider`         | Model provider name.                   | String                      | `"openai"`                                     |
| `gen_ai.request`                  | `attributes.model_parameters` | Model generation parameters.           | Dict                        | `{"temperature":0.7,"max_tokens":256}`         |
| `llm.invocation_parameters`       | `attributes.model_parameters` | Model invocation parameters.           | Dict                        | `{"temperature":0.2}`                          |
| `wandb.display_name`              | `display_name`                | Custom display name for UI.            | String                      | `"User Message"`                               |
| `gcp.vertex.agent.session_id`     | `thread_id`                   | Session or thread identifier.          | String                      | `"thread_123"`                                 |
| `wandb.thread_id`                 | `thread_id`                   | Thread identifier for conversations.   | String                      | `"thread_123"`                                 |
| `wb_run_id`                       | `wb_run_id`                   | Associated W\&B run identifier.        | String                      | `"abc123"`                                     |
| `wandb.wb_run_id`                 | `wb_run_id`                   | Associated W\&B run identifier.        | String                      | `"abc123"`                                     |
| `gcp.vertex.agent.session_id`     | `is_turn`                     | Marks span as conversation turn.       | Boolean                     | `true`                                         |
| `wandb.is_turn`                   | `is_turn`                     | Marks span as conversation turn.       | Boolean                     | `true`                                         |
| `langfuse.startTime`              | `start_time` (override)       | Override span start timestamp.         | Timestamp (ISO8601/unix ns) | `"2024-01-01T12:00:00Z"`                       |
| `langfuse.endTime`                | `end_time` (override)         | Override span end timestamp.           | Timestamp (ISO8601/unix ns) | `"2024-01-01T12:00:01Z"`                       |

## Limitations

The Weave UI does not support rendering OTel trace tool calls in the Chat view. They appear as raw JSON instead.


# Query and export Calls
Source: https://docs.wandb.ai/weave/guides/tracking/querying-calls

Filter, sort, and export Weave call data using the Python SDK, REST API, or the Weave UI for custom analysis.

This page shows you how to query and export Weave Calls so you can analyze trace data outside of Weave. For example, you can build custom dashboards, share results with collaborators, or run offline analysis. You can export Calls from the Weave UI, fetch them programmatically with the Python or TypeScript SDKs, or query them directly using the Service API.

## Export Calls from the Weave UI

In the Weave UI, you can export your data in multiple formats. The UI also shows the Python and cURL code that you can use to export the rows programmatically.

To export Calls:

1. Navigate to [wandb.ai](https://wandb.ai/) and select your project.
2. In the Weave project sidebar, click **Traces**.
3. Select multiple Calls that you want to export by checking the row.
4. In the **Traces** table toolbar, click the export or download button.
5. In the **Export** modal, choose **Selected rows** or **All rows**.
6. Click **Export**.

<Frame>
  <img alt="Traces page showing selected Calls ready for export in the Export modal." />
</Frame>

## Fetch Calls programmatically

To filter, sort, or process Calls outside of the UI, fetch them with one of the Weave SDKs or the Service API. Choose the interface that best matches your workflow.

<Tabs>
  <Tab title="Python">
    To fetch Calls using the Python API, use the [`client.get_calls`](/weave/reference/python-sdk/trace/weave_client#method-get_calls) method:

    ```python lines theme={null}
    import weave

    # Initialize the client
    client = weave.init("your-project-name")

    # Fetch calls
    calls = client.get_calls(filter=...)
    ```
  </Tab>

  <Tab title="TypeScript">
    To fetch Calls using the TypeScript API, use the [`client.getCalls`](/weave/reference/typescript-sdk/classes/weaveclient#getcalls) method:

    ```typescript lines theme={null}
    import * as weave from 'weave'

    // Initialize the client
    const client = await weave.init('intro-example')

    // Fetch calls
    const calls = await client.getCalls(filter=...)
    ```
  </Tab>

  <Tab title="HTTP API">
    The Service API exposes the full query layer. To fetch Calls using the Service API, make a request to the [`/calls/stream_query`](https://docs.wandb.ai/weave/reference/service-api/calls/calls-query-stream) endpoint:

    ```bash theme={null}
    curl -L 'https://trace.wandb.ai/calls/stream_query' \
    -H 'Content-Type: application/json' \
    -H 'Accept: application/json' \
    -d '{
    "project_id": "string",
    "filter": {
        "op_names": [
            "string"
        ],
        "input_refs": [
            "string"
        ],
        "output_refs": [
            "string"
        ],
        "parent_ids": [
            "string"
        ],
        "trace_ids": [
            "string"
        ],
        "call_ids": [
            "string"
        ],
        "trace_roots_only": true,
        "wb_user_ids": [
            "string"
        ],
        "wb_run_ids": [
            "string"
        ]
    },
    "limit": 100,
    "offset": 0,
    "sort_by": [
        {
        "field": "string",
        "direction": "asc"
        }
    ],
    "query": {
        "$expr": {}
    },
    "include_costs": true,
    "include_feedback": true,
    "columns": [
        "string"
    ],
    "expand_columns": [
        "string"
    ]
    }'
    ```
  </Tab>
</Tabs>

For complete details about Call properties and fields, see the [Call schema reference](/weave/guides/tracking/call-schema-reference).

## Export Call metrics

When you need aggregate insights, such as cost or latency trends, rather than the underlying Call records, use the metrics endpoint described in this section.

You can also use the Weave Service API's [POST `/calls/stats` endpoint](https://docs.wandb.ai/weave/reference/service-api/calls/call-stats) to retrieve metrics about your Calls without retrieving the Call data itself. You can retrieve information about your Calls, such as latency and cost, and aggregate them by sum, average, minimum, maximum, and count. For example, you can retrieve:

* Total token usage
* Average latency
* Maximum tokens used
* Total cost
* Minimum input tokens

The endpoint provides several filtering options so you can target Calls made within specified times, and by other properties, such as:

* Op name
* Trace ID
* Thread ID
* User ID

The following example demonstrates how to retrieve Calls generated from an Op named `web_app` over two days. Replace `[YOUR-TEAM-NAME/YOUR-PROJECT-NAME]` with your team and project names:

<CodeGroup>
  ```python Python lines {10-11,16-24} theme={null}
  import requests
  import json
  import os

  url = "https://trace.wandb.ai/calls/stats"

  payload = {
      "project_id": "[YOUR-TEAM-NAME/YOUR-PROJECT-NAME]",
      "start": "2026-03-01T00:00:00Z",
  # Specify the size of the buckets, in seconds.
      "granularity": 86400,
      "filter": {
          "trace_roots_only": True,
          "op_names": ["web_app"]
      },
  # Specify metrics and their aggregate function
      "usage_metrics": [
          {"metric": "total_tokens", "aggregations": ["sum"]},
          {"metric": "total_cost", "aggregations": ["sum"]}
      ],
      "call_metrics": [
          {"metric": "call_count", "aggregations": ["sum"]},
          {"metric": "error_count", "aggregations": ["sum"]},
          {"metric": "latency_ms", "aggregations": ["avg", "min", "max"], "percentiles": [50, 95, 99]}
      ]
  }

  API_KEY = os.getenv("WANDB_API_KEY")

  response = requests.post(url, json=payload, auth=("api", API_KEY))

  print(json.dumps(response.json(), indent=2))
  ```

  ```typescript TypeScript lines {6-7,13-20} theme={null}
  const url = "https://trace.wandb.ai/calls/stats";

  const payload = {
    project_id: "[YOUR-TEAM-NAME/YOUR-PROJECT-NAME]",
    start: "2026-03-01T00:00:00Z",
  // Specify the size of the buckets, in seconds.
    granularity: 86400,
    filter: {
      trace_roots_only: true,
      op_names: ["web_app"],
    },
  // Specify metrics and their aggregate function
    usage_metrics: [
      { metric: "total_tokens", aggregations: ["sum"] },
      { metric: "total_cost", aggregations: ["sum"] },
    ],
    call_metrics: [
      { metric: "call_count", aggregations: ["sum"] },
      { metric: "error_count", aggregations: ["sum"] },
      { metric: "latency_ms", aggregations: ["avg", "min", "max"], percentiles: [50, 95, 99] },
    ],
  };

  const API_KEY = process.env.WANDB_API_KEY!;

  const response = await fetch(url, {
    method: "POST",
    headers: {
      "Content-Type": "application/json",
      Authorization: "Basic " + btoa(`api:${API_KEY}`),
    },
    body: JSON.stringify(payload),
  });

  const data = await response.json();
  console.log(JSON.stringify(data, null, 2));
  ```
</CodeGroup>

The request also specifies how to aggregate the metrics. You can aggregate metrics by `sum`, `count`, `avg`, `min`, `max`, and `count`.

The endpoint returns a JSON object. The following example response shows two days' worth of metrics. Each day (bucket) appears as its own object in the `usage_buckets` and `call_buckets` arrays. Each array breaks down the metrics differently:

* `usage_buckets`: Groups Call metrics for each day by the model used.
* `call_buckets`: Groups Call metrics for each day regardless of the model used.

Set the granularity field (in seconds) in the request to change the bucket size.

```json lines theme={null}
{
  "start": "2026-03-03T00:00:00Z",
  "end": "2026-03-04T21:34:39.746539Z",
  "granularity": 86400,
  "timezone": "UTC",
  "usage_buckets": [
    {
      "timestamp": "2026-03-03T00:00:00",
      "model": "gpt-4o-2024-08-06",
      "sum_total_tokens": 498.0,
      "sum_input_tokens": 219.0,
      "sum_output_tokens": 279.0,
      "count": 5,
      "sum_total_cost": 0.0033374999156876584
    },
    {
      "timestamp": "2026-03-03T00:00:00",
      "model": "gpt-5-2025-08-07",
      "sum_total_tokens": 0.0,
      "sum_input_tokens": 0.0,
      "sum_output_tokens": 0.0,
      "count": 0,
      "sum_total_cost": 0.0
    },
    {
      "timestamp": "2026-03-04T00:00:00",
      "model": "gpt-4o-2024-08-06",
      "sum_total_tokens": 58.0,
      "sum_input_tokens": 27.0,
      "sum_output_tokens": 31.0,
      "count": 1,
      "sum_total_cost": 0.0003774999904635479
    },
    {
      "timestamp": "2026-03-04T00:00:00",
      "model": "gpt-5-2025-08-07",
      "sum_total_tokens": 427.0,
      "sum_input_tokens": 26.0,
      "sum_output_tokens": 401.0,
      "count": 1,
      "sum_total_cost": 0.00404249989787786
    }
  ],
  "call_buckets": [
    {
      "timestamp": "2026-03-03T00:00:00",
      "sum_call_count": 6,
      "sum_error_count": 1,
      "avg_latency_ms": 1505.6666666666667,
      "min_latency_ms": 525,
      "max_latency_ms": 2524,
      "p50_latency_ms": 1534.0,
      "p95_latency_ms": 2328.5,
      "p99_latency_ms": 2484.9000000000005,
      "count": 6
    },
    {
      "timestamp": "2026-03-04T00:00:00",
      "sum_call_count": 2,
      "sum_error_count": 0,
      "avg_latency_ms": 3645.0,
      "min_latency_ms": 1739,
      "max_latency_ms": 5551,
      "p50_latency_ms": 3645.0,
      "p95_latency_ms": 5360.4,
      "p99_latency_ms": 5512.88,
      "count": 2
    }
  ]
}
```

You can query metrics for a maximum time range of 31 days. For more information about available options, see the [Service API reference](https://docs.wandb.ai/weave/reference/service-api/calls/call-stats).


# Redact PII from traces
Source: https://docs.wandb.ai/weave/guides/tracking/redact-pii

Automatically redact Personally Identifiable Information from traces

<Warning>
  This feature is only accessible through the Python SDK.
</Warning>

Some organizations process Personally Identifiable Information (PII) such as names, phone numbers, and email addresses in their Large Language Model (LLM) workflows. Storing this data in W\&B Weave poses compliance and security risks. Stripping this data before you log it helps keep your agent compliant with policies like [GDPR](https://gdpr.eu/) and [HIPAA](https://www.hhs.gov/hipaa/index.html).

The *Sensitive Data Protection* feature lets you automatically redact Personally Identifiable Information (PII) from a [trace](/weave/guides/tracking) before Weave sends it to Weave servers. This feature integrates [Microsoft Presidio](https://microsoft.github.io/presidio/) into the Weave Python SDK, so you can control redaction settings at the SDK level.

The Sensitive Data Protection feature introduces the following functionality to the Python SDK:

* A `redact_pii` setting, which you can enable or disable in the `weave.init()` call to turn on PII redaction.
* Automatic redaction of [common entities](#entities-redacted-by-default) when `redact_pii = True`.
* Customizable redaction fields with the configurable `redact_pii_fields` setting.
* Exclusion of specific entities from redaction with the `redact_pii_exclude_fields` setting.

## Enable PII redaction

This section walks you through installing the required dependencies, turning on PII redaction in `weave.init()`, and optionally tailoring which entities Weave redacts. To get started with the Sensitive Data Protection feature in Weave, complete the following steps:

1. Install the required dependencies:

   ```bash theme={null}
   pip install presidio-analyzer presidio-anonymizer
   ```

2. Modify your `weave.init()` call to enable redaction. When `redact_pii=True`, Weave [redacts common entities by default](#entities-redacted-by-default):

   ```python lines theme={null}
   import weave

   weave.init("my-project", settings={"redact_pii": True})
   ```

3. Optional: Customize redaction fields with the `redact_pii_fields` parameter:

   ```python lines theme={null}
   weave.init("my-project", settings={"redact_pii": True, "redact_pii_fields":["CREDIT_CARD", "US_SSN"]})
   ```

   For a full list of the entities that can be detected and redacted, see [PII entities supported by Presidio](https://microsoft.github.io/presidio/supported_entities/).

4. Optional: Exclude specific entities from redaction with the `redact_pii_exclude_fields` parameter. This is useful when you want to keep the default redaction but preserve certain entity types. The following example shows how to redact all [default entities](#entities-redacted-by-default) except `EMAIL_ADDRESS` and `PERSON`:

   ```python lines theme={null}
   weave.init("my-project", settings={"redact_pii": True, "redact_pii_exclude_fields":["EMAIL_ADDRESS", "PERSON"]})
   ```

After you complete these steps, Weave redacts the configured PII entities from traces before sending them to Weave servers.

## Entities redacted by default

This section lists the PII entity types that Weave redacts automatically when you enable PII redaction without specifying `redact_pii_fields`. Weave automatically redacts the following entities when you enable PII redaction:

* `CREDIT_CARD`
* `CRYPTO`
* `EMAIL_ADDRESS`
* `ES_NIF`
* `FI_PERSONAL_IDENTITY_CODE`
* `IBAN_CODE`
* `IN_AADHAAR`
* `IN_PAN`
* `IP_ADDRESS`
* `LOCATION`
* `PERSON`
* `PHONE_NUMBER`
* `UK_NHS`
* `UK_NINO`
* `US_BANK_NUMBER`
* `US_DRIVER_LICENSE`
* `US_PASSPORT`
* `US_SSN`

## Redact sensitive keys with `REDACT_KEYS`

In addition to PII redaction, the Weave SDK supports redaction of custom keys with `REDACT_KEYS`. This is useful when you want to protect additional sensitive data that might not fall under the PII category but must remain private. Examples include:

* API keys
* Authentication headers
* Tokens
* Internal IDs
* Config values

### Pre-defined `REDACT_KEYS`

Weave automatically redacts the following sensitive keys by default:

```json theme={null}
[
  "api_key",
  "auth_headers",
  "authorization"
]
```

### Add your own keys

If your application uses other field names that contain sensitive values, you can extend the default `REDACT_KEYS` list. Add your own custom keys that you want to redact from traces:

```python lines theme={null}
import weave
from weave.utils import sanitize

client = weave.init("my-project", settings={"redact_pii": True})

# Add custom keys to redact
sanitize.add_redact_key("client_id")
sanitize.add_redact_key("token")

client_id = "123"
token = "789"

@weave.op
def test(client_id, token):
    return client_id + token

test(client_id, token)
```

When you view them in the Weave UI, the values of `client_id` and `token` appear as `"REDACTED"`:

```python lines theme={null}
client_id = "REDACTED"
token = "REDACTED"
```

## Usage information

* This feature is only available in the Python SDK.
* When you enable redaction, processing time increases because of the Presidio dependency.


# Set Call display name
Source: https://docs.wandb.ai/weave/guides/tracking/set-call-display

Set or override the display name for a Call in W&B Weave tracing

Ops produce Calls. An Op is a function or method that you decorate with `@weave.op`. By default, the Op's name is the function name, and the associated Calls have the same display name.

A custom display name makes Calls easier to identify in the Weave UI. Use one to distinguish between Calls of the same Op, surface dynamic context like input values or timestamps, or align Call names with your team's conventions.

You can override the display name for all Calls of a given Op in several ways. Choose the approach that matches the scope you need: a single Call, all Calls of an Op, or the Op itself.

<Tabs>
  <Tab title="Python">
    1. Change the display name at the time of calling the Op. The following example uses the `__weave` dictionary to set the Call display name that takes precedence over the Op display name:

    ```python lines theme={null}
    result = my_function("World", __weave={"display_name": "My Custom Display Name"})
    ```

    2. Change the display name on a per-Call basis. The following example uses the [`Op.call`](/weave/reference/python-sdk/trace/op#function-call) method to return a `call` object. You can then use that object to set the display name with [`call.set_display_name`](/weave/reference/python-sdk/trace/weave_client#method-set_display_name):

    ```python lines theme={null}
    result, call = my_function.call("World")
    call.set_display_name("My Custom Display Name")
    ```

    3. Change the display name for all Calls of a given Op. The following example sets the new display name in the `@weave.op` function decorator itself to affect all Calls for the Op:

    ```python lines theme={null}
    @weave.op(call_display_name="My Custom Display Name")
    def my_function(name: str):
        return f"Hello, {name}!"
    ```

    The `call_display_name` can also be a function that takes in a `call` object and returns a string. Weave passes the `call` object automatically when the function runs. You can use it to dynamically generate names based on the function's name, Call inputs, fields, and so on. This is useful when you want each Call's name to reflect runtime context.

    One common use case is appending a timestamp to the function's name.

    ```python lines theme={null}
    from datetime import datetime

    @weave.op(call_display_name=lambda call: f"{call.func_name}__{datetime.now()}")
    def func():
        return ...
    ```

    You can also log custom metadata using `.attributes`.

    ```python lines theme={null}
    def custom_attribute_name(call):
        model = call.attributes["model"]
        revision = call.attributes["revision"]
        now = call.attributes["date"]

        return f"{model}__{revision}__{now}"

    @weave.op(call_display_name=custom_attribute_name)
    def func():
        return ...

    with weave.attributes(
        {
            "model": "finetuned-llama-3.1-8b",
            "revision": "v0.1.2",
            "date": "2024-08-01",
        }
    ):
        func()  # the display name will be "finetuned-llama-3.1-8b__v0.1.2__2024-08-01"

        with weave.attributes(
            {
                "model": "finetuned-gpt-4o",
                "revision": "v0.1.3",
                "date": "2024-08-02",
            }
        ):
            func()  # the display name will be "finetuned-gpt-4o__v0.1.3__2024-08-02"
    ```

    4. Change the display name of the Op itself. Calls associated with an Op have the same display name, so renaming the Op is the most direct way to rename every Call it produces. If you override the name of the Op itself, the display name of the Call also changes. You can do this in two ways:

       * Set the `name` property of the Op before you log any Calls:
         ```python lines theme={null}
         my_function.name = "My Custom Op Name"
         ```

       * Set the `name` option on the Op decorator:
         ```python lines theme={null}
         @weave.op(name="My Custom Op Name")
         ```
  </Tab>

  <Tab title="TypeScript">
    To override the default name of a Call, use the `callDisplayName` option when calling `weave.op()`.

    ```typescript lines {2} theme={null}
    const extractDinosOp = weave.op(extractDinos, {
    callDisplayName: (input: string) => `Your New Display Name`
    });
    ```
  </Tab>
</Tabs>

If you need to rename a Call after you've already logged it, you can also [update a Call's display name](/weave/guides/tracking/update-call) after execution.


# Trace threads
Source: https://docs.wandb.ai/weave/guides/tracking/threads

Trace and analyze multi-turn conversations in your LLM applications using threads.

With W\&B Weave *Threads*, you can track and analyze multi-turn conversations in your LLM applications. Threads group related Calls under a shared `thread_id`, so you can visualize complete sessions and track conversation-level metrics across turns. You can create threads programmatically and visualize them in the Weave UI.

To get started with Threads, do the following:

1. Familiarize yourself with the basics of Threads.
   * [Use cases](#use-cases)
   * [Definitions](#definitions)
   * [The UI experience](#ui-overview)
   * [API specification](#api-specification)
2. Try the code samples, which demonstrate common usage patterns and real-world use cases.
   * [Basic usage examples](#basic-thread-creation)
   * [Advanced usage examples](#manual-agent-loop-implementation)

## Use cases

Threads are useful when you want to organize and analyze:

* Multi-turn conversations
* Session-based workflows
* Any sequence of related operations

Threads let you group Calls by context, making it easier to understand how your system responds across multiple steps. For example, you can track a single user session, an agent's chain of decisions, or a complex request that spans infrastructure and business logic layers.

By structuring your application with threads and turns, you get cleaner metrics and better visibility in the Weave UI. Instead of seeing every low-level Op, you can focus on the high-level steps that matter.

## Definitions

### Thread

A *Thread* is a logical grouping of related Calls that share a common conversational context. A Thread:

* Has a unique `thread_id`
* Contains one or more *turns*
* Maintains context across Calls
* Represents complete user sessions or interaction flows

### Turn

A *Turn* is a high-level operation within a Thread, displayed in the UI as individual rows in a thread view. Each Turn:

* Represents one logical step in a conversation or workflow
* Is a direct child of a thread context and may include nested lower-level calls (not shown in thread-level stats)

### Call

A *Call* is any `@weave.op`-decorated function execution in your application.

* *Turn Calls* are top-level operations that start new turns.
* *Nested Calls* are lower-level operations within a turn.

### Trace

A *Trace* captures the full Call stack for a single operation. Threads group traces together that are part of the same logical conversation or session. In other words, a thread is made up of multiple turns, each representing one part in the conversation. For more information about Traces, see the [Tracing overview](/weave/guides/tracking/tracing).

## UI overview

In the Weave project sidebar, select **Threads** to access the [Threads list view](#threads-list-view).

<Frame>
  <img alt="The Threads icon in the Weave sidebar" />
</Frame>

### Threads list view

* Lists recent threads in your project.
* Columns include number of turns, start time, and last updated.
* Click a row to open its [detail drawer](#threads-detail-drawer).

<Frame>
  <img alt="The Threads list view" />
</Frame>

### Threads detail drawer

* Click any row to open the detail drawer for that row.
* Shows all turns within a thread.
* Turns are listed in the order they started (based on their start time, not by duration or end time).
* Includes Call-level metadata (latency, inputs, outputs).
* Optionally shows message content or structured data if logged.
* To view the full execution of a turn, you can open it from the thread detail drawer. This lets you drill into all nested operations that occurred during that turn.
* If a turn includes messages extracted from LLM calls, they appear in the chat pane. These messages typically come from calls made by supported integrations (for example, `openai.ChatCompletion.create`) and must meet certain criteria to display. For more information, see [Chat view behavior](#chat-view-behavior).

### Chat view behavior

The chat pane displays structured message data extracted from LLM Calls made during each turn. This view gives you a conversational-style rendering of the interaction.

<Frame>
  <img alt="The Threads chat pane showing structured LLM messages" />
</Frame>

#### What qualifies as a message

Weave extracts messages from Calls within a turn that represent direct interactions with LLM providers (for example, sending a prompt and receiving a response). Only Calls that aren't further nested inside other Calls appear as messages. This avoids duplicating intermediate steps or aggregated internal logic.

Typically, automatically patched third-party SDKs emit messages, for example:

* `openai.ChatCompletion.create`
* `anthropic.Anthropic.completion`

#### What happens when no messages are present

If a turn doesn't emit any messages, the chat pane shows an empty message section for that turn. The chat pane may still include messages from other turns in the same thread.

#### Turn and chat interactions

* Clicking a turn scrolls the chat pane to that turn's message location (pinning behavior).
* Scrolling the chat pane highlights the corresponding turn in the turn list.

#### Navigate to and from the trace view

You can open the full trace for a turn by clicking into it.

A back button appears in the upper left corner to return to the thread detail view. Weave doesn't preserve UI state (such as scroll position) across the transition.

<Frame>
  <img alt="The Threads detail drawer with the back button to return to the thread view" />
</Frame>

## SDK usage

The following sections describe how to create and manage threads programmatically using the Weave SDK. Each example demonstrates a different strategy for organizing turns and threads in your application. For most examples, you should provide your own LLM call or system behavior inside the stub functions.

* To track a session or conversation, use the `weave.thread()` context manager.
* Decorate logical operations with `@weave.op` to track them as turns or nested Calls.
* If you pass a `thread_id`, Weave uses it to group all operations in that block under the same thread. If you omit the `thread_id`, Weave auto-generates a unique one for you.

The return value from `weave.thread()` is a `ThreadContext` object with a `thread_id` property, which you can log, reuse, or pass to other systems.

Nested `weave.thread()` contexts always start a new thread unless you reuse the same `thread_id`. Ending a child context doesn't interrupt or overwrite the parent context. This allows for forked thread structures or layered thread orchestration, depending on your app logic.

### Basic thread creation

The following code sample demonstrates how to use `weave.thread()` to group one or more operations under a shared `thread_id`. This is a straightforward way to start using Threads in your application.

```python lines theme={null}
import weave

@weave.op
def say_hello(name: str) -> str:
    return f"Hello, {name}!"

# Start a new thread context
with weave.thread() as thread_ctx:
    print(f"Thread ID: {thread_ctx.thread_id}")
    say_hello("Bill Nye the Science Guy")
```

### Manual agent loop implementation

This example shows how to manually define a conversational agent using `@weave.op` decorators and `weave.thread()` context management. Each call to `process_user_message` creates a new turn in the thread. You can use this pattern when building your own agent loop and want full control over how context and nesting are handled.

Use the auto-generated thread ID for short-lived interactions, or pass a custom session ID (like `user_session_123`) to persist thread context across sessions.

```python lines theme={null}
import weave

class ConversationAgent:
    @weave.op
    def process_user_message(self, message: str) -> str:
        """
        TURN-LEVEL OPERATION: This represents one conversation turn.
        Only this function will be counted in thread statistics.
        """
        # Store user message
        # Generate AI response through nested calls
        response = self._generate_response(message)
        # Store assistant response
        return response

    @weave.op
    def _generate_response(self, message: str) -> str:
        """NESTED CALL: Implementation details, not counted in thread stats."""
        context = self._retrieve_context(message)     # Another nested call
        intent = self._classify_intent(message)       # Another nested call
        response = self._call_llm(message, context)   # LLM call (nested)
        return self._format_response(response)        # Final nested call

    @weave.op
    def _retrieve_context(self, message: str) -> str:
        # Vector DB lookup, knowledge base query, etc.
        return "retrieved_context"

    @weave.op
    def _classify_intent(self, message: str) -> str:
        # Intent classification logic
        return "general_inquiry"

    @weave.op
    def _call_llm(self, message: str, context: str) -> str:
        # OpenAI/Anthropic/etc API call
        return "llm_response"

    @weave.op
    def _format_response(self, response: str) -> str:
        # Response formatting logic
        return f"Formatted: {response}"

# Usage: Thread context established automatically
agent = ConversationAgent()

# Establish thread context - each process_user_message call becomes a turn
with weave.thread() as thread_ctx:  # Auto-generates thread_id
    print(f"Thread ID: {thread_ctx.thread_id}")

    # Each call to process_user_message creates 1 turn + multiple nested calls
    agent.process_user_message("Hello, help with setup")           # Turn 1
    agent.process_user_message("What languages do you recommend?") # Turn 2
    agent.process_user_message("Explain Python vs JavaScript")     # Turn 3

# Result: Thread with 3 turns, ~15-20 total calls (including nested)

# Alternative: Use explicit thread_id for session tracking
session_id = "user_session_123"
with weave.thread(session_id) as thread_ctx:
    print(f"Session Thread ID: {thread_ctx.thread_id}")  # "user_session_123"

    agent.process_user_message("Continue our previous conversation")  # Turn 1 in this session
    agent.process_user_message("Can you summarize what we discussed?") # Turn 2 in this session
```

### Manual agent with unbalanced call depth

This example demonstrates that turns can be defined at different depths in the call stack depending on how thread context is applied. The sample uses two providers (OpenAI and Anthropic), each with a different call depth before reaching the turn boundary.

All turns share the same `thread_id`, but the turn boundary appears at different levels in the stack depending on the provider logic. This is useful when calls need to be traced differently for different backends, while still grouping them into the same thread.

```python lines theme={null}
import weave
import random
import asyncio

class OpenAIProvider:
    """OpenAI branch: 2 levels deep call chain to turn boundary"""

    @weave.op
    def route_to_openai(self, user_input: str, thread_id: str) -> str:
        """Level 1: Route and prepare OpenAI request"""
        # Input validation, routing logic, basic preprocessing
        print(f"  L1: Routing to OpenAI for: {user_input}")

        # This is the turn boundary - wrap with thread context
        with weave.thread(thread_id):
            # Call Level 2 directly - this creates the call chain depth
            return self.execute_openai_call(user_input)

    @weave.op
    def execute_openai_call(self, user_input: str) -> str:
        """Level 2: TURN BOUNDARY - Execute OpenAI API call"""
        print(f"    L2: Executing OpenAI API call")
        response = f"OpenAI GPT-4 response: {user_input}"
        return response


class AnthropicProvider:
    """Anthropic branch: 3 levels deep call chain to turn boundary"""

    @weave.op
    def route_to_anthropic(self, user_input: str, thread_id: str) -> str:
        """Level 1: Route and prepare Anthropic request"""
        # Input validation, routing logic, provider selection
        print(f"  L1: Routing to Anthropic for: {user_input}")

        # Call Level 2 - this creates call chain depth
        return self.authenticate_anthropic(user_input, thread_id)

    @weave.op
    def authenticate_anthropic(self, user_input: str, thread_id: str) -> str:
        """Level 2: Handle Anthropic authentication and setup"""
        print(f"    L2: Authenticating with Anthropic")

        # Authentication, rate limiting, session management
        auth_token = "anthropic_key_xyz_authenticated"

         # This is the turn boundary - wrap with thread context at Level 3
        with weave.thread(thread_id):
            # Call Level 3 - further nesting the call chain
            return self.execute_anthropic_call(user_input, auth_token)

    @weave.op
    def execute_anthropic_call(self, user_input: str, auth_token: str) -> str:
        """Level 3: TURN BOUNDARY - Execute Anthropic API call"""
        print(f"      L3: Executing Anthropic API call with auth")
        response = f"Anthropic Claude response (auth: {auth_token[:15]}...): {user_input}"
        return response


class MultiProviderAgent:
    """Main agent that routes between providers with different call chain depths"""

    def __init__(self):
        self.openai_provider = OpenAIProvider()
        self.anthropic_provider = AnthropicProvider()

    def handle_conversation_turn(self, user_input: str, thread_id: str) -> str:
        """
        Route to different providers with imbalanced call chain depths.
        Thread context is applied at different nesting levels in each chain.
        """
        # Randomly choose provider for demonstration
        use_openai = random.choice([True, False])

        if use_openai:
            print(f"Choosing OpenAI (2-level call chain)")
            # OpenAI: Level 1 → Level 2 (turn boundary)
            response = self.openai_provider.route_to_openai(user_input, thread_id)
            return f"[OpenAI Branch] {response}"
        else:
            print(f"Choosing Anthropic (3-level call chain)")
            # Anthropic: Level 1 → Level 2 → Level 3 (turn boundary)
            response = self.anthropic_provider.route_to_anthropic(user_input, thread_id)
            return f"[Anthropic Branch] {response}"


async def main():
    agent = MultiProviderAgent()
    conversation_id = "nested_depth_conversation_999"

    # Multi-turn conversation with different call chain depths
    conversation_turns = [
        "What's deep learning?",
        "Explain neural network backpropagation",
        "How do attention mechanisms work?",
        "What's the transformer architecture?",
        "Compare CNNs vs RNNs"
    ]

    print(f"Starting conversation: {conversation_id}")

    for i, user_input in enumerate(conversation_turns, 1):
        print(f"\\n--- Turn {i} ---")
        print(f"User: {user_input}")

        # Same thread_id used across different call chain depths
        response = agent.handle_conversation_turn(user_input, conversation_id)
        print(f"Agent: {response}")

if __name__ == "__main__":
    asyncio.run(main())

# Expected Result: Single thread with 5 turns
# - OpenAI turns: thread context at Level 2 in call chain
#   Call stack: route_to_openai() → execute_openai_call() ← thread context here
# - Anthropic turns: thread context at Level 3 in call chain
#   Call stack: route_to_anthropic() → authenticate_anthropic() → execute_anthropic_call() ← thread context here
# - All turns share thread_id: "nested_depth_conversation_999"
# - Turn boundaries marked at different call stack depths
# - Supporting operations in call chain tracked as nested calls, not turns
```

### Resume a previous session

Sometimes you must resume a previously started session and continue adding Calls to the same thread. In other cases, you may not be able to resume an existing session and must start a new thread instead.

When implementing optional thread resumption, never leave the `thread_id` parameter as `None`, as this disables thread grouping. Instead, always provide a valid thread ID. To create a new thread, generate a unique identifier using a function like `generate_id()`.

When no `thread_id` is specified, Weave's internal implementation automatically generates a random UUID v7. You can replicate this behavior in your own `generate_id()` function or use any unique string value you prefer.

```python lines theme={null}
import weave
import uuidv7
import argparse

def generate_id():
    """Generate a unique thread ID using UUID v7."""
    return str(uuidv7.uuidv7())

@weave.op
def load_history(session_id):
    """Load conversation history for the given session."""
    # Your implementation here
    return []

# Parse command line arguments for session resumption
parser = argparse.ArgumentParser()
parser.add_argument("--session-id", help="Existing session ID to resume")
args = parser.parse_args()

# Determine thread ID: resume existing session or create new one
if args.session_id:
    thread_id = args.session_id
    print(f"Resuming session: {thread_id}")
else:
    thread_id = generate_id()
    print(f"Starting new session: {thread_id}")

# Establish thread context for tracking calls
with weave.thread(thread_id) as thread_ctx:
    # Load or initialize conversation history
    history = load_history(thread_id)
    print(f"Active thread ID: {thread_ctx.thread_id}")

    # Your application logic here.
```

### Nested threads

This example illustrates how to structure complex applications using multiple coordinated threads.

Each layer runs in its own thread context, allowing for clean separation of concerns. The parent application thread coordinates these layers by setting up thread IDs using a shared `ThreadContext`. Use this pattern when you want to analyze or monitor different parts of the system independently while still tying them to a shared session.

```python lines theme={null}
import weave
from contextlib import contextmanager
from typing import Dict

# Global thread context for coordinating nested threads
class ThreadContext:
    def __init__(self):
        self.app_thread_id = None
        self.infra_thread_id = None
        self.logic_thread_id = None

    def setup_for_request(self, request_id: str):
        self.app_thread_id = f"app_{request_id}"
        self.infra_thread_id = f"{self.app_thread_id}_infra"
        self.logic_thread_id = f"{self.app_thread_id}_logic"

# Global instance
thread_ctx = ThreadContext()

class InfrastructureLayer:
    """Handles all infrastructure operations in a dedicated thread"""

    @weave.op
    def authenticate_user(self, user_id: str) -> Dict:
        # Authentication logic...
        return {"user_id": user_id, "authenticated": True}

    @weave.op
    def call_payment_gateway(self, amount: float) -> Dict:
        # Payment processing...
        return {"status": "approved", "amount": amount}

    @weave.op
    def update_inventory(self, product_id: str, quantity: int) -> Dict:
        # Inventory management...
        return {"product_id": product_id, "updated": True}

    def execute_operations(self, user_id: str, order_data: Dict) -> Dict:
        """Execute all infrastructure operations in dedicated thread context"""
        with weave.thread(thread_ctx.infra_thread_id):
            auth_result = self.authenticate_user(user_id)
            payment_result = self.call_payment_gateway(order_data["amount"])
            inventory_result = self.update_inventory(order_data["product_id"], order_data["quantity"])

            return {
                "auth": auth_result,
                "payment": payment_result,
                "inventory": inventory_result
            }


class BusinessLogicLayer:
    """Handles business logic in a dedicated thread"""

    @weave.op
    def validate_order(self, order_data: Dict) -> Dict:
        # Validation logic...
        return {"valid": True}

    @weave.op
    def calculate_pricing(self, order_data: Dict) -> Dict:
        # Pricing calculations...
        return {"total": order_data["amount"], "tax": order_data["amount"] * 0.08}

    @weave.op
    def apply_business_rules(self, order_data: Dict) -> Dict:
        # Business rules...
        return {"rules_applied": ["standard_processing"], "priority": "normal"}

    def execute_logic(self, order_data: Dict) -> Dict:
        """Execute all business logic in dedicated thread context"""
        with weave.thread(thread_ctx.logic_thread_id):
            validation = self.validate_order(order_data)
            pricing = self.calculate_pricing(order_data)
            rules = self.apply_business_rules(order_data)

            return {"validation": validation, "pricing": pricing, "rules": rules}


class OrderProcessingApp:
    """Main application orchestrator"""

    def __init__(self):
        self.infra = InfrastructureLayer()
        self.business = BusinessLogicLayer()

    @weave.op
    def process_order(self, user_id: str, order_data: Dict) -> Dict:
        """Main order processing - becomes a turn in the app thread"""

        # Execute nested operations in their dedicated threads
        infra_results = self.infra.execute_operations(user_id, order_data)
        logic_results = self.business.execute_logic(order_data)

        # Final orchestration
        return {
            "order_id": f"order_12345",
            "status": "completed",
            "infra_results": infra_results,
            "logic_results": logic_results
        }


# Usage with global thread context coordination
def handle_order_request(request_id: str, user_id: str, order_data: Dict):
    # Setup thread context for this request
    thread_ctx.setup_for_request(request_id)

    # Execute in app thread context
    with weave.thread(thread_ctx.app_thread_id):
        app = OrderProcessingApp()
        result = app.process_order(user_id, order_data)
        return result

# Example usage
order_result = handle_order_request(
    request_id="req_789",
    user_id="user_001",
    order_data={"product_id": "laptop", "quantity": 1, "amount": 1299.99}
)

# Expected Thread Structure:
#
# App Thread: app_req_789
# └── Turn: process_order() ← Main orchestration
#
# Infra Thread: app_req_789_infra
# ├── Turn: authenticate_user() ← Infrastructure operation 1
# ├── Turn: call_payment_gateway() ← Infrastructure operation 2
# └── Turn: update_inventory() ← Infrastructure operation 3
#
# Logic Thread: app_req_789_logic
# ├── Turn: validate_order() ← Business logic operation 1
# ├── Turn: calculate_pricing() ← Business logic operation 2
# └── Turn: apply_business_rules() ← Business logic operation 3
#
# Benefits:
# - Clear separation of concerns across threads
# - No parameter drilling of thread IDs
# - Independent monitoring of app/infra/logic layers
# - Global coordination through thread context
```

## API specification

The following sections describe the Threads query endpoint, its request and response schemas, and common query patterns you can use to retrieve thread data programmatically.

### Endpoint

Endpoint: `POST /threads/query`

### Request schema

```python lines theme={null}
class ThreadsQueryReq:
    project_id: str
    limit: Optional[int] = None
    offset: Optional[int] = None
    sort_by: Optional[list[SortBy]] = None  # Supported fields: thread_id, turn_count, start_time, last_updated
    sortable_datetime_after: Optional[datetime] = None   # Filter threads by granule optimization
    sortable_datetime_before: Optional[datetime] = None  # Filter threads by granule optimization
```

### Response schema

```python lines theme={null}
class ThreadSchema:
    thread_id: str           # Unique identifier for the thread
    turn_count: int          # Number of turn calls in this thread
    start_time: datetime     # Earliest start time of turn calls in this thread
    last_updated: datetime   # Latest end time of turn calls in this thread

class ThreadsQueryRes:
    threads: List[ThreadSchema]
```

### Query recent active threads

This example fetches the 50 most recently updated threads. Replace `my-project` with your actual project ID.

```python lines theme={null}
# Get most recently active threads
response = client.threads_query(ThreadsQueryReq(
    project_id="my-project",
    sort_by=[SortBy(field="last_updated", direction="desc")],
    limit=50
))

for thread in response.threads:
    print(f"Thread {thread.thread_id}: {thread.turn_count} turns, last active {thread.last_updated}")
```

### Query threads by activity level

This example fetches the 20 most active threads, ordered by number of turns.

```python lines theme={null}
# Get threads with most activity (most turns)
response = client.threads_query(ThreadsQueryReq(
    project_id="my-project",
    sort_by=[SortBy(field="turn_count", direction="desc")],
    limit=20
))
```

### Query recent threads only

This example returns threads that started in the past 24 hours. You can change the time window by adjusting the value of `days` in `timedelta`.

```python lines theme={null}
from datetime import datetime, timedelta

# Get threads started in the last 24 hours
yesterday = datetime.now() - timedelta(days=1)
response = client.threads_query(ThreadsQueryReq(
    project_id="my-project",
    sortable_datetime_after=yesterday,
    sort_by=[SortBy(field="start_time", direction="desc")]
))
```


# Trace agent integrations
Source: https://docs.wandb.ai/weave/guides/tracking/trace-agent-integrations

Connect W&B Weave to agent SDKs, agent harnesses, and custom agents so sessions appear in the Agents view.

<Note>
  Weave for Agents is in public preview. Features, APIs, and the Agents view UI may change before general availability.
</Note>

W\&B Weave can trace multi-turn agentic applications without hand-instrumenting every turn. For supported agent SDKs and harnesses, call `weave.init()` and Weave autopatches the framework. Weave captures sessions, turns, LLM calls, and tool calls, then renders them in the **Agents** view. This page links to integration guides for three categories: agent harnesses, agent SDKs, and custom agents (including those that emit OpenTelemetry spans).

## Agent harnesses

Agent harnesses are end-user agent runtimes (such as coding agents and developer tools) that produce spans Weave can capture. Install the appropriate plugin or extension, and Weave routes your harness sessions to the **Agents** view:

* **[Claude Code plugin](/weave/guides/integrations/agents/claude-code-harness)**
* **[Codex plugin](/weave/guides/integrations/agents/codex-harness)**
* **[OpenClaw plugin](/weave/guides/integrations/agents/openclaw-harness)**
* **[Pi extension](/weave/guides/integrations/agents/pi-dev-harness)**

## Agent SDKs

Agent SDKs are libraries for building agents and multi-agent workflows in your own application code. Weave autopatches the following SDKs:

* **[OpenAI Agents SDK](/weave/guides/integrations/agents/openai-agents-sdk)**
* **[Google Agent Development Kit (ADK)](/weave/guides/integrations/agents/google-adk)**
* **[Claude Agent SDK](/weave/guides/integrations/agents/claude-agents-sdk)**

## Custom agents and OpenTelemetry

Use the Weave SDK directly to instrument custom agents, including any agent that emits OpenTelemetry spans. Weave accepts any OTel span and has special handling for [GenAI semantic-convention attributes](https://opentelemetry.io/docs/specs/semconv/gen-ai/gen-ai-agent-spans/) so your spans render in the **Agents** view.

To learn how to instrument custom agents with the Weave SDK directly, see [Trace your agents](/weave/guides/tracking/trace-agents). For a walkthrough, see the [Trace an agent quickstart](/weave/agents-quickstart).


# Trace your agents
Source: https://docs.wandb.ai/weave/guides/tracking/trace-agents

Use the Weave SDK to instrument multi-turn agentic applications and view them in the Agents tab.

<Note>
  Weave for Agents is in public preview. Features, APIs, and the Agents view UI may change before general availability.
</Note>

Learn how to instrument a multi-turn agentic application using the W\&B Weave SDK so that you can view, debug, and evaluate your agent's behavior. This guide is intended for developers who build or integrate agents and want structured visibility into sessions, turns, LLM calls, and tool executions.

The Weave SDK for Agents models the full lifecycle of a multi-turn agent conversation: the agent that owns many sessions, the session that groups turns together, each user-agent exchange (turn), the LLM calls within a turn, and the tool executions that an LLM triggers. Traces appear in the **Agents** tab of your Weave project. Each session shows a multi-turn timeline with nested tool calls, token usage, and feedback.

If you're tracing individual functions as Ops with the `@weave.op` decorator, see [Trace LLM applications](/weave/guides/tracking/tracing) instead.

## Before you begin

To get started, install the `weave` package and initialize your project. This step registers your team and project with Weave so that the SDK routes spans to the correct location in the UI.

<Tabs>
  <Tab title="Python">
    ```bash lines theme={null}
    pip install weave
    ```

    Replace `[YOUR-TEAM]` with your W\&B team name and `[YOUR-PROJECT]` with your W\&B project name.

    ```python lines theme={null}
    import weave

    weave.init("[YOUR-TEAM]/[YOUR-PROJECT]")
    ```

    Call `weave.init()` before any `start_session()`, `start_turn()`, `start_llm()`, or `start_tool()` call. All agent tracing functions no-op silently when tracing is disabled or the init call is absent, so you can leave instrumentation in production code and control it through configuration.
  </Tab>

  <Tab title="TypeScript">
    ```bash lines theme={null}
    npm install weave
    ```

    Replace `[YOUR-TEAM]` with your W\&B team name and `[YOUR-PROJECT]` with your W\&B project name.

    ```typescript lines theme={null}
    import * as weave from 'weave';

    await weave.init('[YOUR-TEAM]/[YOUR-PROJECT]');
    ```

    Call `weave.init()` before any `startSession()`, `startTurn()`, `startLLM()`, or `startTool()` call. All agent tracing functions no-op silently when tracing is disabled or the init call is absent, so you can leave instrumentation in production code and control it through configuration.
  </Tab>
</Tabs>

## The agent data model

Weave models agent behavior as a hierarchy of one-to-many relationships. Each agent can have many sessions, each session can have many turns, each turn can have many LLM calls, and each LLM call can trigger many tool calls.

| Concept   | Weave SDK class | OTel span type                                      | Description                                                                  |
| --------- | --------------- | --------------------------------------------------- | ---------------------------------------------------------------------------- |
| Agent     | *(no class)*    | *(no span, grouped by `agent_name`)*                | An agentic application in the Agents tab that contains one or more sessions. |
| Session   | `Session`       | *(no span, turns are grouped by `conversation_id`)* | A conversation or run that contains one or more turns.                       |
| Turn      | `Turn`          | `invoke_agent`                                      | One user message and the agent's complete response.                          |
| LLM call  | `LLM`           | `chat`                                              | One call to a language model API.                                            |
| Tool call | `Tool`          | `execute_tool`                                      | One tool call triggered by an LLM response.                                  |

The following diagram shows how one agent spans many sessions, one session spans many turns, and so on.

```mermaid theme={null}
flowchart TB
    Agent["Agent<br/>agent_name"]

    Agent --> S1 & S2

    S1["Session 1<br/>conversation_id<br/>(no OTel span)"]
    S2["Session 2<br/>conversation_id<br/>(no OTel span)"]

    S1 --> T1 & T2
    S2 --> T3

    T1["Turn 1<br/>invoke_agent<br/>(root span, own trace)"]
    T2["Turn 2<br/>invoke_agent<br/>(root span, own trace)"]
    T3["Turn 1<br/>invoke_agent<br/>(root span, own trace)"]

    T1 --> L1 & L2
    L1["LLM call<br/>chat"]
    L2["LLM call<br/>chat"]

    L1 --> Tool1["Tool call<br/>execute_tool"]

    classDef agent fill:#DE72FF33,stroke:#454B52,stroke-width:2px
    classDef session fill:#FFD95C33,stroke:#454B52,stroke-width:2px
    classDef turn fill:#00CDDB33,stroke:#454B52,stroke-width:2px
    classDef llm fill:#FFCBAD33,stroke:#454B52,stroke-width:2px
    classDef tool fill:#f4f4f5,stroke:#454B52,stroke-width:2px

    class Agent agent
    class S1,S2 session
    class T1,T2,T3 turn
    class L1,L2 llm
    class Tool1 tool
```

A session groups turns by a shared `conversation_id` attribute rather than a parent span, so each turn starts its own OTel trace. This design supports distributed tracing and parallel execution. The client sends spans directly to the OTel collector without any server-side aggregation.

<Tip>
  To integrate Weave with SDKs or harnesses such as the Claude Agent SDK or Codex, see [Trace agent integrations](/weave/guides/tracking/trace-agent-integrations). Weave autopatches into several agent-building SDKs and agent harnesses for quick integration.
</Tip>

## Agent tracing APIs

The following sections describe each top-level tracing function and the arguments it accepts. Use them to instrument the session, turn, LLM call, and tool call layers of the data model described in the previous section.

Weave exposes the following top-level functions. Each function returns an object that works as a context manager (using `with` in Python, or `try/finally` in TypeScript) or that you can close manually by calling `.end()`.

### Start a session

`start_session()` / `startSession()` sets a `conversation_id` attribute on all child spans so that turns are grouped in the Agents tab. If you pass a `session_id`, it must be stable across the lifetime of the conversation. Reuse the same ID to add new turns to an existing session. When you omit `session_id`, the SDK generates a UUID automatically.

The active session is stored in context (a Python `ContextVar` or Node.js `AsyncLocalStorage`), so any code running in the same async context can retrieve it with `weave.get_current_session()` / `weave.getCurrentSession()` without passing the session object explicitly.

<Tabs>
  <Tab title="Python">
    ```python lines theme={null}
    session = weave.start_session(
        agent_name="my-agent",    # Required: identifies the agent in the UI.
        session_id="",            # Optional: stable ID to group turns; auto-generated when empty.
        model="",                 # Optional: default model for turns in this session.
        session_name="",          # Optional: human-readable label shown in the UI.
        include_content=True,     # Optional: set False to omit message bodies from spans.
        continue_parent_trace=False,  # Optional: attach to an existing OTel trace instead of starting a new one.
    )
    ```
  </Tab>

  <Tab title="TypeScript">
    ```typescript lines theme={null}
    const session = weave.startSession({
      agentName: 'my-agent',  // Optional: identifies the agent in the UI.
      sessionId: '',          // Optional: stable ID to group turns, auto-generated when empty.
      model: '',              // Optional: default model for turns in this session.
    });
    ```
  </Tab>
</Tabs>

### Start a turn

`start_turn()` / `startTurn()` creates a new `invoke_agent` span that becomes the root of a new OTel trace. Weave uses this span to represent one complete user-agent exchange in the timeline view.

When called as a top-level function, it resolves the active session from context and inherits its conversation ID. If no session is active, Weave creates the turn without a `conversation_id` and doesn't group it with other turns.

<Tabs>
  <Tab title="Python">
    ```python lines theme={null}
    turn = weave.start_turn(
        user_message="What is the weather in Tokyo?",  # The user's input text.
        agent_name="my-agent",   # Optional: overrides the session-level agent name.
        model="gpt-4o",          # Optional: model used for this turn.
    )
    ```
  </Tab>

  <Tab title="TypeScript">
    ```typescript lines theme={null}
    const turn = weave.startTurn({
      agentName: 'my-agent',  // Optional: overrides the session-level agent name.
      model: 'gpt-4o',        // Optional: model used for this turn.
    });
    ```
  </Tab>
</Tabs>

### Start an LLM call

`start_llm()` / `startLLM()` creates a `chat` span nested under the current turn. Weave uses this span to display token usage, model name, input and output messages, and reasoning in the Agents view.

<Tabs>
  <Tab title="Python">
    ```python lines theme={null}
    llm = weave.start_llm(
        model="gpt-4o",             # The model identifier.
        provider_name="openai",     # Required: provider name, for example "openai", "anthropic".
        system_instructions=["Be concise."],  # Optional: system prompt strings.
    )
    ```
  </Tab>

  <Tab title="TypeScript">
    ```typescript lines theme={null}
    const llm = weave.startLLM({
      model: 'gpt-4o',          // The model identifier.
      providerName: 'openai',   // Optional: provider name, for example "openai", "anthropic".
    });
    ```
  </Tab>
</Tabs>

After the LLM call completes, assign the response data to the `llm` object before it closes:

<Tabs>
  <Tab title="Python">
    ```python lines theme={null}
    with weave.start_llm(model="gpt-4o", provider_name="openai") as llm:
        response = openai_client.chat.completions.create(...)
        llm.input_messages = [Message(role="user", content="...")]
        llm.output_messages = [Message(role="assistant", content=response.choices[0].message.content)]
        llm.usage = Usage(
            input_tokens=response.usage.prompt_tokens,
            output_tokens=response.usage.completion_tokens,
        )
    ```
  </Tab>

  <Tab title="TypeScript">
    ```typescript lines theme={null}
    const llm = weave.startLLM({ model: 'gpt-4o', providerName: 'openai' });
    try {
      const response = await openaiClient.chat.completions.create({ ... });
      llm.inputMessages = [{ role: 'user', content: '...' }];
      llm.outputMessages = [{ role: 'assistant', content: response.choices[0].message.content ?? '' }];
      llm.usage = {
        inputTokens: response.usage?.prompt_tokens,
        outputTokens: response.usage?.completion_tokens,
      };
    } finally {
      llm.end();
    }
    ```
  </Tab>
</Tabs>

Pass `provider_name` / `providerName` explicitly. Weave doesn't infer it from the model string.

### Start a tool call

`start_tool()` / `startTool()` creates an `execute_tool` span. The span becomes a child of whatever OTel span is active in context (typically the `chat` span of the LLM call that produced the tool call).

<Tabs>
  <Tab title="Python">
    ```python lines theme={null}
    tool = weave.start_tool(
        name="get_weather",                  # Tool name as declared to the LLM.
        arguments='{"city": "Tokyo"}',       # JSON string of the tool arguments.
        tool_call_id="call_abc123",          # Optional: tool call ID from the LLM response.
    )
    ```
  </Tab>

  <Tab title="TypeScript">
    ```typescript lines theme={null}
    const tool = weave.startTool({
      name: 'get_weather',            // Tool name as declared to the LLM.
      args: '{"city": "Tokyo"}',      // Optional: JSON string of the tool arguments.
      toolCallId: 'call_abc123',      // Optional: tool call ID from the LLM response.
    });
    ```
  </Tab>
</Tabs>

Assign the tool result before closing:

<Tabs>
  <Tab title="Python">
    ```python lines theme={null}
    with weave.start_tool(name="get_weather", arguments='{"city": "Tokyo"}') as tool:
        result = get_weather_api("Tokyo")
        tool.result = result  # Accepts dict, list, or string. JSON-encoded automatically.
    ```
  </Tab>

  <Tab title="TypeScript">
    ```typescript lines theme={null}
    const tool = weave.startTool({ name: 'get_weather', args: '{"city": "Tokyo"}' });
    try {
      tool.result = await getWeatherApi('Tokyo');
    } finally {
      tool.end();
    }
    ```
  </Tab>
</Tabs>

## Usage patterns for agent tracing

The following sections describe how to combine these functions depending on how your agent code is structured.

The following examples use two types from the Weave SDK:

* `Message` represents a single entry in a conversation: a user input, an assistant response, a system prompt, or a tool result. Assign to `llm.input_messages` / `llm.inputMessages` to record what the model received and produced.
* `Usage` captures token counts from the LLM response and is assigned to `llm.usage`.

Weave uses both to populate the Agents view with the inputs, outputs, and token usage of each LLM call. For all supported data types, see the API reference.

### Context manager or try-finally pattern

For most agents, use a context manager pattern in Python or a try-finally pattern in TypeScript. The span closes and sends at the end of the block, even if an exception occurs.

Weave stores the active session, turn, and LLM call in context, so any function called within a block can call `start_llm()` / `startLLM()` or `start_tool()` / `startTool()` without holding an explicit reference to the parent. This works across module boundaries as long as the code runs in the same async context. To retrieve the active objects from anywhere in the call stack, use `weave.get_current_session()` / `weave.getCurrentSession()`, `weave.get_current_turn()` / `weave.getCurrentTurn()`, and `weave.get_current_llm()` / `weave.getCurrentLLM()`.

<Tabs>
  <Tab title="Python">
    ```python lines highlight="13,14,17,25,29" theme={null}
    import weave
    from weave.session.session import Message, Usage

    # Placeholder functions: replace with your own implementations.
    def call_openai(*args, **kwargs):
        pass  # Replace with your LLM client call.

    def get_weather_api(city: str) -> str:
        return "24°C, sunny"  # Replace with your weather API call.

    weave.init("[YOUR-TEAM]/[YOUR-PROJECT]")

    with weave.start_session(agent_name="weather-bot") as session:
        with session.start_turn(user_message="What is the weather in Tokyo?") as turn:

            # First LLM call: returns a tool call.
            with weave.start_llm(model="gpt-4o", provider_name="openai") as llm:
                response = call_openai(...)
                llm.input_messages = [Message(role="user", content="What is the weather?")]
                llm.think("User wants weather data, I should call get_weather.")
                llm.output("Let me check the weather for you.")
                llm.usage = Usage(input_tokens=100, output_tokens=20)

                # Tool call: child of the LLM call that requested it.
                with weave.start_tool(name="get_weather", arguments='{"city":"Tokyo"}') as tool:
                    tool.result = get_weather_api("Tokyo")  # Returns "24°C, sunny".

            # Second LLM call: synthesizes the final answer.
            with weave.start_llm(model="gpt-4o", provider_name="openai") as llm:
                llm.input_messages = [Message(role="user", content="What is the weather?")]
                llm.output("It is 24°C and sunny in Tokyo today.")
                llm.usage = Usage(input_tokens=150, output_tokens=30)
    ```
  </Tab>

  <Tab title="TypeScript">
    ```typescript lines highlight="11,13,16,24,35" theme={null}
    import * as weave from 'weave';
    import type { Message, Usage } from 'weave';

    // Placeholder function: replace with your own implementation.
    async function getWeatherApi(city: string): Promise<string> {
      return '24°C, sunny';  // Replace with your weather API call.
    }

    await weave.init('[YOUR-TEAM]/[YOUR-PROJECT]');

    const session = weave.startSession({ agentName: 'weather-bot' });
    try {
      const turn = session.startTurn({ agentName: 'weather-bot' });
      try {
        // First LLM call: returns a tool call.
        const llm = weave.startLLM({ model: 'gpt-4o', providerName: 'openai' });
        try {
          llm.inputMessages = [{ role: 'user', content: 'What is the weather?' }];
          llm.think('User wants weather data, I should call get_weather.');
          llm.output('Let me check the weather for you.');
          llm.usage = { inputTokens: 100, outputTokens: 20 };

          // Tool call: child of the LLM call that requested it.
          const tool = weave.startTool({ name: 'get_weather', args: '{"city":"Tokyo"}' });
          try {
            tool.result = await getWeatherApi('Tokyo');  // Returns "24°C, sunny".
          } finally {
            tool.end();
          }
        } finally {
          llm.end();
        }

        // Second LLM call: synthesizes the final answer.
        const llm2 = weave.startLLM({ model: 'gpt-4o', providerName: 'openai' });
        try {
          llm2.inputMessages = [{ role: 'user', content: 'What is the weather?' }];
          llm2.output('It is 24°C and sunny in Tokyo today.');
          llm2.usage = { inputTokens: 150, outputTokens: 30 };
        } finally {
          llm2.end();
        }
      } finally {
        turn.end();
      }
    } finally {
      session.end();
    }
    ```
  </Tab>
</Tabs>

### Manual start and end pattern

Use `.end()` explicitly when you can't use `with` blocks or `try/finally`. For example, when you open and close spans in different function calls, or when you manage async lifecycle outside a coroutine. You're responsible for calling `.end()` on every object you create, so that spans close and flush to the collector.

<Tabs>
  <Tab title="Python">
    ```python lines highlight="1,2,4,9,15" theme={null}
    session = weave.start_session(agent_name="weather-bot")
    turn = session.start_turn(user_message="What is the weather?")

    llm = weave.start_llm(model="gpt-4o", provider_name="openai")
    llm.input_messages = [Message(role="user", content="What is the weather?")]
    llm.output("Let me check.")
    llm.usage = Usage(input_tokens=100, output_tokens=20)

    tool = weave.start_tool(name="get_weather", arguments='{"city": "Tokyo"}')
    tool.result = "24°C, sunny"
    tool.end()   # end() is idempotent — safe to call more than once.

    llm.end()

    llm2 = weave.start_llm(model="gpt-4o", provider_name="openai")
    llm2.output("It is 24°C and sunny in Tokyo.")
    llm2.usage = Usage(input_tokens=150, output_tokens=30)
    llm2.end()

    turn.end()
    session.end()
    ```
  </Tab>

  <Tab title="TypeScript">
    ```typescript lines highlight="1,2,4,9,15" theme={null}
    const session = weave.startSession({ agentName: 'weather-bot' });
    const turn = session.startTurn({ agentName: 'weather-bot' });

    const llm = weave.startLLM({ model: 'gpt-4o', providerName: 'openai' });
    llm.inputMessages = [{ role: 'user', content: 'What is the weather?' }];
    llm.output('Let me check.');
    llm.usage = { inputTokens: 100, outputTokens: 20 };

    const tool = weave.startTool({ name: 'get_weather', args: '{"city": "Tokyo"}' });
    tool.result = '24°C, sunny';
    tool.end();  // end() is idempotent: safe to call more than once.

    llm.end();

    const llm2 = weave.startLLM({ model: 'gpt-4o', providerName: 'openai' });
    llm2.output('It is 24°C and sunny in Tokyo.');
    llm2.usage = { inputTokens: 150, outputTokens: 30 };
    llm2.end();

    turn.end();
    session.end();
    ```
  </Tab>
</Tabs>

## Semantic conventions

The Weave SDK emits OTel spans that conform to the [GenAI semantic conventions](https://opentelemetry.io/docs/specs/semconv/gen-ai/gen-ai-spans/) and [GenAI agent span conventions](https://opentelemetry.io/docs/specs/semconv/gen-ai/gen-ai-agent-spans/). Weave accepts any OTel span, stores all attributes, and makes them queryable. You can add arbitrary attributes to spans with the standard OTel span API alongside Weave's tracing objects.

## How spans appear in the Weave UI

After you instrument your agent with the preceding patterns and run it, your traces appear in the **Agents** tab of your Weave project at `https://wandb.ai/[YOUR-TEAM]/[YOUR-PROJECT]/weave/agents`.

* The **Sessions list** shows all sessions with a minimap of turn activity.
* The **multi-turn session view** opens when you click a session and shows each turn, its LLM calls, tool executions, token counts, and any attached feedback.
* Each `chat` span shows the input messages, output messages, model name, and usage.
* Each `execute_tool` span shows the tool name, arguments, and result.

For details on viewing Agents data in Weave, see [View agent activity](/weave/guides/tracking/view-agent-activity).


# Batch logging for your agent
Source: https://docs.wandb.ai/weave/guides/tracking/trace-agents-batch

Manually log agent traces for frameworks that have already completed the LLM call and need to record it.

<Note>
  Weave for Agents is in public preview. Features, APIs, and the Agents view UI may change before general availability.
</Note>

For frameworks that have already completed the LLM call and only need to record it, use `weave.log_turn` and `weave.log_session`. All spans are created and ended immediately without keeping any context managers open.

Data logged this way can be historical. No live session is needed. Set `session_id` to any stable string that uniquely identifies the conversation. Turns that share the same `session_id` are grouped as a single session in the **Agents** view.

If you're building your own agent loop, use the real-time instrumentation APIs described in [Trace your agents](/weave/guides/tracking/trace-agents) instead.

## Log a turn

To record a single completed turn after it happens, use `weave.log_turn`. It accepts a fully-formed turn, including all LLM and tool spans.

<Tabs>
  <Tab title="Python">
    ```python lines highlight="1,3,28" theme={null}
    weave.init("[YOUR-TEAM]/[YOUR-PROJECT]")

    from weave.session.session import LLM, Message, Tool, Usage

    llm_span = LLM(
        model="gpt-4o",
        provider_name="openai",
        input_messages=[Message(role="user", content="What is the weather?")],
        output_messages=[Message(role="assistant", content="Let me check.")],
        usage=Usage(input_tokens=100, output_tokens=20),
    )

    tool_span = Tool(
        name="get_weather",
        arguments='{"city": "Tokyo"}',
        result='"24°C, sunny"',
    )

    llm_span2 = LLM(
        model="gpt-4o",
        provider_name="openai",
        input_messages=[Message(role="user", content="What is the weather?")],
        output_messages=[Message(role="assistant", content="It is 24°C and sunny.")],
        usage=Usage(input_tokens=150, output_tokens=30),
    )

    # Log a turn with all its spans.
    weave.log_turn(
        session_id="my-session-abc",
        agent_name="weather-bot",
        messages=[
            Message(role="user", content="What is the weather in Tokyo?"),
            Message(role="assistant", content="It is 24°C and sunny in Tokyo."),
        ],
        spans=[llm_span, tool_span, llm_span2],
    )
    ```
  </Tab>

  <Tab title="TypeScript">
    ```plaintext theme={null}
    This feature is not available in the TypeScript SDK yet.
    ```
  </Tab>
</Tabs>

`log_turn` returns a `LogResult` containing the trace IDs of the emitted spans.

An optional `model` parameter on `log_turn` sets the model on the turn's own span, not on the child LLM spans. Each `LLM` span carries its own `model` independently. If a turn uses multiple models, set `model` on `log_turn` to whichever you consider the primary model for that turn.

## Log a session

To bulk-import a complete, multi-turn session at once, use `weave.log_session`. The `turns` parameter accepts a list of `Turn` objects, each constructed the same way as the previous `log_turn` example.

<Tabs>
  <Tab title="Python">
    ```python lines theme={null}
    weave.log_session(
        session_id="my-session-abc",
        agent_name="weather-bot",
        turns=[turn_1, turn_2],
    )
    ```
  </Tab>

  <Tab title="TypeScript">
    ```plaintext theme={null}
    This feature is not available in the TypeScript SDK yet.
    ```
  </Tab>
</Tabs>


# Disable tracing
Source: https://docs.wandb.ai/weave/guides/tracking/trace-disable

Learn options to disable or conditionally turn off W&B Weave tracing

This page describes how to disable W\&B Weave tracing in your application. Use these options when you want to prevent traces from being logged, for example, during local development, in test environments, or for specific code paths where tracing isn't needed. Choose the approach that best matches the scope of control you need: program-wide, per-client, or per-code-block.

## Environment variable

Use an environment variable when you want to disable tracing for the entire program without modifying application code.

To unconditionally disable tracing for the entire program, set the environment variable `WEAVE_DISABLED=true`.

`WEAVE_DISABLED` is read only once, at function-definition time. You can't use this variable to toggle tracing at runtime.

## Client initialization

To conditionally disable tracing for a specific Weave client, initialize the client with the `disabled` flag in init settings.

<Tabs>
  <Tab title="Python">
    ```python lines theme={null}
    import weave

    # Initialize the client
    client = weave.init(..., settings={"disabled": True})
    ```
  </Tab>

  <Tab title="TypeScript">
    ```plaintext theme={null}
    This feature is not available for the TypeScript SDK yet.
    ```
  </Tab>
</Tabs>

## Context manager

To conditionally disable tracing only within a specific block of code, use a tracing context manager. Use `with tracing_disabled()` to suppress tracing **only for the function calls executed inside the `with` block**. Use it in application code to scope which calls shouldn't be logged.

<Tabs>
  <Tab title="Python">
    ```python lines theme={null}
    import weave
    from weave.trace.context.call_context import tracing_disabled

    client = weave.init('your-team/your-project-name')

    @weave.op
    def my_op():
        ...

    with tracing_disabled():
        my_op()
    ```
  </Tab>

  <Tab title="TypeScript">
    ```plaintext theme={null}
    This feature is not available for the TypeScript SDK yet.
    ```
  </Tab>
</Tabs>

Although tracing behavior is fixed when functions are defined, you can use this for runtime control when you combine it with application logic. For example, wrap the context manager in a conditional to dynamically enable or disable tracing based on a runtime value:

<Tabs>
  <Tab title="Python">
    ```python lines theme={null}
    if should_trace:
        my_op()
    else:
        with tracing_disabled():
            my_op()
    ```
  </Tab>

  <Tab title="TypeScript">
    ```plaintext theme={null}
    This feature is not available for the TypeScript SDK yet.
    ```
  </Tab>
</Tabs>


# Trace generator functions
Source: https://docs.wandb.ai/weave/guides/tracking/trace-generator-func

Track sync and async generator functions with W&B Weave tracing

W\&B Weave supports tracing both sync and async generator functions, including deeply nested patterns. This page shows you how to decorate generator functions with `@weave.op` so that Weave captures their inputs, yielded outputs, and the full hierarchy of nested calls. Use this when you have streaming or lazily evaluated code paths whose outputs you want to inspect in the Weave UI alongside your other traces.

<Warning>
  Because generators yield values lazily, Weave logs outputs only when you fully consume the generator (for example, when you convert it to a list).
  To ensure Weave captures outputs in the trace, fully consume the generator (for example, with `list()`).
</Warning>

<Tabs>
  <Tab title="Python">
    ```python theme={null}
    from typing import Generator
    import weave

    weave.init("my-project")

    # This function uses a sync generator.
    # Weave will trace the Call and its input (`x`),
    # but output values are only captured once the generator is consumed (for example, with `list()`).
    @weave.op
    def basic_gen(x: int) -> Generator[int, None, None]:
        yield from range(x)

    # A normal sync function used within the generator pipeline.
    # Its calls are also traced independently by Weave.
    @weave.op
    def inner(x: int) -> int:
        return x + 1

    # A sync generator that calls another traced function (`inner`).
    # Each yielded value comes from a separate traced Call to `inner`.
    @weave.op
    def nested_generator(x: int) -> Generator[int, None, None]:
        for i in range(x):
            yield inner(i)

    # A generator that composes the above generator.
    # Tracing here produces a hierarchical Call tree:
    # - `deeply_nested_generator` (parent)
    #   - `nested_generator` (child)
    #     - `inner` (grandchild)
    @weave.op
    def deeply_nested_generator(x: int) -> Generator[int, None, None]:
        for i in range(x):
            for j in nested_generator(i):
                yield j

    # The generator must be *consumed* for Weave to capture outputs.
    # This is true for both sync and async generators.
    res = deeply_nested_generator(4)
    list(res)  # Triggers tracing of all nested calls and yields
    ```
  </Tab>

  <Tab title="TypeScript">
    ```plaintext theme={null}
    This feature is not available in the TypeScript SDK yet.
    ```
  </Tab>
</Tabs>

The following screenshot shows the **Traces** page with a selected trace of the preceding code. The center panel shows the trace tree for the selected trace. The trace tree shows the `deeply_nested_generator`, `nested_generator`, and `inner` Ops in the trace tree hierarchy.

<img alt="Weave Traces page showing a selected trace tree illustrating deeply nested Ops" />

## Consume generators

The following section explains why you must consume generators for Weave to record their outputs, and which consumption patterns work.

Weave captures generator outputs only after you fully consume the generator. Consume the generator by iterating over it (for example, with `list()`, a `for` loop, or `next()` until exhaustion). The same applies to async generators when you use `async for` or equivalent consumption.

For more information about decorating functions and methods with `@weave.op`, see [Create calls](/weave/guides/tracking/create-call).

## Accumulate yielded values into a single trace

If you want Weave to record a combined result (such as a joined string or a list) instead of the raw sequence of yielded values, use an accumulator.

You can use the `accumulator` parameter of `weave.op` to customize how yielded values are combined from generator functions, for example, to join streamed text tokens into a single string. The accumulator is a two-argument function that Weave calls once per yielded value, building up a result incrementally.

<Note>
  The `accumulator` parameter isn't available for TypeScript.
</Note>

The following example demonstrates a custom accumulator that appends each yielded value to a list, so Weave records that list as the call output after you fully consume the generator.

```python theme={null}
from typing import Generator
import weave

weave.init("your-team-name/your-project-name")

# Weave calls this after each yield. acc is None on the first call.
# The last value you return becomes the traced Op output.
def list_accumulator(acc, value):
    if acc is None:
        acc = []
    acc.append(value)
    return acc

# Set the accumulator parameter
@weave.op(accumulator=list_accumulator)
def basic_gen_with_accumulator(x: int) -> Generator[int, None, None]:
    yield from range(x)

# Iterate to completion so every yield runs and the accumulator can produce the final traced output.
result = list(basic_gen_with_accumulator(3))
print(result)
```


# Use trace plots
Source: https://docs.wandb.ai/weave/guides/tracking/trace-plots

Visualize trace-level metrics with interactive charts in Weave to explore latency, cost, and token usage patterns for LLM applications.

The *trace plots* tool in W\&B Weave lets you explore, visualize, and debug trace-level metrics (for example, latency, cost, and tokens) using interactive charts. You can use the default trace plots or create your own through the custom trace plot builder. This guide shows you how to open the trace plots panel, work with the default plots, and build custom plots tailored to your project's data.

<Frame>
  <img alt="Trace plots in action" />
</Frame>

## Get started

Follow these steps to open the trace plots side pane for your project:

1. Navigate to your project's **Traces** page.
2. Optional: Select **Filter** to filter selected traces (for example, by datetime or operation).
3. In the upper right corner of the **Traces** view, click the **Show Metrics** icon to open the trace plots side pane.

   <img alt="The Show Metrics icon" />

   From here, you can:

   * View the [default trace plots](#default-trace-plots).
   * Create a [custom trace plot](#create-a-custom-trace-plot).
4. Charts update dynamically based on your trace filters and selections.

## Default trace plots

When you first open the trace plots panel, Weave automatically generates trace plots based on your available project trace data:

* Bar chart (cost or latency grouped over time bins).
* Line chart (latency over time).
* Scatter plot (for example, prompt tokens versus completion tokens).

Each trace plot is interactive:

* Hover for tooltips.
* Drag to zoom.
* Double-click to reset.
* Click points in scatter plots to open a specific trace.

## Create a custom trace plot

If the default trace plots don't show the metrics or relationships you need, you can build your own. To create a custom trace plot, do the following:

1. From the trace plots side pane, click **➕ Add Chart**.

   <img alt="➕ Add Chart button." />

2. In the pop-up, select one of the available trace plot types:

   <img alt="Available custom trace plot types" />

3. Configure your trace plot for the selected type. For information on configuration options by trace plot type, see [Trace plot settings by plot type](#trace-plot-settings-by-plot-type).

4. Click **Save chart** to save your chart.

Your custom trace plot now appears in the trace plots side pane alongside the default plots and updates dynamically with your trace filters and selections.

### Trace plot settings by plot type

When you add or edit a custom trace plot, the available configuration options vary depending on the selected trace plot type. The following table describes the configurable options.

| Setting     | Scatter Plot | Line Chart           | Bar Chart            |
| ----------- | ------------ | -------------------- | -------------------- |
| Y-axis      | ✓ Required   | ✓ Required           | ✓ Required           |
| X-axis      | ✓ Selectable | Fixed (`started at`) | Fixed (`started at`) |
| Grouping    | ✓ Optional   | ✓ Optional           | ✓ Optional           |
| Binning     | ✗ Not used   | ✓ Used               | ✓ Used               |
| Aggregation | ✗ Not used   | ✓ Used               | ✓ Used               |


# Trace sub-agents
Source: https://docs.wandb.ai/weave/guides/tracking/trace-sub-agents

Use Weave's sub-agent span to trace sub-agent delegations and view nested agent invocations.

<Note>
  Weave for Agents is in public preview. Features, APIs, and the Agents view UI may change before general availability.
</Note>

This guide shows you how to use W\&B Weave to trace sub-agents so that delegated agent invocations appear as nested spans in the same trace as the parent turn. Tracing sub-agents lets you see the full hierarchy of an agent's reasoning, including which specialist agents the parent called, what they did, and how they contributed to the final answer. This guide is for developers instrumenting multi-agent systems with Weave.

A sub-agent is a delegated agent invocation that runs inside a turn. Use sub-agents when one agent hands off to another, such as when a supervisor agent dispatches a specialist agent.

When you instrument them with Weave, sub-agents emit a nested `invoke_agent` OpenTelemetry (OTel) span in the same trace as the parent turn. In the **Agents** view, this nesting renders as a sub-agent invocation under the turn that triggered it, with its own LLM calls and tool calls grouped beneath.

## Sub-agent data model

Before instrumenting your code, it helps to understand how Weave represents sub-agents in a trace. The `weave.start_subagent` span maps to the OTel `invoke_agent` span and emits the same operation name as the parent turn. Weave distinguishes the two by their parent-child relationship in the trace:

```plaintext theme={null}
Turn (invoke_agent — root span)
├── LLM call (chat)              ← parent agent's reasoning
│   └── SubAgent (invoke_agent)  ← delegation happens here
│       ├── LLM call (chat)      ← sub-agent's own LLM call
│       └── Tool call (execute_tool)
└── LLM call (chat)              ← parent agent synthesizes the final answer
```

Sub-agents inherit the active session's `conversation_id`, so they're grouped with the rest of the conversation in the **Agents** view.

```python lines theme={null}
sub = weave.start_subagent(
    name="research-specialist",   # Required: identifies this sub-agent in the UI.
    model="gpt-4o",               # Optional: defaults to the parent session's model if empty.
)
```

`weave.start_subagent` creates an `invoke_agent` span that automatically becomes a child of whatever span is currently active in OTel context, typically the parent turn or the LLM call that triggered the delegation. OTel context propagation handles the parent-child relationship, so you don't need explicit delegation.

## Trace a single sub-agent

The following example runs a supervisor agent that receives a request and delegates it to a research-specialist sub-agent that uses a Wikipedia search tool to find the answer.

Weave captures the full hierarchy by wrapping the conversation in `weave.start_session` and then a `session.start_turn`. Weave then captures the sub-agent trace using the `weave.start_subagent` block for the specialist, and records each LLM call and tool execution as child spans.

The routing logic is intentionally omitted from these examples to focus on the tracing between agents.

```python lines highlight="4,10,16,22,30" theme={null}
import weave
from weave.session.session import Message, Usage

weave.init("[YOUR-TEAM]/[YOUR-PROJECT]")

with weave.start_session(agent_name="supervisor") as session:
    with session.start_turn(user_message="Research the founders of Anthropic.") as turn:

        # Supervisor LLM call: decide which specialist to delegate to.
        with weave.start_llm(model="gpt-4o", provider_name="openai") as llm:
            llm.input_messages = [Message(role="user", content="Research the founders of Anthropic.")]
            llm.output("Delegating to the research specialist.")
            llm.usage = Usage(input_tokens=80, output_tokens=10)

        # Delegate to the research specialist as a sub-agent.
        with weave.start_subagent(name="research-specialist", model="gpt-4o") as sub:
            with sub.llm(model="gpt-4o", provider_name="openai") as sub_llm:
                sub_llm.input_messages = [Message(role="user", content="Find founders of Anthropic.")]
                sub_llm.output("I should search for this.")
                sub_llm.usage = Usage(input_tokens=120, output_tokens=15)

                with weave.start_tool(name="wikipedia_search", arguments='{"query":"Anthropic"}') as tool:
                    tool.result = "Anthropic was founded by Dario and Daniela Amodei in 2021."

            with sub.llm(model="gpt-4o", provider_name="openai") as sub_llm:
                sub_llm.output("Anthropic was founded by Dario and Daniela Amodei in 2021.")
                sub_llm.usage = Usage(input_tokens=200, output_tokens=25)

        # Back in the supervisor turn: synthesize the final answer.
        with weave.start_llm(model="gpt-4o", provider_name="openai") as llm:
            llm.output("Anthropic was founded by Dario and Daniela Amodei in 2021.")
            llm.usage = Usage(input_tokens=300, output_tokens=20)
```

In the **Agents** view, the sub-agent appears as a nested `invoke_agent` block inside the turn, with its own LLM calls and tool calls grouped beneath. The supervisor's direct LLM calls remain siblings of the sub-agent.

## Trace multiple sub-agents

The following example runs a content-pipeline agent that handles a single request by delegating to three sibling sub-agents in sequence: a `researcher` that gathers facts, a `writer` that drafts the post, and a `reviewer` that polishes the final output.

Weave captures all three sub-agents as siblings under the same turn by opening a separate `weave.start_subagent` block for each. Because each sub-agent inherits the active turn's OTel context, the sub-agents appear as peer `invoke_agent` spans nested under the turn rather than under each other.

```python lines highlight="1,5,11,15,22" theme={null}
with weave.start_session(agent_name="content-pipeline") as session:
    with session.start_turn(user_message="Write a short blog post about Anthropic.") as turn:

        # Researcher sub-agent: gather facts.
        with weave.start_subagent(name="researcher", model="gpt-4o") as researcher:
            with researcher.llm(model="gpt-4o", provider_name="openai") as sub_llm:
                sub_llm.input_messages = [Message(role="user", content="Find key facts about Anthropic.")]
                sub_llm.output("I should search Wikipedia.")
                sub_llm.usage = Usage(input_tokens=80, output_tokens=15)

                with weave.start_tool(name="wikipedia_search", arguments='{"query":"Anthropic"}') as tool:
                    tool.result = "Anthropic was founded by Dario and Daniela Amodei in 2021."

        # Writer sub-agent: draft the post.
        with weave.start_subagent(name="writer", model="gpt-4o") as writer:
            with writer.llm(model="gpt-4o", provider_name="openai") as sub_llm:
                sub_llm.input_messages = [Message(role="user", content="Draft a post using the research.")]
                sub_llm.output("Anthropic, founded in 2021 by Dario and Daniela Amodei, builds AI safety research...")
                sub_llm.usage = Usage(input_tokens=180, output_tokens=120)

        # Reviewer sub-agent: polish the draft.
        with weave.start_subagent(name="reviewer", model="gpt-4o") as reviewer:
            with reviewer.llm(model="gpt-4o", provider_name="openai") as sub_llm:
                sub_llm.input_messages = [Message(role="user", content="Review and tighten the draft.")]
                sub_llm.output("Final post: Anthropic, founded in 2021 by Dario and Daniela Amodei, builds AI safety research...")
                sub_llm.usage = Usage(input_tokens=200, output_tokens=140)
```

In the **Agents** view, the turn contains three sibling sub-agent invocations, each with its own LLM call nested beneath, and the `researcher` includes its tool call. None of the sub-agents are children of each other.

## Trace nested sub-agents

A sub-agent can itself delegate to another sub-agent. Each `start_subagent` call nests under whatever span is currently active in OTel context.

```python lines highlight="1,2,4,5,11,16" theme={null}
with weave.start_session(agent_name="orchestrator") as session:
    with session.start_turn(user_message="Compare Anthropic and OpenAI.") as turn:

        with weave.start_subagent(name="research-coordinator") as coordinator:
            with weave.start_subagent(name="anthropic-researcher") as r1:
                with r1.llm(model="gpt-4o", provider_name="openai") as sub_llm:
                    sub_llm.output("Anthropic facts...")
                    sub_llm.usage = Usage(input_tokens=120, output_tokens=30)

                # Nested: the researcher delegates to its own summarizer sub-agent.
                with weave.start_subagent(name="anthropic-summarizer") as summarizer:
                    with summarizer.llm(model="gpt-4o", provider_name="openai") as sub_llm:
                        sub_llm.output("Anthropic summary: ...")
                        sub_llm.usage = Usage(input_tokens=80, output_tokens=20)

            with weave.start_subagent(name="openai-researcher") as r2:
                with r2.llm(model="gpt-4o", provider_name="openai") as sub_llm:
                    sub_llm.output("OpenAI facts...")
                    sub_llm.usage = Usage(input_tokens=120, output_tokens=30)
```

The example produces three levels of nesting under the turn:

```plaintext theme={null}
turn (invoke_agent)
└── research-coordinator (invoke_agent)
    ├── anthropic-researcher (invoke_agent)
    │   ├── chat
    │   └── anthropic-summarizer (invoke_agent)   ← nested inside anthropic-researcher
    │       └── chat
    └── openai-researcher (invoke_agent)          ← sibling of anthropic-researcher
        └── chat
```

In the **Agents** view, `research-coordinator` appears as a sub-agent of the turn, `anthropic-researcher` and `openai-researcher` appear as siblings under the coordinator, and `anthropic-summarizer` appears as a sub-agent of `anthropic-researcher`.


# Link a W&B run to trace function calls
Source: https://docs.wandb.ai/weave/guides/tracking/trace-to-run

Associate Weave traces with W&B runs for experiment tracking

With W\&B Weave, you can trace function calls in your code and link them directly to the [W\&B runs](https://docs.wandb.ai/models/runs/) in which they ran.
When you trace a function with `@weave.op()` and call it inside a `wandb.init()` context, Weave automatically associates the trace with the W\&B run.
The **Traces** table shows links to any associated runs.

## View a W\&B run in the Traces table

<Tabs>
  <Tab title="Python">
    The following Python code shows how Weave links traced Ops to W\&B
    runs when you run them inside a `wandb.init()` context. These traces appear in the
    Weave UI and are associated with the corresponding run.

    To view a W\&B run as a Weave trace:

    1. In the terminal, install dependencies.

    ```bash lines theme={null}
    pip install wandb weave
    ```

    2. Log in to W\&B.

    ```bash lines theme={null}
    wandb login
    ```

    3. In the following script, replace `your-team-name/your-project-name` with your W\&B entity and project:

    ```python lines theme={null}
    import wandb
    import weave

    def example_wandb(projname):
        # Split projname into entity and project
        entity, project = projname.split("/", 1)

        # Initialize Weave context for tracing
        weave.init(projname)

        # Define a traceable Op
        @weave.op()
        def say(message: str) -> str:
            return f"I said: {message}"

        # First W&B run
        with wandb.init(
            entity=entity,
            project=project,
            notes="Experiment 1",
            tags=["baseline", "paper1"],
        ) as run:
            say("Hello, world!")
            say("How are you!")
            run.log({"messages": 2})

        # Second W&B run
        with wandb.init(
            entity=entity,
            project=project,
            notes="Experiment 2",
            tags=["baseline", "paper1"],
        ) as run:
            say("Hello, world from experiment 2!")
            say("How are you!")
            run.log({"messages": 2})

    if __name__ == "__main__":
        # Replace this with your actual W&B username/project
        example_wandb("your-team-name/your-project-name")
    ```

    4. Run the script.

    ```bash lines theme={null}
    python weave_trace_with_wandb.py
    ```

    5. Navigate to the [W\&B app](https://wandb.ai) and select your project.
    6. In the **Weave project sidebar**, click **Traces**. The **Traces** table displays links to any associated runs.

    You can now open any trace from the **Traces** table to inspect the linked W\&B run and review its associated experiment context.
  </Tab>

  <Tab title="TypeScript">
    ```text theme={null}
    This feature is not available for the TypeScript SDK yet.
    ```
  </Tab>
</Tabs>


# Navigate the Weave Trace view
Source: https://docs.wandb.ai/weave/guides/tracking/trace-tree

Use Weave's Trace view to make sense of complex execution paths in your LLM and agentic apps.

The Weave Trace view helps you make sense of complex execution paths in your LLM and agentic apps. Whether you're debugging an agentic app with dozens of nested calls, or tracking the flow of a single model prediction, the Trace view provides a clear visualization of what happened. It also provides alternate ways to view and understand your application flow.

This guide is for developers who use Weave to instrument LLM or agentic applications and want to inspect the resulting traces. It describes how to move through the trace stack, filter and search for functions called by your code, switch between visual representations, and more.

The following sections describe how to open the Trace view, orient yourself within the Traces page, navigate and filter a trace, switch between alternative visualizations, and inspect details for a specific op.

## Get started

Follow these steps to open the Trace view for a project so you can begin inspecting traces:

1. Navigate to [https://wandb.ai](https://wandb.ai) and select your project.
2. In the project sidebar, select **Traces** to view all traces saved for your project.
3. Select a trace to open the Trace Details view. The Trace Details view displays additional panels with a hierarchical breakdown of the trace execution.

## Traces page overview

The Traces page is composed of three core panels:

* **Left panel**: A sortable, paginated list of all traces for the project.
  * This traces table includes additional data such as tokens, cost, and latency.
* **Center panel**: Interactive trace view for a selected trace. The trace tree shows a hierarchy of all the methods tracked within the trace.
  * The trace tree displays [ops](/weave/guides/tracking/ops#automatically-track-function-calls-using-ops), which are `@weave.op()`-decorated functions that were called during the trace.
* **Right panel**: Details for a selected op within the selected trace.

<Frame>
  <img alt="Traces page showing a selected trace and selected op details" />
</Frame>

### Filter within a trace

You can filter the contents of a trace using the following controls:

* **Regex filter by name(s)**: Use the text field above the trace tree to filter ops by name or type, such as `tool`, `openai.response.create`.
* **Metrics**: Control whether to display the following data metrics when available: cost, tokens, and latency.

### Navigate a trace

The trace tree shows a hierarchy of all the methods tracked within the trace. To move up or down the tree, use `Cmd` (macOS) or `Alt` (Windows/Linux) + Up Arrow (↑) or Down Arrow (↓).

Several scrubbers below the trace tree provide rapid navigation across states within it. You can use the sliders to navigate through your trace.

Expand the panel to see all available scrubbers:

* **Timeline**: Chronological order of events within the trace.
* **Peers**: Ops sharing the same type. For example, if you're examining details of a function called `predict`, you can use this scrubber to immediately jump to the next execution of `predict` within the trace.
* **Siblings**: Ops with the same parent. Use this scrubber to iterate over ops nested under the parent function call.
* **Stack**: Traverse up or down the call stack.
* **Path**: (Only available in code composition view) Iterate through all calls with the same code path as the selected call.

<Frame>
  <img alt="Trace tree panel showing search filter and scrubbers" />
</Frame>

### Alternative trace tree views

At the top of the panel, you can switch between multiple visual representations of the trace tree based on your debugging needs. Use **code composition view** to understand call logic, **flame graph view** to understand performance over time, and **graph view** to understand structure.

The following sections describe each available view.

#### Traces (default)

The default view of the trace tree shows stack hierarchy, cost per op (if available), execution time, and status indicators.

#### Code composition view

In the code composition view, boxes represent ops and their nested calls. This is helpful for visualizing the flow of function calls. In this view, you can select a box to drill into that op and filter the call path.

<Frame>
  <img alt="Trace view showing the code view of a trace" />
</Frame>

#### Flame graph

The flame graph view provides a timeline-based visualization of execution depth and duration. This is helpful when you want to understand performance diagnostics over time. You can select into frames to isolate sub-traces.

<Frame>
  <img alt="Trace view showing a flame graph of ops within a trace" />
</Frame>

#### Graph view

The graph view shows hierarchical relationships between ops. This is helpful for understanding parent/child relationships.

<Frame>
  <img alt="Trace view showing a functional graph view of ops and relationships within a trace" />
</Frame>

### View details for a trace op

After you select an op in the trace tree, details for that op display in the next panel. These details are grouped into the following tabs:

* **Call**: The input and output to the op execution.
* **Code**: The code that was used when the call was made.
* **Feedback**: Any available [feedback](/weave/guides/tracking/feedback) for the op. You can provide feedback directly within Weave or through the API.
* **Scores**: Any available [scores](/weave/guides/evaluation/scorers) for the op. Calls are scored by running Evaluations.
* **Summary**: General information about the op.
* **Use**: Code snippets that you can use to programmatically retrieve the call and add reactions, notes, or feedback.


# Understand Ops, Calls, and Traces
Source: https://docs.wandb.ai/weave/guides/tracking/tracing

Learn how Ops, Calls, and Traces create the foundation of W&B Weave's tracing system.

W\&B Weave's tracing system is built on four related concepts: Ops, Calls, Traces, and Threads. This page defines each concept and explains how they relate to one another, so you can interpret what Weave captures when you instrument and run your application.

## Ops

An **Op** is a versioned, tracked function. When you decorate a function with `@weave.op()` (Python) or wrap it with `weave.op()` (TypeScript), Weave automatically captures its code, inputs, outputs, and execution metadata. Ops are the building blocks of tracing, evaluation scorers, and any tracked computation.

<CodeGroup>
  ```python Python theme={null}
      @weave.op
      async def my_function(){
        ...  }
  ```

  ```typescript Typescript theme={null}
  function myFunction() {
      ...
  }

  const myFunctionOp = weave.op(myFunction)
  ```
</CodeGroup>

## Calls

A **Call** is a logged execution of an Op. Every time an Op runs, Weave creates a Call that captures:

* Input arguments
* Output value
* Timing and latency
* Parent-child relationships (for nested calls)
* Any errors that occur

Calls show up as **Traces** in the Weave UI and provide the data for debugging, analysis, and evaluation. For the full Call object structure and properties, see the [Call schema reference](/weave/guides/tracking/call-schema-reference).

Calls are similar to spans in the [OpenTelemetry](https://opentelemetry.io) data model. A Call can:

* Belong to a Trace, a collection of Calls in the same execution context.
* Have parent and child Calls, forming a tree structure.

## Traces

Traces are full trees of Calls that share the same execution context. Each Trace contains an ID (`trace_id`) you can use to retrieve the entire tree of Calls. If you retrieve Call information using the Call's ID, Weave returns data about the specified Call only, and none of its child Calls.

## Threads

Threads are collections of Traces related to a single session or conversation. You can use Threads to analyze or score the entire conversation instead of individual Calls.

The following diagram illustrates the relationships between Threads, Traces, and Calls:

```text theme={null}
Thread: "session-abc"
  ├── Turn 1 (trace_id: aaa) -> user says "Hi"
  │     ├── LLM call
  │     └── format response
  ├── Turn 2 (trace_id: bbb) -> user says "What is the capital of France?"
  │     ├── RAG retrieval
  │     ├── LLM call
  │     └── format response
  └── Turn 3 (trace_id: ccc) -> user says "Thanks"
        └── LLM call
```


# Update and delete Calls
Source: https://docs.wandb.ai/weave/guides/tracking/update-call

Modify display names, add feedback, and delete Calls in W&B Weave

This page describes how to modify Calls in W\&B Weave after they've been created. Most Call properties are immutable after creation, but Weave supports the following mutations:

* [Set display name](#set-display-name)
* [Add feedback](#add-feedback)
* [Delete a Call](#delete-a-call)

You can perform all of these mutations from the UI by navigating to the Call detail page. To update a Call in the web app:

1. Navigate to [wandb.ai](https://wandb.ai/) and select your project.
2. In the Weave project sidebar, click **Traces**.
3. Find the Call you want to view in the table.
4. Click the Call to open its details page.
5. Click the **Feedback** tab in the Call detail's tab bar.

From the Call detail page, you can edit the display name of the Call, add feedback, or delete the Call.

<Frame>
  <img alt="Weave Traces page excerpt of a selected Call details panel showing how you can rename or delete a Call" />
</Frame>

## Set display name

<Tabs>
  <Tab title="Python">
    To set the display name of a Call, use the [`Call.set_display_name()`](/weave/reference/python-sdk/trace/weave_client#method-set-display-name) method.

    ```python lines theme={null}
    import weave

    # Initialize the client
    client = weave.init("your-project-name")

    # Get a specific Call by its ID
    call = client.get_call("call-uuid-here")

    # Set the display name of the Call
    call.set_display_name("My Custom Display Name")
    ```
  </Tab>

  <Tab title="TypeScript">
    To set the display name of a Call, use [`client.updateCall`](/weave/reference/typescript-sdk/classes/weaveclient#updatecall) to update by call ID directly:

    ```typescript lines theme={null}
    import * as weave from 'weave'

    // Initialize the client
    const client = await weave.init('your-project-name')

    // Update the display name of a Call by its ID
    await client.updateCall('call-uuid-here', 'My Custom Display Name')
    ```
  </Tab>

  <Tab title="HTTP API">
    To set the display name of a Call using the Service API, make a request to the [`/call/update`](https://docs.wandb.ai/weave/reference/service-api/calls/call-update) endpoint.

    ```bash lines theme={null}
    curl -L 'https://trace.wandb.ai/call/update' \
    -H 'Content-Type: application/json' \
    -H 'Accept: application/json' \
    -d '{
        "project_id": "string",
        "call_id": "string",
        "display_name": "string",
    }'
    ```
  </Tab>
</Tabs>

You can also [set a Call's display name at execution](/weave/guides/tracking/get-call-object).

## Add feedback

For more information, see [Collect feedback and use annotations](/weave/guides/tracking/feedback).

## Delete a Call

<Tabs>
  <Tab title="Python">
    To delete a Call using the Python API, use the [`Call.delete`](/weave/reference/python-sdk/trace/weave_client#method-delete) method.

    ```python lines theme={null}
    import weave

    # Initialize the client
    client = weave.init("your-project-name")

    # Get a specific Call by its ID
    call = client.get_call("call-uuid-here")

    # Delete the Call
    call.delete()
    ```
  </Tab>

  <Tab title="TypeScript">
    ```plaintext lines theme={null}
    This feature is not available in the TypeScript SDK yet.
    ```
  </Tab>

  <Tab title="HTTP API">
    To delete a Call using the Service API, make a request to the [`/calls/delete`](https://docs.wandb.ai/weave/reference/service-api/calls/calls-delete) endpoint.

    ```bash lines theme={null}
    curl -L 'https://trace.wandb.ai/calls/delete' \
    -H 'Content-Type: application/json' \
    -H 'Accept: application/json' \
    -d '{
        "project_id": "string",
        "call_ids": [
            "string"
        ],
    }'
    ```
  </Tab>
</Tabs>

## Delete multiple Calls

<Tabs>
  <Tab title="Python">
    To delete batches of Calls using the Python API, pass a list of Call IDs to `delete_calls()`.

    <Important>
      * You can delete a maximum of `1000` Calls at a time.
      * Deleting a Call also deletes all of its children.
    </Important>

    ```python lines theme={null}
    import weave

    # Initialize the client
    client = weave.init("my-project")

    # Get all Calls from client
    all_calls = client.get_calls()

    # Get list of first 1000 Call objects
    first_1000_calls = all_calls[:1000]

    # Get list of first 1000 Call IDs
    first_1000_calls_ids = [c.id for c in first_1000_calls]

    # Delete first 1000 Calls by ID
    client.delete_calls(call_ids=first_1000_calls_ids)
    ```
  </Tab>

  <Tab title="TypeScript">
    ```plaintext lines theme={null}
    This feature is not available in the TypeScript SDK yet.
    ```
  </Tab>
</Tabs>


# View agent activity
Source: https://docs.wandb.ai/weave/guides/tracking/view-agent-activity

Use W&B Weave's Agents view to understand what your agent did, how much it cost, and exactly where things went right or wrong.

<Note>
  Weave for Agents is in public preview. Features, APIs, and the Agents view UI may change before general availability.
</Note>

The Agents view gives you a turn-by-turn record of every conversation your agent had, along with token usage, tool invocations, and execution spans.

Agent applications are hard to debug because the interesting behavior happens between the user's request and the final response. The Agents view in W\&B Weave makes that middle layer visible. Every conversation your agent had is captured here, with the full message history, span-level execution detail, and token costs attached. You can see at a glance whether an agent completed its task, how many tool calls it made, and where time or budget was spent. For teams building and iterating on agents, this is the starting point for understanding behavior in production.

## Get started

To open the Agents view for your project:

1. Navigate to [https://wandb.ai](https://wandb.ai) and select your project.
2. In the sidebar menu, select **Agents** to view all agent conversations saved for your project.

The Agents view opens on the **Agents** tab, where each agent that has logged traces to the project appears as a card. From here, you can drill into individual conversations, spans, and signals using the tabs described in the following sections.

## Agents tab

The **Agents** tab gives you a high-level view of all agents that have logged
traces to this project. Use it to spot which agents are active and to compare
latency and error rates across agents. It also helps you identify agents that need
attention before drilling into individual conversations.

<img alt="The Agents tab showing a grid of agent cards, each displaying conversation count, span count, average latency, and error rate." />

It's useful for scenarios such as:

* **Monitoring a fleet of agents.** The card grid lets you compare latency and
  error rate across all agents at once without opening individual conversations.
  A latency spike or a newly red error rate on one card signals a regression
  worth investigating.
* **Identifying stale agents.** Sorting by **Last seen** highlights agents that
  haven't recorded activity recently. This is useful for confirming a deployment
  is live or spotting agents that may have stopped logging traces unexpectedly.
* **Comparing versions.** The version count on each card shows how many
  distinct versions of that agent have been deployed. A high version count
  alongside a rising error rate may indicate a regression introduced in a recent
  deployment.
* **Drilling into an agent.** Click any card to open the detail panel for that
  agent, from which you can navigate to its conversations or spans:

<img alt="An agent card selected in the Agents tab, with the detail panel open on the right showing agent metadata and navigation options." />

### Agent cards

Each agent is represented as a card showing:

| Field                  | Description                                                                             |
| ---------------------- | --------------------------------------------------------------------------------------- |
| **Agent name**         | The name logged with the agent's traces.                                                |
| **Last seen**          | How long ago the agent last recorded activity.                                          |
| **Version**            | The number of distinct `agent_version` values recorded across the agent's spans.        |
| **Activity histogram** | A bar chart of recent conversation volume, giving a quick sense of usage trends.        |
| **Conversations**      | Total number of conversations recorded.                                                 |
| **Spans**              | Total number of spans recorded across all conversations.                                |
| **Latency (avg.)**     | Average end-to-end duration per invocation.                                             |
| **Error rate**         | Percentage of invocations that returned an error. Displays in red when greater than 0%. |

### Find and sort agents

Use the **Search agents** field to filter cards by agent name.

Use the sort dropdown (default: **Last seen**) to reorder the grid. The
available sort options are:

* **Last seen**: Most recently active agents first.
* **Most invocations**: Highest conversation volume first.
* **Most input tokens**: Highest token consumption first.
* **Most errors**: Highest error count first.

Sorting by **Most errors** is useful for a quick daily health check: agents
with non-zero error rates surface immediately, and the red error rate on the
card confirms at a glance which need investigation.

## Conversations tab

The **Conversations** tab on the Agents page lets you browse, filter, and
inspect individual agent runs. Use it to investigate failures, measure token
costs, and understand the sequence of LLM calls and tool executions that made
up a run.

For high-level questions about what an agent said and did across a conversation, start with the Conversations tab.

<img alt="The Conversations tab showing a list of agent conversations with columns for agent name, invocations, spans, token counts, and last activity." />

### Conversations table

The conversation table shows one row per conversation. The following columns
appear by default:

| Column            | Description                                                                             |
| ----------------- | --------------------------------------------------------------------------------------- |
| **Conversation**  | The conversation ID and a preview of the first message.                                 |
| **Last message**  | A preview of the most recent message, with a role indicator.                            |
| **Agent**         | The name of the agent or agents involved.                                               |
| **Invocations**   | How many times the agent was invoked during the conversation.                           |
| **Spans**         | Total number of spans recorded. Higher span counts indicate more branching or tool use. |
| **In tokens**     | Input tokens consumed.                                                                  |
| **Out tokens**    | Output tokens generated.                                                                |
| **Started**       | When the conversation began.                                                            |
| **Last activity** | How long ago the last message was recorded.                                             |

To show or hide additional columns, click **Columns** in the toolbar.

### Filters and time window

Use the **Filter** bar to narrow results by agent, model, error status, or
other attributes.

Use the time window selector (**1m**, **1h**, **6h**, **24h**, **7d**, or
**30d**) to restrict the list to conversations that were active within that
period. The conversation volume histogram above the list updates to reflect the
selected window.

Hover over any column header in the conversation list to filter that column to
a specific value or range.

### Agent conversation detail

Click a conversation row to open a detail panel with two sub-tabs: **Turns** and **Events**.

<img alt="A conversation selected in the Conversations tab, with the detail panel open showing the turn-by-turn message thread, tool calls, reasoning, and the Events timeline on the right." />

#### Turns

The conversation detail turns panel shows each turn in chronological order, numbered from 1.

Each turn displays the number of intermediate responses and tool calls, and
the total wall-clock duration. Expand a turn to see the full message thread.

##### Messages

Within a turn, messages are grouped by role.

**User messages** show the message text and any attached media or content
references.

**Assistant messages** show the following:

* The agent name and the model used (for example, `gpt-5.5-2026-04-23`).
* Timestamp and duration.
* Input and output token counts (for example, `16086 in 295 out`).
* An expandable **Reasoning** section when the model used extended thinking.
* The response text, which collapses automatically for long responses.

**Tool calls** show the tool name, timestamp, and duration. If argument or
result data is available, the tool call is expandable and shows **Args** and
**Result** in a key-value table. If the call failed, an **ERROR** badge
appears.

##### Error states

When a tool call returns an error status, a red **ERROR** badge appears inline
next to it. In the Events timeline, that event also displays in red regardless
of its type.

#### Events

The **Events** panel on the right shows a color-coded strip that represents
the sequence of events within the selected turn.

In the Events timeline, each segment's color indicates the event type.

| Color   | Event type                       |
| ------- | -------------------------------- |
| Purple  | User message                     |
| Green   | Assistant message                |
| Blue    | Tool call                        |
| Sienna  | Sub-agent invocation             |
| Magenta | Agent handoff                    |
| Gray    | Context compaction               |
| Red     | Any event that returned an error |

Use the Events timeline to get a quick sense of how a turn was structured. For
example, you can see whether it was LLM-heavy, tool-heavy, or involved sub-agent
delegation before reading the full message thread.

##### Scores

If any signals are active for the project, a **Scores** section provides metrics for the conversation. It shows the signal scorer name, an overall numeric rating
from 0 to 1, a confidence percentage, and the individual rubric points that
contributed to the score. Each rubric point also shows its own confidence. Use this to
understand not just whether a turn scored well, but which specific rubric
criteria passed or failed.

##### Meta summary

The **Meta summary** section shows aggregate statistics for the selected
conversation.

| Field            | Description                                                    |
| ---------------- | -------------------------------------------------------------- |
| **Tokens**       | Total input and output tokens.                                 |
| **Tool calls**   | Number of tool calls across all turns.                         |
| **Messages**     | Total message count.                                           |
| **Session time** | Wall-clock duration from first to last message.                |
| **Turn page**    | Which turns are currently displayed, and the total turn count. |

##### Token breakdown

The **Token breakdown** section shows cache and reasoning details for the
selected conversation.

| Field               | Description                                                                           |
| ------------------- | ------------------------------------------------------------------------------------- |
| **Cache read**      | Tokens served from the prompt cache.                                                  |
| **Cache written**   | Tokens written to the prompt cache.                                                   |
| **Cache hit rate**  | Percentage of input tokens served from cache. A higher rate reduces cost and latency. |
| **Reasoning**       | Tokens spent on extended thinking.                                                    |
| **Reasoning ratio** | Percentage of output tokens spent on extended thinking.                               |

##### Participants

The **Participants** section lists the agents and models involved in the
conversation. In multi-agent conversations, different turns may show different
model names here.

## Spans tab

The **Spans** tab shows every individual span recorded across all agent
activity in the project. The Conversations tab aggregates activity into
dialogue-level rows. The Spans tab exposes the raw operations underneath: each
LLM call, tool execution, and agent invocation as its own row. Use it to trace
exactly which call was slow, which model consumed unexpected tokens, or which
tool invocation failed.

<img alt="The Spans tab showing spans of the agent conversation." />

### Spans table

The span table shares most columns with the Conversations table (agent, model,
tool, token counts, status). Some columns unique to this view are:

| Column        | Description                                                                                                                                        |
| ------------- | -------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Span**      | The span name and ID, with its trace ID below.                                                                                                     |
| **Kind**      | The OpenTelemetry span kind for this operation (such as `INTERNAL`, `SERVER`, or `CLIENT`).                                                        |
| **Operation** | The operation type (such as `chat`, `execute_tool`, or `invoke_agent`).                                                                            |
| **Finished**  | The finish reason returned by the model (such as `stop` or `max_tokens`). Populated only for `chat` spans where the model reports a finish reason. |

Additional columns for cache token breakdowns, reasoning tokens, LLM
parameters, and W\&B run metadata are available through the **Columns** button.

The Spans tab is most useful when you need operation-level precision that the
Conversations tab doesn't provide:

* **Identifying expensive calls.** Sort by **In** or **Out** tokens to find
  which individual LLM calls are driving cost, rather than seeing totals at
  the conversation level.
* **Debugging a specific operation type.** Filter by **Operation** to isolate
  all `execute_tool` spans and check error rates, or all `chat` spans for a
  specific model.
* **Investigating truncation.** Filter **Finished** by `max_tokens` to find
  spans where the model hit its token limit rather than completing normally.
* **Correlating with a W\&B run.** Hidden-by-default columns expose W\&B run IDs
  and run steps, letting you link a specific span back to a training or
  evaluation run in W\&B.

### Trace groups

Click any row to select its trace and highlight all other spans that share the same trace ID. This shows you the full set of operations that ran as part of one agent invocation. Grouping here is by trace, not by conversation. This means a single conversation may contain multiple traces if it involved sub-agent delegation.

### Agent invocation detail

Click a row in the **Spans** table to open a detail panel populated with data from the complete agent invocation.

At the top of the detail panel, a flame graph shows the wall-clock
position and relative duration of every span in the selected trace. Each span
appears as a colored block scaled to its actual duration, positioned at its start
time in milliseconds from the beginning of the trace. Use the timeline to:

* **Spot the longest operation at a glance.** Wide blocks indicate spans that
  dominated total latency.
* **See parallelism.** Overlapping blocks indicate spans that executed
  concurrently rather than sequentially.
* **Inspect any span inline.** Click a block in the timeline to load that
  span's details into view, including its input messages and output messages,
  token counts, and other metadata.

<img alt="The Spans tab with a trace selected, highlighting a group of related spans in the table and showing the flame graph in the detail panel on the right." />

You can alternatively view the child spans as a hierarchical trace tree by selecting the **Show trace tree** icon in the detail panel header.

## Signals tab

The **Signals** tab shows tags and ratings for your agent's conversations. Signals surface quality and safety issues to flag problems, find patterns, and highlight the traces that need your attention. Use signals to automatically score the quality of your agent's responses, notice when a user is frustrated, or flag NSFW content.

For setup and detailed usage, see [Monitor your agents with signals](/weave/guides/tracking/view-agent-signals).


# Monitor your agents with signals
Source: https://docs.wandb.ai/weave/guides/tracking/view-agent-signals

Use W&B Weave signals to score agent turns, apply tags, and surface quality and safety issues in production.

<Note>
  Weave for Agents is in public preview. Features, APIs, and the Agents view UI may change before general availability.
</Note>

In W\&B Weave's [Agents view](/weave/guides/tracking/view-agent-activity), the Signals tab shows tags and ratings for your agent's conversations. Signals surface quality and safety issues to flag problems, find patterns, and highlight the traces that need your attention. Use signals to automatically score the quality of your agent's responses, notice when a user is frustrated, or flag NSFW content.

## Get started

To view signals for your project:

1. Navigate to [https://wandb.ai](https://wandb.ai) and select your project.
2. In the sidebar menu, select **Agents** to view all agent conversations saved for your project.
3. In the tab bar, select **Signals**.

<img alt="The Signals tab showing a list of scored turns for the agent." />

## Key terms

* **Turn**: One back-and-forth exchange between the user and the agent.
* **Rating**: A numeric score between 0.0 and 1.0 assigned to a matching span.
* **Tags**: Labels assigned to matching spans, such as "user-frustration" or "nsfw".

## Signals table

The Signals tab displays a table of scored turns from your agent. Each row represents the output of one of your signal monitors. The following columns appear by default.

| Column           | Description                                                                                                                                                                               |
| ---------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Type**         | The part of the conversation that gets scored. Only `turn` is supported.                                                                                                                  |
| **Scorer**       | The name of the signal that produced this score.                                                                                                                                          |
| **Last message** | A preview of the last message in the scored turn, with the role shown below.                                                                                                              |
| **Agent**        | The agent associated with the scored turn.                                                                                                                                                |
| **Scores**       | The numeric rating from 0.0 to 1.0, or a tag if matched. We recommend using consistent ratings where 1 indicates good and 0 indicates bad, but your scorers can use any scale you define. |
| **Trend**        | Displays an inline chart that shows how this signal trends over time. Shows either the average value (for ratings) or the count (for tags).                                               |
| **When**         | When the signal was scored.                                                                                                                                                               |

Use the time window selector and **Filter** bar to narrow results by scorer, agent, score range, or time period. The **Score volume** timeline shows counts of signals that Weave evaluated for ratings or tagged. It reflects the rows shown in the table and supports drag-to-filter on the timeline.

## Create a new signal

To start scoring your agent's turns, create a signal. Select **+ New signal** to create a scorer for your agent, then configure the following options.

### Scorer type

Choose to create either a **Rating** scorer or a **Tags** scorer.

* **Rating**: Assigns a score between 0 and 1 to each matching span.
* **Tags**: Assigns up to 10 tags to each matching span. The signals UI only displays rows for spans that matched at least one tag, so your tag scorer might run successfully even if you don't see any output.

### Only score turns matching

Use this selector to restrict which turns the signal scores, such as a specific `Agent name`, `Operation name`, `Tool name`, or `Request model`. Weave combines multiple filters with `AND` logic. To score every turn, select the **x** at the end of the filter row to remove it.

### Prompt template

Choose a starter template from the following tables, then adjust the prompt that appears inline as **Scorer prompt**. Weave resolves template variables, such as `{input_messages}`, `{output_messages}`, and `{system_instructions}`, during scoring.

#### Rating templates

| Template              | What it evaluates                                                                                                                                        |
| --------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **User Satisfaction** | Whether the user is satisfied (positive feedback, follow-up engagement, task completion) or dissatisfied (complaints, repeated rephrasing, abandonment). |
| **User Good Intent**  | Whether the user's intent is benign and legitimate, versus jailbreak attempts, harmful requests, or prompt injection.                                    |
| **Safe-for-Work**     | Whether the conversation is appropriate for any professional setting, versus explicit, violent, or otherwise inappropriate workplace content.            |
| **Response Quality**  | Whether the agent's response is accurate, complete, and directly addresses the user's request.                                                           |

#### Tags templates

| Template                            | What it detects                                                                                                                                                                      |
| ----------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| **User Frustration**                | User shows signs of frustration, anger, confusion, or dissatisfaction.                                                                                                               |
| **Malicious Intent (Jailbreaking)** | User attempts to jailbreak the system, extract restricted content, perform prompt injection, use role-play exploits, or otherwise manipulate the agent into ignoring its guardrails. |
| **NSFW**                            | User input or agent output contains explicit sexual content, graphic violence, or other material inappropriate for a workplace setting.                                              |
| **Low Quality Response**            | Agent output that is factually wrong, off-topic, evasive, repetitive, lacks justification when refusing, or otherwise fails to address the user's request.                           |

### Scorer name

Choose the display name for this signal.

### Advanced settings

Under **Advanced**, configure the following options.

* **Inference model**: The LLM to use for scoring. [Serverless Inference](/inference/) is the recommended default when available.
* **Sample rate**: For high-traffic agents, set a sample rate to score a fraction of turns instead of every turn and reduce cost.

## Manage and edit signals

Select **Manage signals** to open a drawer that lists all active signals for the project. From there you can toggle signals on or off, delete them, or edit any signal. The editor shows the same fields as **+ New signal**.

## Troubleshoot signals

Signal activity appears under **Traces** in the project sidebar. If you don't see the expected signal matches, debug using the **Traces** table. For example, check the scorer name and the **Status** column for error conditions. Scorer execution errors show a red indicator for **Status** and include error details.


# View and customize trace display
Source: https://docs.wandb.ai/weave/guides/tracking/view-call

View Calls in the UI and customize how trace data is displayed

After you create Calls in W\&B Weave, you often want to open a single call to inspect its inputs, outputs, and metadata. This page shows how to view a call in the UI or in the SDK, and how to customize how Weave renders trace data in the UI using `weave.Markdown`. Use the following tabs to choose the interface that fits your workflow.

<Tabs>
  <Tab title="Web App">
    To view a Call in the UI:

    1. Navigate to [wandb.ai](https://wandb.ai/) and select your project.
    2. In the Weave project sidebar, click **Traces**.
    3. Find the Call you want to view in the table.
    4. Click the Call to open its details page.

    For details on the Trace view, see [Navigate the Weave Trace view](/weave/guides/tracking/trace-tree).
  </Tab>

  <Tab title="Python">
    To view a call using the W\&B Weave Python SDK, use the [`get_call`](/weave/reference/python-sdk/trace/weave_client#method-get_call) method:

    ```python lines theme={null}
    import weave

    # Initialize the client
    client = weave.init("your-project-name")

    # Get a specific call by its ID
    call = client.get_call("call-uuid-here")

    print(call)
    ```
  </Tab>

  <Tab title="TypeScript">
    ```typescript lines theme={null}
    import * as weave from 'weave'

    // Initialize the client
    const client = await weave.init('intro-example')

    // Get a specific call by its ID
    const call = await client.getCall('call-uuid-here')

    console.log(call)
    ```
  </Tab>

  <Tab title="HTTP API">
    To view a call using the Service API, make a request to the [`/call/read`](https://docs.wandb.ai/weave/reference/service-api/calls/call-read) endpoint.

    ```bash theme={null}
    curl -L 'https://trace.wandb.ai/call/read' \
    -H 'Content-Type: application/json' \
    -H 'Accept: application/json' \
    -d '{
        "project_id": "string",
        "id": "string",
    }'
    ```
  </Tab>
</Tabs>

## Customize rendered traces with `weave.Markdown`

After you can view a Call, you may want to control how its inputs and outputs appear in the UI. Use `weave.Markdown` to customize how Weave displays your trace information without losing the original data. This lets you render your inputs and outputs as readable blocks of formatted content while preserving the underlying data structure.

<Tabs>
  <Tab title="Python">
    Use `postprocess_inputs` and `postprocess_output` functions in your `@weave.op` decorator to format your trace data. The following code sample uses postprocessors to render a call in Weave with more readable formatting:

    ```python lines theme={null}
    import weave

    def postprocess_inputs(query) -> weave.Markdown:
        search_box = f"""
    **Search Query:**
    ``+`
    {query}
    ``+`
    """
        return {"search_box": weave.Markdown(search_box),
                "query": query}

    def postprocess_output(docs) -> weave.Markdown:
        formatted_docs = f"""
    # {docs[0]["title"]}

    {docs[0]["content"]}

    [Read more]({docs[0]["url"]})

    ---

    # {docs[1]["title"]}

    {docs[1]["content"]}

    [Read more]({docs[1]["url"]})
    """
        return weave.Markdown(formatted_docs)

    @weave.op(
        postprocess_inputs=postprocess_inputs,
        postprocess_output=postprocess_output,
    )
    def rag_step(query):
        # example newspaper articles of the companies on the S&P 500 
        docs = [
            {
                "title": "OpenAI",
                "content": "OpenAI is a company that makes AI models.",
                "url": "https://www.openai.com",
            },
            {
                "title": "Google",
                "content": "Google is a company that makes search engines.",
                "url": "https://www.google.com",
            },
        ]
        return docs

    if __name__ == "__main__":
        weave.init('markdown_renderers')
        rag_step("Tell me about OpenAI")
    ```
  </Tab>

  <Tab title="Typescript">
    ```plaintext theme={null}
    This feature is not available in the TypeScript SDK yet.
    ```
  </Tab>
</Tabs>

After you run the op, Weave renders the formatted Markdown alongside the original data in the Call's details page. The following screenshot shows the unformatted and formatted outputs side by side.

<img alt="A Call rendered in the Weave UI using the code sample." />


# Write-ahead log
Source: https://docs.wandb.ai/weave/guides/tracking/write-ahead-log

Improve the resilience of W&B Weave trace data capture with the write-ahead log

The write-ahead log (WAL) persists API requests to disk before sending them to the server. By default, the client buffers requests in memory. If the process exits unexpectedly, buffered data is lost. Weave's WAL addresses this by writing requests to disk before sending them.

When you enable the WAL, the client writes each API request to a local JSONL file first, then flushes the file to the server. If a process crashes or the server is unreachable, the data remains on disk and Weave sends it automatically when the client runs again.

The WAL is especially useful in environments where processes might be interrupted or the server might be temporarily unavailable:

* **Container orchestration**: Pods that might be evicted or OOM-killed before background threads finish uploading traces.
* **Distributed training**: Multiple processes that write traces in parallel, where any single process might fail.
* **Unreliable networks**: Environments where connectivity to the Weave server is intermittent.
* **Batch jobs**: Long-running jobs where losing trace data from a crash is costly.

For short-lived environments like serverless functions, also consider calling `weave.flush()` or `weave.finish()` before the process exits to ensure all data is uploaded. For more information, see [Trace data loss in worker processes](/support/weave/articles/trace-data-loss-in-worker-processes).

<Note>
  The write-ahead log is opt-in. It will be enabled by default in a future release.
</Note>

## Enable the write-ahead log

To enable the WAL for your Weave client, set the `WEAVE_ENABLE_WAL` environment variable to `true`:

```bash theme={null}
export WEAVE_ENABLE_WAL=true
```

You can also set it in Python before calling `weave.init()`:

```python theme={null}
import os
os.environ["WEAVE_ENABLE_WAL"] = "true"
```

No other code changes are required. The WAL works transparently with existing Weave tracing code.

## How it works

When the WAL is enabled:

1. Weave appends each call to the Weave API (object creates, call starts, call ends, and so on) to a JSONL file on disk instead of holding it only in memory.
2. Each process writes to its own log file, so parallel processes don't conflict.
3. A background sender reads the log files and flushes their contents to the Weave server.
4. After the data is successfully sent, Weave removes the log file.

Log files are stored in `.weave/wal/` within your working directory, organized by entity and project. Each file contains the raw API requests as JSON objects, one per line.

When the client starts, it checks for existing log files from previous runs. If any are found, the sender flushes them along with any new data. This means Weave automatically recovers data written by a process that crashed the next time the client runs.

## Environment variables

The following environment variables control WAL behavior.

| Variable                   | Type   | Default | Description                                                                                                                                                |
| -------------------------- | ------ | ------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `WEAVE_ENABLE_WAL`         | `bool` | `false` | Enables the write-ahead log. When set to `true`, Weave writes the API requests to disk locally before sending them to the server.                          |
| `WEAVE_DISABLE_WAL_SENDER` | `bool` | `false` | Disables the WAL sender. When set to `true`, Weave writes the requests to disk locally but doesn't flush them to the server. This is useful for debugging. |

For a list of all available environment variables for Weave, see the [environment variables](/weave/guides/core-types/env-vars) reference page.


# Quickstart: Track LLM inputs and outputs
Source: https://docs.wandb.ai/weave/quickstart

Get started with W&B Weave by adding tracing to your LLM application to debug and monitor model interactions.

<div>
  <ColabLink />

  <GitHubLink />
</div>

Learn how to track LLM calls with Weave by adding tracing to your code. This quickstart walks you through tracing a request to OpenAI and viewing the results in the Weave UI. It's for developers who are new to Weave and want to instrument an LLM application for debugging and monitoring.

## What you'll learn

This guide shows you how to:

* Import and configure Weave in your code.
* Use the `weave.op` decorator to track your code.
* View traces in the Weave UI.

## Prerequisites

* A [W\&B account](https://wandb.ai/signup)
* Python 3.10+ or Node.js 18+
* Required packages installed:
  * **Python**: `pip install weave openai`
  * **TypeScript**: `npm install weave openai`
* An [OpenAI API key](https://platform.openai.com/api-keys) set as an environment variable

## Log a trace to a new project

In this section, you instrument a small application so that Weave can record its inputs, outputs, and code. To begin tracking your code and logging traces to Weave:

1. Import the `weave` library into your code.
2. Call `weave.init('your_wb_team/project_name')` in your code to send tracking information to your W\&B [team](/platform/app/settings-page/teams) and [project](/platform/hosting/iam/org_team_struct#project). If you don't set a team, Weave sends the traces to your [default team](/platform/app/settings-page/user-settings/#default-team). If the specified project doesn't exist in your team, Weave creates it.
3. Add the [`@weave.op()` decorator](/weave/guides/tracking/ops) to functions you want to track. Although Weave automatically tracks calls to supported LLMs, the decorator lets you track the inputs, outputs, and code of specific functions. The decorator uses the following syntax in TypeScript: `weave.op(your_function)`.

The following example code sends a request to OpenAI (requires an [OpenAI API key](https://platform.openai.com/docs/quickstart)), and Weave records the request's tracing information. The request asks the OpenAI model to extract dinosaur names from the input and identify each dinosaur's diet (herbivore or carnivore).

The following example code tracks your first project with Weave:

<Tabs>
  <Tab title="Python">
    ```python lines {1-2,7-8,27-28} theme={null}
    # Imports the Weave library
    import weave
    from openai import OpenAI

    client = OpenAI()

    # Weave automatically tracks the inputs, outputs and code of this function
    @weave.op()
    def extract_dinos(sentence: str) -> dict:
        response = client.chat.completions.create(
            model="gpt-4o",
            messages=[
                {
                    "role": "system",
                    "content": """In JSON format extract a list of `dinosaurs`, with their `name`,
    their `common_name`, and whether its `diet` is a herbivore or carnivore"""
                },
                {
                    "role": "user",
                    "content": sentence
                }
                ],
                response_format={ "type": "json_object" }
            )
        return response.choices[0].message.content

    # Initializes Weave, and sets the team and project to log data to
    weave.init('your-team/traces-quickstart')

    sentence = """I watched as a Tyrannosaurus rex (T. rex) chased after a Triceratops (Trike), \
    both carnivore and herbivore locked in an ancient dance. Meanwhile, a gentle giant \
    Brachiosaurus (Brachi) calmly munched on treetops, blissfully unaware of the chaos below."""

    result = extract_dinos(sentence)
    print(result)
    ```
  </Tab>

  <Tab title="TypeScript">
    ```typescript lines {1-2,7-8,20,24-25} theme={null}
    // Imports the Weave library
    import * as weave from 'weave';
    import OpenAI from 'openai';

    const openai = new OpenAI();

    // Weave automatically tracks the inputs, outputs and code of this function  
    async function extractDinos(input: string) {
      const response = await openai.chat.completions.create({
        model: 'gpt-4o',
        messages: [
          {
            role: 'user',
            content: `In JSON format extract a list of 'dinosaurs', with their 'name', their 'common_name', and whether its 'diet' is a herbivore or carnivore: ${input}`,
          },
        ],
      });
      return response.choices[0].message.content;
    }
    const extractDinosOp = weave.op(extractDinos);

    async function main() {

      // Initializes Weave, and sets the team and project to log data to
      await weave.init('your-team/traces-quickstart');

      const result = await extractDinosOp(
        'I watched as a Tyrannosaurus rex (T. rex) chased after a Triceratops (Trike), both carnivore and herbivore locked in an ancient dance. Meanwhile, a gentle giant Brachiosaurus (Brachi) calmly munched on treetops, blissfully unaware of the chaos below.'
      );
      console.log(result);
    }

    main();

    ```
  </Tab>
</Tabs>

When you call the `extract_dinos` function, Weave outputs links in the terminal to view your traces. The output looks like this:

```text theme={null}
weave:  $ pip install weave --upgrade
weave: Logged in as Weights & Biases user: example-username.
weave: View Weave data at https://wandb.ai/your-team/traces-quickstart/weave
weave: 🍩 https://wandb.ai/your-team/traces-quickstart/r/call/019ae171-7f32-7c96-8b42-931a32f900b7
{
  "dinosaurs": [
    {
      "name": "Tyrannosaurus rex",
      "common_name": "T. rex",
      "diet": "carnivore"
    },
    {
      "name": "Triceratops",
      "common_name": "Trike",
      "diet": "herbivore"
    },
    {
      "name": "Brachiosaurus",
      "common_name": "Brachi",
      "diet": "herbivore"
    }
  ]
}
```

## See traces of your application in your project

Now that your code has produced a trace, you can inspect it in the Weave UI. Click the link in your terminal or paste it into your browser to open the Weave UI. In the **Traces** panel of the Weave UI, click the trace to see its data, such as its input, output, latency, and token usage.

<img alt="Weave UI Traces panel showing the trace for the extract_dinos function, including its input, output, latency, and token usage." />

You have now logged and viewed your first Weave trace.

## Learn more about traces

* Learn how to [decorate your functions and retrieve call information](/weave/guides/tracking/tracing).
* Try the [Playground](/weave/guides/tools/playground) to test different models on logged traces.
* [Explore integrations](/weave/guides/integrations/). Weave automatically tracks calls made to OpenAI, Anthropic, and many more LLM libraries. If your LLM library isn't one of the Weave integrations, you can track calls to other LLM libraries or frameworks by wrapping them with `@weave.op()`.

## Next steps

[Get started evaluating your app](/weave/tutorial-eval) and then see how to [evaluate a RAG application](/weave/tutorial-rag).


# Learn Weave with Serverless Inference
Source: https://docs.wandb.ai/weave/quickstart-inference

Learn Weave fundamentals using Serverless Inference to trace model calls, compare outputs, and run evaluations.

<div>
  <ColabLink />

  <GitHubLink />
</div>

This guide shows you how to use W\&B Weave with [Serverless Inference](/inference) to learn Weave fundamentals. Use Serverless Inference to build and trace LLM applications using live open source models without setting up your own infrastructure or managing API keys from multiple providers. With your W\&B API key, you can interact with [all models hosted by Serverless Inference](/inference/models). By the end of this guide, you'll have traced LLM calls, compared models, and run an evaluation that you can review in the Weave UI.

## What you'll learn

This guide shows you how to:

* Set up Weave and Serverless Inference.
* Build a basic LLM application with automatic tracing.
* Compare multiple models.
* Evaluate model performance on a dataset.
* View your results in the Weave UI.

## Prerequisites

* A [W\&B account](https://wandb.ai/signup)
* Python 3.10+ or Node.js 18+
* Required packages installed:
  * **Python**: `pip install weave openai`
  * **TypeScript**: `npm install weave openai`
* An [OpenAI API key](https://platform.openai.com/api-keys) set as an environment variable

## Trace your first LLM call

This section shows you how to make a single LLM call and have Weave trace it automatically, so you can confirm your setup is working before moving on to more complex examples.

To begin, copy and paste the following code example. The code example uses Llama 3.1-8B from Serverless Inference.

When you run this code, Weave:

* Traces your LLM call automatically.
* Logs inputs, outputs, latency, and token usage.
* Provides a link to view your trace in the Weave UI.

<Tabs>
  <Tab title="Python">
    ```python lines theme={null}
    import weave
    import openai

    # Initialize Weave. Replace [YOUR-TEAM] with your team name.
    weave.init("[YOUR-TEAM]/inference-quickstart")

    # Create an OpenAI-compatible client pointing to Serverless Inference
    client = openai.OpenAI(
        base_url='https://api.inference.wandb.ai/v1',
        api_key="YOUR_WANDB_API_KEY",  # Replace with your actual API key
        project="[YOUR-TEAM]/my-first-weave-project",  # Required for usage tracking
    )

    # Decorate your function to enable tracing. Use the standard OpenAI client
    @weave.op()
    def ask_llama(question: str) -> str:
        response = client.chat.completions.create(
            model="meta-llama/Llama-3.1-8B-Instruct",
            messages=[
                {"role": "system", "content": "You are a helpful assistant."},
                {"role": "user", "content": question}
            ],
        )
        return response.choices[0].message.content

    # Call your function - Weave automatically traces everything
    result = ask_llama("What are the benefits of using W&B Weave for LLM development?")
    print(result)
    ```
  </Tab>

  <Tab title="TypeScript">
    ```typescript lines theme={null}
    import * as weave from 'weave';
    import OpenAI from 'openai';

    // Initialize Weave. Replace values enclosed in "[]" with your own.
    await weave.init("[YOUR-TEAM]/inference-quickstart")

    // Create an OpenAI-compatible client pointing to Serverless Inference
    const client = new OpenAI({
        baseURL: 'https://api.inference.wandb.ai/v1',  // Serverless Inference endpoint
        apiKey: process.env.WANDB_API_KEY || 'YOUR_WANDB_API_KEY', // Replace with your API key or set the WANDB_API_KEY environment variable
    });

    // Wrap your function with weave.op to enable tracing
    const askLlama = weave.op(async function askLlama(question: string): Promise<string> {
    const response = await client.chat.completions.create({
        model: 'meta-llama/Llama-3.1-70B-Instruct',
        messages: [
        { role: 'system', content: 'You are a helpful assistant.' },
        { role: 'user', content: question }
        ],
    });
    return response.choices[0].message.content || '';
    });

    // Call your function - Weave automatically traces everything
    const result = await askLlama('What are the benefits of using W&B Weave for LLM development?');
    console.log(result);
    ```
  </Tab>
</Tabs>

## Build a text summarization application

Now that you've traced a single LLM call, this section shows how Weave traces nested operations across multiple functions, so you can see how a real multi-step LLM application is captured in the UI.

Next, run this code, which is a basic summarization app that shows how Weave traces nested operations:

<Tabs>
  <Tab title="Python">
    ```python lines theme={null}
    import weave
    import openai

    # Initialize Weave. Replace values enclosed in "[]" with your own.
    weave.init("[YOUR-TEAM]/inference-quickstart")

    client = openai.OpenAI(
        base_url='https://api.inference.wandb.ai/v1',
        api_key="YOUR_WANDB_API_KEY",  # Replace with your actual API key
        project="[YOUR-TEAM]/my-first-weave-project",  # Required for usage tracking
    )

    @weave.op()
    def extract_key_points(text: str) -> list[str]:
        """Extract key points from a text."""
        response = client.chat.completions.create(
            model="meta-llama/Llama-3.1-8B-Instruct",
            messages=[
                {"role": "system", "content": "Extract 3-5 key points from the text. Return each point on a new line."},
                {"role": "user", "content": text}
            ],
        )
        # Returns response without blank lines
        return [line for line in response.choices[0].message.content.strip().splitlines() if line.strip()]

    @weave.op()
    def create_summary(key_points: list[str]) -> str:
        """Create a concise summary based on key points."""
        points_text = "\n".join(f"- {point}" for point in key_points)
        response = client.chat.completions.create(
            model="meta-llama/Llama-3.1-8B-Instruct",
            messages=[
                {"role": "system", "content": "Create a one-sentence summary based on these key points."},
                {"role": "user", "content": f"Key points:\n{points_text}"}
            ],
        )
        return response.choices[0].message.content

    @weave.op()
    def summarize_text(text: str) -> dict:
        """Main summarization pipeline."""
        key_points = extract_key_points(text)
        summary = create_summary(key_points)
        return {
            "key_points": key_points,
            "summary": summary
        }

    # Try it with sample text
    sample_text = """
    The Apollo 11 mission was a historic spaceflight that landed the first humans on the Moon 
    on July 20, 1969. Commander Neil Armstrong and lunar module pilot Buzz Aldrin descended 
    to the lunar surface while Michael Collins remained in orbit. Armstrong became the first 
    person to step onto the Moon, followed by Aldrin 19 minutes later. They spent about 
    two and a quarter hours together outside the spacecraft, collecting samples and taking photographs.
    """

    result = summarize_text(sample_text)
    print("Key Points:", result["key_points"])
    print("\nSummary:", result["summary"])
    ```
  </Tab>

  <Tab title="TypeScript">
    ```typescript lines theme={null}
    import * as weave from 'weave';
    import OpenAI from 'openai';

    // Initialize Weave. Replace with your-team/your-project
    await weave.init('[YOUR-TEAM]/inference-quickstart');

    const client = new OpenAI({
    baseURL: 'https://api.inference.wandb.ai/v1',
    apiKey: process.env.WANDB_API_KEY || 'YOUR_WANDB_API_KEY',  // Replace with your API key or set the WANDB_API_KEY environment variable
    });

    const extractKeyPoints = weave.op(async function extractKeyPoints(text: string): Promise<string[]> {
    const response = await client.chat.completions.create({
        model: 'meta-llama/Llama-3.1-8B-Instruct',
        messages: [
        { role: 'system', content: 'Extract 3-5 key points from the text. Return each point on a new line.' },
        { role: 'user', content: text }
        ],
    });
    // Returns response without blank lines
    const content = response.choices[0].message.content || '';
    return content.split('\n').map(line => line.trim()).filter(line => line.length > 0);
    });

    const createSummary = weave.op(async function createSummary(keyPoints: string[]): Promise<string> {
    const pointsText = keyPoints.map(point => `- ${point}`).join('\n');
    const response = await client.chat.completions.create({
        model: 'meta-llama/Llama-3.1-8B-Instruct',
        messages: [
        { role: 'system', content: 'Create a one-sentence summary based on these key points.' },
        { role: 'user', content: `Key points:\n${pointsText}` }
        ],
    });
    return response.choices[0].message.content || '';
    });

    const summarizeText = weave.op(async function summarizeText(text: string): Promise<{key_points: string[], summary: string}> {
    const keyPoints = await extractKeyPoints(text);
    const summary = await createSummary(keyPoints);
    return {
        key_points: keyPoints,
        summary: summary
    };
    });

    // Try it with sample text
    const sampleText = `
    The Apollo 11 mission was a historic spaceflight that landed the first humans on the Moon 
    on July 20, 1969. Commander Neil Armstrong and lunar module pilot Buzz Aldrin descended 
    to the lunar surface while Michael Collins remained in orbit. Armstrong became the first 
    person to step onto the Moon, followed by Aldrin 19 minutes later. They spent about 
    two and a quarter hours together outside the spacecraft, collecting samples and taking photographs.
    `;

    const result = await summarizeText(sampleText);
    console.log('Key Points:', result.key_points);
    console.log('\nSummary:', result.summary);
    ```
  </Tab>
</Tabs>

## Compare multiple models

A common use case for Weave is comparing how different models respond to the same prompt. Serverless Inference provides access to multiple models. Use the following code to compare the performance of Llama and DeepSeek's respective responses:

<Tabs>
  <Tab title="Python">
    ```python lines theme={null}
    import weave
    import openai

    # Initialize Weave. Replace with your-team/your-project
    weave.init("[YOUR-TEAM]/inference-quickstart")

    client = openai.OpenAI(
        base_url='https://api.inference.wandb.ai/v1',
        api_key="YOUR_WANDB_API_KEY",  # Replace with your actual API key
        project="[YOUR-TEAM]/my-first-weave-project",  # Required for usage tracking
    )

    # Define a Model class to compare different LLMs
    class InferenceModel(weave.Model):
        model_name: str
        
        @weave.op()
        def predict(self, question: str) -> str:
            response = client.chat.completions.create(
                model=self.model_name,
                messages=[
                    {"role": "user", "content": question}
                ],
            )
            return response.choices[0].message.content

    # Create instances for different models
    llama_model = InferenceModel(model_name="meta-llama/Llama-3.1-8B-Instruct")
    deepseek_model = InferenceModel(model_name="deepseek-ai/DeepSeek-V3.1")

    # Compare their responses
    test_question = "Explain quantum computing in one paragraph for a high school student."

    print("Llama 3.1 8B response:")
    print(llama_model.predict(test_question))
    print("\n" + "="*50 + "\n")
    print("DeepSeek V3 response:")
    print(deepseek_model.predict(test_question))
    ```
  </Tab>

  <Tab title="TypeScript">
    ```typescript lines theme={null}
    import * as weave from 'weave';
    import OpenAI from 'openai';

    // Initialize Weave. Replace with your-team/your-project
    await weave.init("[YOUR-TEAM]/inference-quickstart")

    const client = new OpenAI({
      baseURL: 'https://api.inference.wandb.ai/v1',
      apiKey: process.env.WANDB_API_KEY || 'YOUR_WANDB_API_KEY', // Replace with your API key or set the WANDB_API_KEY environment variable
    });

    // Create model functions using weave.op (weave.Model is not supported in TypeScript)
    function createModel(modelName: string) {
      return weave.op(async function predict(question: string): Promise<string> {
        const response = await client.chat.completions.create({
          model: modelName,
          messages: [
            { role: 'user', content: question }
          ],
        });
        return response.choices[0].message.content || '';
      });
    }

    // Create instances for different models
    const llamaModel = createModel('meta-llama/Llama-3.1-8B-Instruct');
    const deepseekModel = createModel('deepseek-ai/DeepSeek-V3.1');

    // Compare their responses
    const testQuestion = 'Explain quantum computing in one paragraph for a high school student.';

    console.log('Llama 3.1 8B response:');
    console.log(await llamaModel(testQuestion));
    console.log('\n' + '='.repeat(50) + '\n');
    console.log('DeepSeek V3 response:');
    console.log(await deepseekModel(testQuestion));
    ```
  </Tab>
</Tabs>

## Evaluate model performance

Going beyond ad-hoc comparisons, this section shows you how to run a structured evaluation across a dataset so you can systematically measure and compare model quality.

Evaluate how well a model performs on a Q\&A task using Weave's built-in `EvaluationLogger`. This provides structured evaluation tracking with automatic aggregation, token usage capture, and rich comparison features in the UI.

Append the following code to the script you used in the prior section:

<Tabs>
  <Tab title="Python">
    ```python lines theme={null}
    from typing import Optional
    from weave import EvaluationLogger

    # Create a simple dataset
    dataset = [
        {"question": "What is 2 + 2?", "expected": "4"},
        {"question": "What is the capital of France?", "expected": "Paris"},
        {"question": "Name a primary color", "expected_one_of": ["red", "blue", "yellow"]},
    ]

    # Define a scorer
    @weave.op()
    def accuracy_scorer(expected: str, output: str, expected_one_of: Optional[list[str]] = None) -> dict:
        """Score the accuracy of the model output."""
        output_clean = output.strip().lower()
        
        if expected_one_of:
            is_correct = any(option.lower() in output_clean for option in expected_one_of)
        else:
            is_correct = expected.lower() in output_clean
        
        return {"correct": is_correct, "score": 1.0 if is_correct else 0.0}

    # Evaluate a model using Weave's EvaluationLogger
    def evaluate_model(model: InferenceModel, dataset: list[dict]):
        """Run evaluation on a dataset using Weave's built-in evaluation framework."""
        # Initialize EvaluationLogger BEFORE calling the model to capture token usage
        # This is especially important for Serverless Inference to track costs
        # Convert model name to a valid format (replace non-alphanumeric chars with underscores)
        safe_model_name = model.model_name.replace("/", "_").replace("-", "_").replace(".", "_")
        eval_logger = EvaluationLogger(
            model=safe_model_name,
            dataset="qa_dataset"
        )
        
        for example in dataset:
            # Get model prediction
            output = model.predict(example["question"])
            
            # Log the prediction
            pred_logger = eval_logger.log_prediction(
                inputs={"question": example["question"]},
                output=output
            )
            
            # Score the output
            score = accuracy_scorer(
                expected=example.get("expected", ""),
                output=output,
                expected_one_of=example.get("expected_one_of")
            )
            
            # Log the score
            pred_logger.log_score(
                scorer="accuracy",
                score=score["score"]
            )
            
            # Finish logging for this prediction
            pred_logger.finish()
        
        # Log summary - Weave automatically aggregates the accuracy scores
        eval_logger.log_summary()
        print(f"Evaluation complete for {model.model_name} (logged as: {safe_model_name}). View results in the Weave UI.")

    # Compare multiple models - a key feature of Weave's evaluation framework
    models_to_compare = [
        llama_model,
        deepseek_model,
    ]

    for model in models_to_compare:
        evaluate_model(model, dataset)

    # In the Weave UI, navigate to the Evals tab to compare results across models
    ```
  </Tab>

  <Tab title="TypeScript">
    ```typescript lines theme={null}
    import { EvaluationLogger } from 'weave';

    // Create a simple dataset
    interface DatasetExample {
      question: string;
      expected?: string;
      expected_one_of?: string[];
    }

    const dataset: DatasetExample[] = [
      { question: 'What is 2 + 2?', expected: '4' },
      { question: 'What is the capital of France?', expected: 'Paris' },
      { question: 'Name a primary color', expected_one_of: ['red', 'blue', 'yellow'] },
    ];

    // Define a scorer
    const accuracyScorer = weave.op(function accuracyScorer(args: {
      expected: string;
      output: string;
      expected_one_of?: string[];
    }): { correct: boolean; score: number } {
      const outputClean = args.output.trim().toLowerCase();
      
      let isCorrect: boolean;
      if (args.expected_one_of) {
        isCorrect = args.expected_one_of.some(option => 
          outputClean.includes(option.toLowerCase())
        );
      } else {
        isCorrect = outputClean.includes(args.expected.toLowerCase());
      }
      
      return { correct: isCorrect, score: isCorrect ? 1.0 : 0.0 };
    });

    // Evaluate a model using Weave's EvaluationLogger
    async function evaluateModel(
      model: (question: string) => Promise<string>,
      modelName: string,
      dataset: DatasetExample[]
    ): Promise<void> {
      // Initialize EvaluationLogger BEFORE calling the model to capture token usage
      // This is especially important for Serverless Inference to track costs
      // Convert model name to a valid format (replace non-alphanumeric chars with underscores)
      const safeModelName = modelName.replace(/\//g, '_').replace(/-/g, '_').replace(/\./g, '_');
      const evalLogger = new EvaluationLogger({
        name: 'inference_evaluation',
        model: { name: safeModelName },
        dataset: 'qa_dataset'
      });
      
      for (const example of dataset) {
        // Get model prediction
        const output = await model(example.question);
        
        // Log the prediction
        const predLogger = evalLogger.logPrediction(
          { question: example.question },
          output
        );
        
        // Score the output
        const score = await accuracyScorer({
          expected: example.expected || '',
          output: output,
          expected_one_of: example.expected_one_of
        });
        
        // Log the score
        predLogger.logScore('accuracy', score.score);
        
        // Finish logging for this prediction
        predLogger.finish();
      }
      
      // Log summary - Weave automatically aggregates the accuracy scores
      await evalLogger.logSummary();
      console.log(`Evaluation complete for ${modelName} (logged as: ${safeModelName}). View results in the Weave UI.`);
    }

    // Compare multiple models - a key feature of Weave's evaluation framework
    const modelsToCompare = [
      { model: llamaModel, name: 'meta-llama/Llama-3.1-8B-Instruct' },
      { model: deepseekModel, name: 'deepseek-ai/DeepSeek-V3.1' },
    ];

    for (const { model, name } of modelsToCompare) {
      await evaluateModel(model, name, dataset);
    }

    // In the Weave UI, navigate to the Evals tab to compare results across models
    ```
  </Tab>
</Tabs>

After you run these examples, you have traced LLM calls, a nested summarization pipeline, a model comparison, and a full evaluation logged to Weave. Running these examples returns links to the traces in the terminal. Click any link to view traces in the Weave UI.

In the Weave UI, you can:

* Review a timeline of all your LLM calls.
* Inspect inputs and outputs for each operation.
* View token usage and estimated costs (automatically captured by EvaluationLogger).
* Analyze latency and performance metrics.
* Navigate to the **Evals** tab to see aggregated evaluation results.
* Use the **Compare** feature to analyze performance across different models.
* Page through specific examples to see how different models performed on the same inputs.

## Available models

For a complete list of available models, see the [Available Models section](/inference/models) in the Serverless Inference documentation.

## Next steps

With the basics in place, the following resources help you go deeper with Weave and Serverless Inference:

* **Use the Playground**: [Try models interactively](/weave/guides/tools/playground#access-the-playground) in the Weave Playground
* **Build evaluations**: Learn about [systematic evaluation](/weave/guides/core-types/evaluations) of your LLM applications
* **Try other integrations**: Weave works with [OpenAI, Anthropic, and many more](/weave/guides/integrations)

## Troubleshooting

<details>
  <summary>Authentication errors</summary>

  If you get authentication errors:

  1. Verify you have a valid W\&B account.
  2. Check that you're using the correct API key from [wandb.ai/settings](https://wandb.ai/settings).
  3. Ensure your project name follows the format `your-team/your-project`.
</details>

<details>
  <summary>Rate limit errors</summary>

  Serverless Inference has concurrency limits per project. If you hit rate limits:

  * Reduce the number of concurrent requests.
  * Add delays between calls.
  * Consider upgrading your plan for higher limits.

  For more details, see the [limits documentation for Serverless Inference](/inference/usage-limits).
</details>

<details>
  <summary>Running out of credits</summary>

  The free tier includes limited credits. See the [usage and limits documentation](https://docs.wandb.ai/inference/usage-limits/) for details.
</details>


# Build an evaluation
Source: https://docs.wandb.ai/weave/tutorial-eval

Learn how to build an evaluation pipeline with Weave Models and Evaluations

<div>
  <ColabLink />

  <GitHubLink />
</div>

This tutorial walks you through building an end-to-end evaluation pipeline in Weave so you can measure and track the quality of an LLM application as you iterate on it. Evaluations help you compare changes against a consistent set of examples and detect regressions before they reach users. This tutorial targets developers who build LLM-powered applications and want a repeatable way to test them.

Weave provides built-in support for tracking evaluations with [`Model`](/weave/guides/core-types/models) and [`Evaluation`](/weave/guides/core-types/evaluations) classes. The APIs make minimal assumptions, so they fit a range of use cases.

<img alt="Evals hero" />

## What you'll learn

This guide shows you how to:

* Set up a `Model`.
* Create a dataset to test an LLM's responses against.
* Define a scoring function to compare model output to expected outputs.
* Run an evaluation that tests the model against the dataset using the scoring function and an additional built-in scorer.
* View the results of the evaluation in the Weave UI.

By the end, you'll have a working evaluation pipeline that scores an example model against a dataset and logs the results to Weave.

## Prerequisites

* A [W\&B account](https://wandb.ai/signup)
* Python 3.10+ or Node.js 18+
* Required packages installed:
  * **Python**: `pip install weave openai`
  * **TypeScript**: `npm install weave openai`
* An [OpenAI API key](https://platform.openai.com/api-keys) set as an environment variable

## Import the necessary libraries and functions

Import the following libraries into your script:

<Tabs>
  <Tab title="Python">
    ```python lines theme={null}
    import json
    import openai
    import asyncio
    import weave
    from weave.scorers import MultiTaskBinaryClassificationF1
    ```
  </Tab>

  <Tab title="TypeScript">
    ```typescript lines theme={null}
    import * as weave from 'weave';
    import OpenAI from 'openai';
    ```
  </Tab>
</Tabs>

## Build a `Model`

With the libraries in place, the next step is to define the model you want to evaluate.

In Weave, [`Models` are objects](/weave/guides/core-types/models) that capture both the behavior of your model or agent (logic, prompt, parameters) and its versioned metadata (parameters, code, micro-config) so you can track, compare, evaluate, and iterate reliably.

When you instantiate a `Model`, Weave automatically captures its configuration and behaviors and updates the version when changes occur. This lets you track its performance over time as you iterate on it.

To declare a `Model`, subclass `Model` and implement a `predict` function definition that takes one example and returns the response.

The following example model uses OpenAI to extract the names, colors, and flavors of alien fruits from input sentences.

<Tabs>
  <Tab title="Python">
    ```python lines {1,5} theme={null}
    class ExtractFruitsModel(weave.Model):
        model_name: str
        prompt_template: str

        @weave.op()
        async def predict(self, sentence: str) -> dict:
            client = openai.AsyncClient()

            response = await client.chat.completions.create(
                model=self.model_name,
                messages=[
                    {"role": "user", "content": self.prompt_template.format(sentence=sentence)}
                ],
            )
            result = response.choices[0].message.content
            if result is None:
                raise ValueError("No response from model")
            parsed = json.loads(result)
            return parsed
    ```
  </Tab>

  <Tab title="TypeScript">
    ```typescript lines {9} theme={null}
    // Note: weave.Model is not supported in TypeScript yet.
    // Instead, wrap your model-like function with weave.op

    import * as weave from 'weave';
    import OpenAI from 'openai';

    const openaiClient = new OpenAI();

    const model = weave.op(async function myModel({datasetRow}) {
      const prompt = `Extract fields ("fruit": <str>, "color": <str>, "flavor") from the following text, as json: ${datasetRow.sentence}`;
      const response = await openaiClient.chat.completions.create({
        model: 'gpt-3.5-turbo',
        messages: [{ role: 'user', content: prompt }],
        response_format: { type: 'json_object' }
      });
      return JSON.parse(response.choices[0].message.content);
    });
    ```
  </Tab>
</Tabs>

The `ExtractFruitsModel` class inherits from (or subclasses) `weave.Model` so Weave can track the instantiated object. `@weave.op` decorates the `predict` function to track its inputs and outputs.

You can instantiate `Model` objects like this:

<Tabs>
  <Tab title="Python">
    ```python lines theme={null}
    # Set your team and project name
    weave.init('[YOUR-TEAM]/eval_pipeline_quickstart')

    model = ExtractFruitsModel(
        model_name='gpt-3.5-turbo-1106',
        prompt_template='Extract fields ("fruit": <str>, "color": <str>, "flavor": <str>) from the following text, as json: {sentence}'
    )

    sentence = "There are many fruits that were found on the recently discovered planet Goocrux. There are neoskizzles that grow there, which are purple and taste like candy."

    print(asyncio.run(model.predict(sentence)))
    # if you're in a Jupyter Notebook, run:
    # await model.predict(sentence)
    ```
  </Tab>

  <Tab title="TypeScript">
    ```typescript theme={null}
    await weave.init('eval_pipeline_quickstart');

    const sentence = "There are many fruits that were found on the recently discovered planet Goocrux. There are neoskizzles that grow there, which are purple and taste like candy.";

    const result = await model({ datasetRow: { sentence } });

    console.log(result);
    ```
  </Tab>
</Tabs>

## Create a dataset

With a `Model` defined, you now need a dataset to evaluate it against. A [`Dataset`](/weave/guides/core-types/datasets) is a collection of examples stored as a Weave object. Publishing the dataset to Weave versions it and makes it reusable across evaluation runs.

The following example dataset defines three example input sentences and their correct answers (`labels`), and then formats them in a JSON table format that scoring functions can read.

This example builds a list of examples in code, but you can also log them one at a time from your running application.

<Tabs>
  <Tab title="Python">
    ```python lines theme={null}
    sentences = ["There are many fruits that were found on the recently discovered planet Goocrux. There are neoskizzles that grow there, which are purple and taste like candy.",
    "Pounits are a bright green color and are more savory than sweet.",
    "Finally, there are fruits called glowls, which have a very sour and bitter taste which is acidic and caustic, and a pale orange tinge to them."]
    labels = [
        {'fruit': 'neoskizzles', 'color': 'purple', 'flavor': 'candy'},
        {'fruit': 'pounits', 'color': 'bright green', 'flavor': 'savory'},
        {'fruit': 'glowls', 'color': 'pale orange', 'flavor': 'sour and bitter'}
    ]
    examples = [
        {'id': '0', 'sentence': sentences[0], 'target': labels[0]},
        {'id': '1', 'sentence': sentences[1], 'target': labels[1]},
        {'id': '2', 'sentence': sentences[2], 'target': labels[2]}
    ]
    ```
  </Tab>

  <Tab title="TypeScript">
    ```typescript theme={null}
    const sentences = [
      "There are many fruits that were found on the recently discovered planet Goocrux. There are neoskizzles that grow there, which are purple and taste like candy.",
      "Pounits are a bright green color and are more savory than sweet.",
      "Finally, there are fruits called glowls, which have a very sour and bitter taste which is acidic and caustic, and a pale orange tinge to them."
    ];
    const labels = [
      { fruit: 'neoskizzles', color: 'purple', flavor: 'candy' },
      { fruit: 'pounits', color: 'bright green', flavor: 'savory' },
      { fruit: 'glowls', color: 'pale orange', flavor: 'sour and bitter' }
    ];
    const examples = sentences.map((sentence, i) => ({
      id: i.toString(),
      sentence,
      target: labels[i]
    }));
    ```
  </Tab>
</Tabs>

Then create your dataset using the `weave.Dataset()` class and publish it:

<Tabs>
  <Tab title="Python">
    ```python lines {2} theme={null}
    weave.init('eval_pipeline_quickstart')
    dataset = weave.Dataset(name='fruits', rows=examples)
    weave.publish(dataset)
    ```
  </Tab>

  <Tab title="TypeScript">
    ```typescript lines {3-6} theme={null}
    import * as weave from 'weave';
    await weave.init('eval_pipeline_quickstart');
    const dataset = new weave.Dataset({
      name: 'fruits',
      rows: examples
    });
    await dataset.save();
    ```
  </Tab>
</Tabs>

## Define custom scoring functions

Now that you have a model and a dataset, you need a way to measure how well the model performs on each example. Scoring functions compare the model's output to the expected target and produce the metrics that an evaluation reports.

When you use Weave evaluations, Weave expects a `target` to compare `output` against. The following scoring function takes two dictionaries (`target` and `output`) and returns a dictionary of boolean values that indicate whether the output matches the target. The `@weave.op()` decorator enables Weave to track the scoring function's execution.

<Tabs>
  <Tab title="Python">
    ```python lines theme={null}
    @weave.op()
    def fruit_name_score(target: dict, output: dict) -> dict:
        return {'correct': target['fruit'] == output['fruit']}
    ```
  </Tab>

  <Tab title="TypeScript">
    ```typescript theme={null}
    import * as weave from 'weave';

    const fruitNameScorer = weave.op(
      function fruitNameScore({target, output}) {
        return { correct: target.fruit === output.fruit };
      }
    );
    ```
  </Tab>
</Tabs>

For more information on making your own scoring function, see the [Scorers](/weave/guides/evaluation/scorers) guide.

In some applications, you may want to create custom `Scorer` classes. For example, you might create a standardized `LLMJudge` class with specific parameters (such as chat model or prompt), specific row scoring, and aggregate score calculation. For more information, see the tutorial on defining a `Scorer` class in [Model-based evaluation of RAG applications](/weave/tutorial-rag#optional-defining-a-scorer-class).

## Use a built-in scorer and run the evaluation

With the model, dataset, and a custom scorer in place, you're ready to wire everything together into an evaluation run.

Along with custom scoring functions, you can also use [Weave's built-in scorers](/weave/guides/evaluation/builtin_scorers). In the following evaluation, `weave.Evaluation()` uses the `fruit_name_score` function defined in the previous section and the built-in `MultiTaskBinaryClassificationF1` scorer, which computes [F1 scores](https://en.wikipedia.org/wiki/F-score).

The following example runs an evaluation of `ExtractFruitsModel` on the `fruits` dataset using the two scoring functions and logs the results to Weave.

<Tabs>
  <Tab title="Python">
    ```python lines {3-10} theme={null}
    weave.init('eval_pipeline_quickstart')

    evaluation = weave.Evaluation(
        name='fruit_eval',
        dataset=dataset, 
        scorers=[
            MultiTaskBinaryClassificationF1(class_names=["fruit", "color", "flavor"]), 
            fruit_name_score
        ],
    )
    print(asyncio.run(evaluation.evaluate(model)))
    # if you're in a Jupyter Notebook, run:
    # await evaluation.evaluate(model)
    ```
  </Tab>

  <Tab title="TypeScript">
    ```typescript lines {5-9} theme={null}
    import * as weave from 'weave';

    await weave.init('eval_pipeline_quickstart');

    const evaluation = new weave.Evaluation({
      name: 'fruit_eval',
      dataset: dataset,
      scorers: [fruitNameScorer],
    });
    const results = await evaluation.evaluate(model);
    console.log(results);
    ```
  </Tab>
</Tabs>

<Note>
  If you're running from a Python script, you'll need to use `asyncio.run`. However, if you're running from a Jupyter notebook, you can use `await` directly.
</Note>

### Complete example

<Accordion title="Complete evaluation pipeline in one script:">
  <Tabs>
    <Tab title="Python">
      ```python lines theme={null}
      import json
      import asyncio
      import openai
      import weave
      from weave.scorers import MultiTaskBinaryClassificationF1

      # Initialize Weave once
      weave.init('eval_pipeline_quickstart')

      # 1. Define Model
      class ExtractFruitsModel(weave.Model):
          model_name: str
          prompt_template: str

          @weave.op()
          async def predict(self, sentence: str) -> dict:
              client = openai.AsyncClient()
              response = await client.chat.completions.create(
                  model=self.model_name,
                  messages=[{"role": "user", "content": self.prompt_template.format(sentence=sentence)}],
              )
              result = response.choices[0].message.content
              if result is None:
                  raise ValueError("No response from model")
              return json.loads(result)

      # 2. Instantiate model
      model = ExtractFruitsModel(
          model_name='gpt-3.5-turbo-1106',
          prompt_template='Extract fields ("fruit": <str>, "color": <str>, "flavor": <str>) from the following text, as json: {sentence}'
      )

      # 3. Create dataset
      sentences = ["There are many fruits that were found on the recently discovered planet Goocrux. There are neoskizzles that grow there, which are purple and taste like candy.",
      "Pounits are a bright green color and are more savory than sweet.",
      "Finally, there are fruits called glowls, which have a very sour and bitter taste which is acidic and caustic, and a pale orange tinge to them."]
      labels = [
          {'fruit': 'neoskizzles', 'color': 'purple', 'flavor': 'candy'},
          {'fruit': 'pounits', 'color': 'bright green', 'flavor': 'savory'},
          {'fruit': 'glowls', 'color': 'pale orange', 'flavor': 'sour and bitter'}
      ]
      examples = [
          {'id': '0', 'sentence': sentences[0], 'target': labels[0]},
          {'id': '1', 'sentence': sentences[1], 'target': labels[1]},
          {'id': '2', 'sentence': sentences[2], 'target': labels[2]}
      ]

      dataset = weave.Dataset(name='fruits', rows=examples)
      weave.publish(dataset)

      # 4. Define scoring function
      @weave.op()
      def fruit_name_score(target: dict, output: dict) -> dict:
          return {'correct': target['fruit'] == output['fruit']}

      # 5. Run evaluation
      evaluation = weave.Evaluation(
          name='fruit_eval',
          dataset=dataset,
          scorers=[
              MultiTaskBinaryClassificationF1(class_names=["fruit", "color", "flavor"]),
              fruit_name_score
          ],
      )
      print(asyncio.run(evaluation.evaluate(model)))
      ```
    </Tab>

    <Tab title="TypeScript">
      ```typescript lines theme={null}
      import * as weave from 'weave';
      import OpenAI from 'openai';

      // Initialize Weave once
      await weave.init('eval_pipeline_quickstart');

      // 1. Define Model
      // Note: weave.Model is not supported in TypeScript yet.
      // Instead, wrap your model-like function with weave.op
      const openaiClient = new OpenAI();

      const model = weave.op(async function myModel({datasetRow}) {
        const prompt = `Extract fields ("fruit": <str>, "color": <str>, "flavor": <str>) from the following text, as json: ${datasetRow.sentence}`;
        const response = await openaiClient.chat.completions.create({
          model: 'gpt-3.5-turbo',
          messages: [{ role: 'user', content: prompt }],
          response_format: { type: 'json_object' }
        });
        return JSON.parse(response.choices[0].message.content);
      });

      // 2. Create dataset
      const sentences = [
        "There are many fruits that were found on the recently discovered planet Goocrux. There are neoskizzles that grow there, which are purple and taste like candy.",
        "Pounits are a bright green color and are more savory than sweet.",
        "Finally, there are fruits called glowls, which have a very sour and bitter taste which is acidic and caustic, and a pale orange tinge to them."
      ];
      const labels = [
        { fruit: 'neoskizzles', color: 'purple', flavor: 'candy' },
        { fruit: 'pounits', color: 'bright green', flavor: 'savory' },
        { fruit: 'glowls', color: 'pale orange', flavor: 'sour and bitter' }
      ];
      const examples = sentences.map((sentence, i) => ({
        id: i.toString(),
        sentence,
        target: labels[i]
      }));

      const dataset = new weave.Dataset({
        name: 'fruits',
        rows: examples
      });
      await dataset.save();

      // 3. Define scoring function
      const fruitNameScorer = weave.op(
        function fruitNameScore({target, output}) {
          return { correct: target.fruit === output.fruit };
        }
      );

      // 4. Run evaluation
      const evaluation = new weave.Evaluation({
        name: 'fruit_eval',
        dataset: dataset,
        scorers: [fruitNameScorer],
      });
      const results = await evaluation.evaluate(model);
      console.log(results);
      ```
    </Tab>
  </Tabs>
</Accordion>

## View your evaluation results

After the evaluation completes, you can inspect each prediction and scorer result in the Weave UI. Weave automatically captures traces of each prediction and score. To view the results, click the link that the evaluation prints.

<img alt="Evaluation results" />

## Learn more about Weave evaluations

You now have a complete evaluation pipeline. To go deeper into Weave's evaluation features, see the following resources:

* Learn more about how to [build and use scorers](/weave/guides/evaluation/scorers).
* Check out Weave's [built-in scoring functions](/weave/guides/evaluation/builtin_scorers).
* Learn about [Model-Based Evaluation](/weave/guides/evaluation/scorers#model-based-evaluation) for using LLMs as judges.

## Next steps

[Build a RAG application](/weave/tutorial-rag) to learn about evaluating retrieval-augmented generation.


# Evaluate RAG applications
Source: https://docs.wandb.ai/weave/tutorial-rag

Build and evaluate RAG applications using Weave with LLM judges

<div>
  <ColabLink />

  <GitHubLink />
</div>

Retrieval Augmented Generation (RAG) is a common way to build Generative AI applications that have access to custom knowledge bases. This tutorial walks you through building a RAG application and using Weave to track its retrieval steps and evaluate its responses with an LLM judge, so you can measure and improve the quality of the answers your application returns.

This guide is for developers building RAG applications who want to add observability and systematic evaluation to their pipelines.

<img alt="Evals hero" />

## What you'll learn

This guide shows you how to:

* Build a knowledge base.
* Create a RAG application with a retrieval step that finds relevant documents.
* Track retrieval steps with Weave.
* Evaluate RAG applications using an LLM judge to measure context precision.
* Define custom scoring functions.

## Prerequisites

* A [W\&B account](https://wandb.ai/signup)
* Python 3.10+ or Node.js 18+
* Required packages installed:
  * **Python**: `pip install weave openai`
  * **TypeScript**: `npm install weave openai`
* An [OpenAI API key](https://platform.openai.com/api-keys) set as an environment variable

## Build a knowledge base

The knowledge base is the corpus your RAG application searches at query time. In this section, you compute vector embeddings for a small set of articles so that you can later retrieve the most relevant article for a given question.

First, compute the embeddings for the articles. You typically do this once with your articles and put the embeddings and metadata in a database, but here it's done every time the script runs for simplicity.

<Tabs>
  <Tab title="Python">
    ```python lines theme={null}
    from openai import OpenAI
    import weave
    from weave import Model
    import numpy as np
    import json
    import asyncio

    articles = [
        "Novo Nordisk and Eli Lilly rival soars 32 percent after promising weight loss drug results Shares of Denmarks Zealand Pharma shot 32 percent higher in morning trade, after results showed success in its liver disease treatment survodutide, which is also on trial as a drug to treat obesity. The trial “tells us that the 6mg dose is safe, which is the top dose used in the ongoing [Phase 3] obesity trial too,” one analyst said in a note. The results come amid feverish investor interest in drugs that can be used for weight loss.",
        "Berkshire shares jump after big profit gain as Buffetts conglomerate nears $1 trillion valuation Berkshire Hathaway shares rose on Monday after Warren Buffetts conglomerate posted strong earnings for the fourth quarter over the weekend. Berkshires Class A and B shares jumped more than 1.5%, each. Class A shares are higher by more than 17% this year, while Class B has gained more than 18%. Berkshire was last valued at $930.1 billion, up from $905.5 billion where it closed on Friday, according to FactSet. Berkshire on Saturday posted fourth-quarter operating earnings of $8.481 billion, about 28 percent higher than the $6.625 billion from the year-ago period, driven by big gains in its insurance business. Operating earnings refers to profits from businesses across insurance, railroads and utilities. Meanwhile, Berkshires cash levels also swelled to record levels. The conglomerate held $167.6 billion in cash in the fourth quarter, surpassing the $157.2 billion record the conglomerate held in the prior quarter.",
        "Highmark Health says its combining tech from Google and Epic to give doctors easier access to information Highmark Health announced it is integrating technology from Google Cloud and the health-care software company Epic Systems. The integration aims to make it easier for both payers and providers to access key information they need, even if its stored across multiple points and formats, the company said. Highmark is the parent company of a health plan with 7 million members, a provider network of 14 hospitals and other entities",
        "Rivian and Lucid shares plunge after weak EV earnings reports Shares of electric vehicle makers Rivian and Lucid fell Thursday after the companies reported stagnant production in their fourth-quarter earnings after the bell Wednesday. Rivian shares sank about 25 percent, and Lucids stock dropped around 17 percent. Rivian forecast it will make 57,000 vehicles in 2024, slightly less than the 57,232 vehicles it produced in 2023. Lucid said it expects to make 9,000 vehicles in 2024, more than the 8,428 vehicles it made in 2023.",
        "Mauritius blocks Norwegian cruise ship over fears of a potential cholera outbreak Local authorities on Sunday denied permission for the Norwegian Dawn ship, which has 2,184 passengers and 1,026 crew on board, to access the Mauritius capital of Port Louis, citing “potential health risks.” The Mauritius Ports Authority said Sunday that samples were taken from at least 15 passengers on board the cruise ship. A spokesperson for the U.S.-headquartered Norwegian Cruise Line Holdings said Sunday that 'a small number of guests experienced mild symptoms of a stomach-related illness' during Norwegian Dawns South Africa voyage.",
        "Intuitive Machines lands on the moon in historic first for a U.S. company Intuitive Machines Nova-C cargo lander, named Odysseus after the mythological Greek hero, is the first U.S. spacecraft to soft land on the lunar surface since 1972. Intuitive Machines is the first company to pull off a moon landing — government agencies have carried out all previously successful missions. The company's stock surged in extended trading Thursday, after falling 11 percent in regular trading.",
        "Lunar landing photos: Intuitive Machines Odysseus sends back first images from the moon Intuitive Machines cargo moon lander Odysseus returned its first images from the surface. Company executives believe the lander caught its landing gear sideways on the moon's surface while touching down and tipped over. Despite resting on its side, the company's historic IM-1 mission is still operating on the moon.",
    ]

    def docs_to_embeddings(docs: list) -> list:
        openai = OpenAI()
        document_embeddings = []
        for doc in docs:
            response = (
                openai.embeddings.create(input=doc, model="text-embedding-3-small")
                .data[0]
                .embedding
            )
            document_embeddings.append(response)
        return document_embeddings

    article_embeddings = docs_to_embeddings(articles) # Note: you would typically do this once with your articles and put the embeddings & metadata in a database
    ```
  </Tab>

  <Tab title="TypeScript">
    ```typescript lines theme={null}
    require('dotenv').config();
    import { OpenAI } from 'openai';
    import * as weave from 'weave';

    interface Article {
        text: string;
        embedding?: number[];
    }

    const articles: Article[] = [
        { 
            text: `Novo Nordisk and Eli Lilly rival soars 32 percent after promising weight loss drug results Shares of Denmarks Zealand Pharma shot 32 percent higher in morning trade, after results showed success in its liver disease treatment survodutide, which is also on trial as a drug to treat obesity. The trial tells us that the 6mg dose is safe, which is the top dose used in the ongoing [Phase 3] obesity trial too, one analyst said in a note. The results come amid feverish investor interest in drugs that can be used for weight loss.`
        },
        { 
            text: `Berkshire shares jump after big profit gain as Buffetts conglomerate nears $1 trillion valuation Berkshire Hathaway shares rose on Monday after Warren Buffetts conglomerate posted strong earnings for the fourth quarter over the weekend. Berkshires Class A and B shares jumped more than 1.5%, each. Class A shares are higher by more than 17% this year, while Class B has gained more than 18%. Berkshire was last valued at $930.1 billion, up from $905.5 billion where it closed on Friday, according to FactSet. Berkshire on Saturday posted fourth-quarter operating earnings of $8.481 billion, about 28 percent higher than the $6.625 billion from the year-ago period, driven by big gains in its insurance business. Operating earnings refers to profits from businesses across insurance, railroads and utilities. Meanwhile, Berkshires cash levels also swelled to record levels. The conglomerate held $167.6 billion in cash in the fourth quarter, surpassing the $157.2 billion record the conglomerate held in the prior quarter.`
        },
        { 
            text: `Highmark Health says its combining tech from Google and Epic to give doctors easier access to information Highmark Health announced it is integrating technology from Google Cloud and the health-care software company Epic Systems. The integration aims to make it easier for both payers and providers to access key information they need, even if its stored across multiple points and formats, the company said. Highmark is the parent company of a health plan with 7 million members, a provider network of 14 hospitals and other entities`
        }
    ];

    function cosineSimilarity(a: number[], b: number[]): number {
        const dotProduct = a.reduce((sum, val, i) => sum + val * b[i], 0);
        const magnitudeA = Math.sqrt(a.reduce((sum, val) => sum + val * val, 0));
        const magnitudeB = Math.sqrt(b.reduce((sum, val) => sum + val * val, 0));
        return dotProduct / (magnitudeA * magnitudeB);
    }

    const docsToEmbeddings = weave.op(async function(docs: Article[]): Promise<Article[]> {
        const openai = new OpenAI();
        const enrichedDocs = await Promise.all(docs.map(async (doc) => {
            const response = await openai.embeddings.create({
                input: doc.text,
                model: "text-embedding-3-small"
            });
            return {
                ...doc,
                embedding: response.data[0].embedding
            };
        }));
        return enrichedDocs;
    });
    ```
  </Tab>
</Tabs>

## Create a RAG app

With the knowledge base in place, you can now build the RAG application itself. This section combines the retrieval step with an LLM call and wraps both with Weave so that every input and output is tracked automatically.

Next, wrap the retrieval function `get_most_relevant_document` with a `weave.op()` decorator and create a `Model` class. Wrapping the retrieval function with `weave.op()` lets Weave capture its inputs and outputs for every call, which is what makes the retrieval step inspectable later. Call `weave.init('<team-name>/rag-quickstart')` to begin tracking all the inputs and outputs of your functions for later inspection. If you don't specify a team name, Weave records the output to your [W\&B default team or entity](/platform/app/settings-page/user-settings#default-team).

<Tabs>
  <Tab title="Python">
    ```python lines theme={null}
    from openai import OpenAI
    import weave
    from weave import Model
    import numpy as np
    import asyncio

    @weave.op()
    def get_most_relevant_document(query):
        openai = OpenAI()
        query_embedding = (
            openai.embeddings.create(input=query, model="text-embedding-3-small")
            .data[0]
            .embedding
        )
        similarities = [
            np.dot(query_embedding, doc_emb)
            / (np.linalg.norm(query_embedding) * np.linalg.norm(doc_emb))
            for doc_emb in article_embeddings
        ]
        # Get the index of the most similar document
        most_relevant_doc_index = np.argmax(similarities)
        return articles[most_relevant_doc_index]

    class RAGModel(Model):
        system_message: str
        model_name: str = "gpt-3.5-turbo-1106"

        @weave.op()
        def predict(self, question: str) -> dict: # note: `question` will be used later to select data from our evaluation rows
            from openai import OpenAI
            context = get_most_relevant_document(question)
            client = OpenAI()
            query = f"""Use the following information to answer the subsequent question. If the answer cannot be found, write "I don't know."
            Context:
            \"\"\"
            {context}
            \"\"\"
            Question: {question}"""
            response = client.chat.completions.create(
                model=self.model_name,
                messages=[
                    {"role": "system", "content": self.system_message},
                    {"role": "user", "content": query},
                ],
                temperature=0.0,
                response_format={"type": "text"},
            )
            answer = response.choices[0].message.content
            return {'answer': answer, 'context': context}

    # Set your team and project names
    weave.init('<team-name>/rag-quickstart')
    model = RAGModel(
        system_message="You are an expert in finance and answer questions related to finance, financial services, and financial markets. When responding based on provided information, be sure to cite the source."
    )
    model.predict("What significant result was reported about Zealand Pharma's obesity trial?")
    ```
  </Tab>

  <Tab title="TypeScript">
    ```typescript lines theme={null}
    class RAGModel {
        private openai: OpenAI;
        private systemMessage: string;
        private modelName: string;
        private articleEmbeddings: Article[];

        constructor(config: {
            systemMessage: string;
            modelName?: string;
            articleEmbeddings: Article[];
        }) {
            this.openai = new OpenAI();
            this.systemMessage = config.systemMessage;
            this.modelName = config.modelName || "gpt-3.5-turbo-1106";
            this.articleEmbeddings = config.articleEmbeddings;
            this.predict = weave.op(this, this.predict);
        }

        async predict(question: string): Promise<{
            answer: string;
            context: string;
        }> {
            const context = await this.getMostRelevantDocument(question);
            
            const response = await this.openai.chat.completions.create({
                model: this.modelName,
                messages: [
                    { role: "system", content: this.systemMessage },
                    { role: "user", content: `Use the following information to answer the subsequent question. If the answer cannot be found, write "I don't know."
                        Context:
                        """
                        ${context}
                        """
                        Question: ${question}` }
                ],
                temperature: 0
            });

            return {
                answer: response.choices[0].message.content || "",
                context
            };
        }
    }
    ```
  </Tab>
</Tabs>

## Evaluate with an LLM judge

Now that your RAG application is running and being tracked by Weave, the next step is to evaluate how well it answers questions. This section shows how to use an LLM as an automatic judge so you can score your application's responses without writing labels by hand.

When there aren't simple ways to evaluate your application, one approach is to use an LLM to evaluate aspects of it. Here's an example of using an LLM judge to measure the context precision by prompting it to verify whether the context was useful in arriving at the given answer. This prompt was augmented from the popular [RAGAS framework](https://docs.ragas.io/en/stable/).

### Define a scoring function

As in the [Build an Evaluation pipeline tutorial](/weave/tutorial-eval), define a set of example rows to test your app against and a scoring function. The scoring function takes one row and evaluates it. The input arguments should match the corresponding keys in your row, so `question` here is taken from the row dictionary. `output` is the output of the model. The input to the model is taken from the example based on its input argument, so `question` here too. This example uses `async` functions so they run in parallel. For a quick introduction to async, see the [Python `asyncio` documentation](https://docs.python.org/3/library/asyncio.html).

<Tabs>
  <Tab title="Python">
    ```python lines theme={null}
    from openai import OpenAI
    import weave
    import asyncio

    @weave.op()
    async def context_precision_score(question, output):
        context_precision_prompt = """Given question, answer and context verify if the context was useful in arriving at the given answer. Give verdict as "1" if useful and "0" if not with json output.
        Output in only valid JSON format.

        question: {question}
        context: {context}
        answer: {answer}
        verdict: """
        client = OpenAI()

        prompt = context_precision_prompt.format(
            question=question,
            context=output['context'],
            answer=output['answer'],
        )

        response = client.chat.completions.create(
            model="gpt-4-turbo-preview",
            messages=[{"role": "user", "content": prompt}],
            response_format={ "type": "json_object" }
        )
        response_message = response.choices[0].message
        response = json.loads(response_message.content)
        return {
            "verdict": int(response["verdict"]) == 1,
        }

    questions = [
        {"question": "What significant result was reported about Zealand Pharma's obesity trial?"},
        {"question": "How much did Berkshire Hathaway's cash levels increase in the fourth quarter?"},
        {"question": "What is the goal of Highmark Health's integration of Google Cloud and Epic Systems technology?"},
        {"question": "What were Rivian and Lucid's vehicle production forecasts for 2024?"},
        {"question": "Why was the Norwegian Dawn cruise ship denied access to Mauritius?"},
        {"question": "Which company achieved the first U.S. moon landing since 1972?"},
        {"question": "What issue did Intuitive Machines' lunar lander encounter upon landing on the moon?"}
    ]
    evaluation = weave.Evaluation(dataset=questions, scorers=[context_precision_score])
    asyncio.run(evaluation.evaluate(model)) # note: you'll need to define a model to evaluate
    ```
  </Tab>

  <Tab title="TypeScript">
    ```typescript lines theme={null}
    const contextPrecisionScore = weave.op(async function(args: {
        datasetRow: QuestionRow;
        modelOutput: { answer: string; context: string; }
    }): Promise<ScorerResult> {
        const openai = new OpenAI();
        
        const prompt = `Given question, answer and context verify if the context was useful...`;

        const response = await openai.chat.completions.create({
            model: "gpt-4-turbo-preview",
            messages: [{ role: "user", content: prompt }],
            response_format: { type: "json_object" }
        });

        const result = JSON.parse(response.choices[0].message.content || "{}");
        return {
            verdict: parseInt(result.verdict) === 1
        };
    });

    const evaluation = new weave.Evaluation({
        dataset: createQuestionDataset(),
        scorers: [contextPrecisionScore]
    });

    await evaluation.evaluate({
        model: weave.op((args: { datasetRow: QuestionRow }) => 
            model.predict(args.datasetRow.question)
        )
    });
    ```
  </Tab>
</Tabs>

### Optional: Define a `Scorer` class

The scoring function in the previous section works well for simple cases, but a `Scorer` class is useful when you want to reuse the same judge across evaluations or customize how scores are aggregated. The following steps show when and how to define one.

In some applications you may want to create custom evaluation classes (for example, a standardized `LLMJudge` class with parameters such as chat model and prompt), custom scoring of each row, and custom calculation of an aggregate score. Weave defines a list of ready-to-use `Scorer` classes and also makes it easy to create a custom `Scorer`. The following example shows how to create a custom `class CorrectnessLLMJudge(Scorer)`.

At a high level, the steps to create a custom Scorer are:

1. Define a custom class that inherits from `weave.flow.scorer.Scorer`.
2. Overwrite the `score` function and add a `@weave.op()` if you want to track each call of the function.
   * This function has to define an `output` argument where the prediction of the model is passed. Define it as type `Optional[dict]` in case the model returns "None".
   * The rest of the arguments can either be a general `Any` or `dict` or can select specific columns from the dataset that is used to evaluate the model using the `weave.Evaluate` class. They have to have the exact same names as the column names or keys of a single row after being passed to `preprocess_model_input` if that is used.
3. Optional: Overwrite the `summarize` function to customize the calculation of the aggregate score. By default, Weave uses the `weave.flow.scorer.auto_summarize` function if you don't define a custom function.
   * This function has to have a `@weave.op()` decorator.

<Tabs>
  <Tab title="Python">
    ```python lines theme={null}
    from weave import Scorer

    class CorrectnessLLMJudge(Scorer):
        prompt: str
        model_name: str
        device: str

        @weave.op()
        async def score(self, output: Optional[dict], query: str, answer: str) -> Any:
            """Score the correctness of the predictions by comparing the pred, query, target.
            Args:
                - output: the dict that will be provided by the model that is evaluated
                - query: the question asked - as defined in the dataset
                - answer: the target answer - as defined in the dataset
            Returns:
                - single dict {metric name: single evaluation value}"""

            # get_model is defined as general model getter based on provided params (OpenAI,HF...)
            eval_model = get_model(
                model_name = self.model_name,
                prompt = self.prompt
                device = self.device,
            )
            # async evaluation to speed up evaluation - this doesn't have to be async
            grade = await eval_model.async_predict(
                {
                    "query": query,
                    "answer": answer,
                    "result": output.get("result"),
                }
            )
            # output parsing - could be done more reobustly with pydantic
            evaluation = "incorrect" not in grade["text"].strip().lower()

            # the column name displayed in Weave
            return {"correct": evaluation}

        @weave.op()
        def summarize(self, score_rows: list) -> Optional[dict]:
            """Aggregate all the scores that are calculated for each row by the scoring function.
            Args:
                - score_rows: a list of dicts. Each dict has metrics and scores
            Returns:
                - nested dict with the same structure as the input"""

            # if nothing is provided the weave.flow.scorer.auto_summarize function is used
            # return auto_summarize(score_rows)

            valid_data = [x.get("correct") for x in score_rows if x.get("correct") is not None]
            count_true = list(valid_data).count(True)
            int_data = [int(x) for x in valid_data]

            sample_mean = np.mean(int_data) if int_data else 0
            sample_variance = np.var(int_data) if int_data else 0
            sample_error = np.sqrt(sample_variance / len(int_data)) if int_data else 0

            # the extra "correct" layer is not necessary but adds structure in the UI
            return {
                "correct": {
                    "true_count": count_true,
                    "true_fraction": sample_mean,
                    "stderr": sample_error,
                }
            }
    ```
  </Tab>

  <Tab title="TypeScript">
    ```plaintext theme={null}
    This feature is not available in TypeScript yet.
    ```
  </Tab>
</Tabs>

To use this as a scorer, initialize it and pass it to the `scorers` argument in your `Evaluation` like this:

<Tabs>
  <Tab title="Python">
    ```python lines theme={null}
    evaluation = weave.Evaluation(dataset=questions, scorers=[CorrectnessLLMJudge()])
    ```
  </Tab>

  <Tab title="TypeScript">
    ```plaintext theme={null}
    This feature is not available in TypeScript yet.
    ```
  </Tab>
</Tabs>

## Put it all together

This section consolidates everything from the previous steps into a single end-to-end example so you can see how the pieces fit together and adapt them to your own RAG application.

To get the same result for your RAG apps:

* Wrap LLM calls and retrieval step functions with `weave.op()`.
* Optional: Create a `Model` subclass with `predict` function and app details.
* Collect examples to evaluate.
* Create scoring functions that score one example.
* Use `Evaluation` class to run evaluations on your examples.

**NOTE:** Sometimes the async execution of Evaluations triggers a rate limit on the models of OpenAI, Anthropic, and others. To prevent that, you can set an environment variable to limit the number of parallel workers, for example, `WEAVE_PARALLELISM=3`.

Here's the code in its entirety.

<Tabs>
  <Tab title="Python">
    ```python lines {34,52-77} theme={null}
    from openai import OpenAI
    import weave
    from weave import Model
    import numpy as np
    import json
    import asyncio

    # Examples to use for evaluations
    articles = [
        "Novo Nordisk and Eli Lilly rival soars 32 percent after promising weight loss drug results Shares of Denmarks Zealand Pharma shot 32 percent higher in morning trade, after results showed success in its liver disease treatment survodutide, which is also on trial as a drug to treat obesity. The trial “tells us that the 6mg dose is safe, which is the top dose used in the ongoing [Phase 3] obesity trial too,” one analyst said in a note. The results come amid feverish investor interest in drugs that can be used for weight loss.",
        "Berkshire shares jump after big profit gain as Buffetts conglomerate nears $1 trillion valuation Berkshire Hathaway shares rose on Monday after Warren Buffetts conglomerate posted strong earnings for the fourth quarter over the weekend. Berkshires Class A and B shares jumped more than 1.5%, each. Class A shares are higher by more than 17% this year, while Class B has gained more than 18%. Berkshire was last valued at $930.1 billion, up from $905.5 billion where it closed on Friday, according to FactSet. Berkshire on Saturday posted fourth-quarter operating earnings of $8.481 billion, about 28 percent higher than the $6.625 billion from the year-ago period, driven by big gains in its insurance business. Operating earnings refers to profits from businesses across insurance, railroads and utilities. Meanwhile, Berkshires cash levels also swelled to record levels. The conglomerate held $167.6 billion in cash in the fourth quarter, surpassing the $157.2 billion record the conglomerate held in the prior quarter.",
        "Highmark Health says its combining tech from Google and Epic to give doctors easier access to information Highmark Health announced it is integrating technology from Google Cloud and the health-care software company Epic Systems. The integration aims to make it easier for both payers and providers to access key information they need, even if it's stored across multiple points and formats, the company said. Highmark is the parent company of a health plan with 7 million members, a provider network of 14 hospitals and other entities",
        "Rivian and Lucid shares plunge after weak EV earnings reports Shares of electric vehicle makers Rivian and Lucid fell Thursday after the companies reported stagnant production in their fourth-quarter earnings after the bell Wednesday. Rivian shares sank about 25 percent, and Lucids stock dropped around 17 percent. Rivian forecast it will make 57,000 vehicles in 2024, slightly less than the 57,232 vehicles it produced in 2023. Lucid said it expects to make 9,000 vehicles in 2024, more than the 8,428 vehicles it made in 2023.",
        "Mauritius blocks Norwegian cruise ship over fears of a potential cholera outbreak Local authorities on Sunday denied permission for the Norwegian Dawn ship, which has 2,184 passengers and 1,026 crew on board, to access the Mauritius capital of Port Louis, citing “potential health risks.” The Mauritius Ports Authority said Sunday that samples were taken from at least 15 passengers on board the cruise ship. A spokesperson for the U.S.-headquartered Norwegian Cruise Line Holdings said Sunday that 'a small number of guests experienced mild symptoms of a stomach-related illness' during Norwegian Dawns South Africa voyage.",
        "Intuitive Machines lands on the moon in historic first for a U.S. company Intuitive Machines Nova-C cargo lander, named Odysseus after the mythological Greek hero, is the first U.S. spacecraft to soft land on the lunar surface since 1972. Intuitive Machines is the first company to pull off a moon landing — government agencies have carried out all previously successful missions. The company's stock surged in extended trading Thursday, after falling 11 percent in regular trading.",
        "Lunar landing photos: Intuitive Machines Odysseus sends back first images from the moon Intuitive Machines cargo moon lander Odysseus returned its first images from the surface. Company executives believe the lander caught its landing gear sideways on the surface of the moon while touching down and tipped over. Despite resting on its side, the company's historic IM-1 mission is still operating on the moon.",
    ]

    def docs_to_embeddings(docs: list) -> list:
        openai = OpenAI()
        document_embeddings = []
        for doc in docs:
            response = (
                openai.embeddings.create(input=doc, model="text-embedding-3-small")
                .data[0]
                .embedding
            )
            document_embeddings.append(response)
        return document_embeddings

    article_embeddings = docs_to_embeddings(articles) # Note: you would typically do this once with your articles and put the embeddings & metadata in a database

    # Add a decorator to the retrieval step
    @weave.op()
    def get_most_relevant_document(query):
        openai = OpenAI()
        query_embedding = (
            openai.embeddings.create(input=query, model="text-embedding-3-small")
            .data[0]
            .embedding
        )
        similarities = [
            np.dot(query_embedding, doc_emb)
            / (np.linalg.norm(query_embedding) * np.linalg.norm(doc_emb))
            for doc_emb in article_embeddings
        ]
        # Get the index of the most similar document
        most_relevant_doc_index = np.argmax(similarities)
        return articles[most_relevant_doc_index]

    # Create a Model subclass with details about the app, along with a predict function that produces a response
    class RAGModel(Model):
        system_message: str
        model_name: str = "gpt-3.5-turbo-1106"

        @weave.op()
        def predict(self, question: str) -> dict: # note: `question` will be used later to select data from our evaluation rows
            from openai import OpenAI
            context = get_most_relevant_document(question)
            client = OpenAI()
            query = f"""Use the following information to answer the subsequent question. If the answer cannot be found, write "I don't know."
            Context:
            \"\"\"
            {context}
            \"\"\"
            Question: {question}"""
            response = client.chat.completions.create(
                model=self.model_name,
                messages=[
                    {"role": "system", "content": self.system_message},
                    {"role": "user", "content": query},
                ],
                temperature=0.0,
                response_format={"type": "text"},
            )
            answer = response.choices[0].message.content
            return {'answer': answer, 'context': context}

    # Set your team and project names
    weave.init('<team-name>/rag-quickstart')
    model = RAGModel(
        system_message="You are an expert in finance and answer questions related to finance, financial services, and financial markets. When responding based on provided information, be sure to cite the source."
    )

    # Here is our scoring function uses our question and output to product a score
    @weave.op()
    async def context_precision_score(question, output):
        context_precision_prompt = """Given question, answer and context verify if the context was useful in arriving at the given answer. Give verdict as "1" if useful and "0" if not with json output.
        Output in only valid JSON format.

        question: {question}
        context: {context}
        answer: {answer}
        verdict: """
        client = OpenAI()

        prompt = context_precision_prompt.format(
            question=question,
            context=output['context'],
            answer=output['answer'],
        )

        response = client.chat.completions.create(
            model="gpt-4-turbo-preview",
            messages=[{"role": "user", "content": prompt}],
            response_format={ "type": "json_object" }
        )
        response_message = response.choices[0].message
        response = json.loads(response_message.content)
        return {
            "verdict": int(response["verdict"]) == 1,
        }

    questions = [
        {"question": "What significant result was reported about Zealand Pharma's obesity trial?"},
        {"question": "How much did Berkshire Hathaway's cash levels increase in the fourth quarter?"},
        {"question": "What is the goal of Highmark Health's integration of Google Cloud and Epic Systems technology?"},
        {"question": "What were Rivian and Lucid's vehicle production forecasts for 2024?"},
        {"question": "Why was the Norwegian Dawn cruise ship denied access to Mauritius?"},
        {"question": "Which company achieved the first U.S. moon landing since 1972?"},
        {"question": "What issue did Intuitive Machines' lunar lander encounter upon landing on the moon?"}
    ]

    # Define an Evaluation object and pass example questions along with scoring functions
    evaluation = weave.Evaluation(dataset=questions, scorers=[context_precision_score])
    asyncio.run(evaluation.evaluate(model))
    ```
  </Tab>

  <Tab title="TypeScript">
    ```typescript lines theme={null}
    require('dotenv').config();
    import { OpenAI } from 'openai';
    import * as weave from 'weave';

    interface Article {
        text: string;
        embedding?: number[];
    }

    const articles: Article[] = [
        { 
            text: `Novo Nordisk and Eli Lilly rival soars 32 percent after promising weight loss drug results Shares of Denmarks Zealand Pharma shot 32 percent higher in morning trade, after results showed success in its liver disease treatment survodutide, which is also on trial as a drug to treat obesity. The trial tells us that the 6mg dose is safe, which is the top dose used in the ongoing [Phase 3] obesity trial too, one analyst said in a note. The results come amid feverish investor interest in drugs that can be used for weight loss.`
        },
        { 
            text: `Berkshire shares jump after big profit gain as Buffetts conglomerate nears $1 trillion valuation Berkshire Hathaway shares rose on Monday after Warren Buffetts conglomerate posted strong earnings for the fourth quarter over the weekend. Berkshires Class A and B shares jumped more than 1.5%, each. Class A shares are higher by more than 17% this year, while Class B has gained more than 18%. Berkshire was last valued at $930.1 billion, up from $905.5 billion where it closed on Friday, according to FactSet. Berkshire on Saturday posted fourth-quarter operating earnings of $8.481 billion, about 28 percent higher than the $6.625 billion from the year-ago period, driven by big gains in its insurance business. Operating earnings refers to profits from businesses across insurance, railroads and utilities. Meanwhile, Berkshires cash levels also swelled to record levels. The conglomerate held $167.6 billion in cash in the fourth quarter, surpassing the $157.2 billion record the conglomerate held in the prior quarter.`
        },
        { 
            text: `Highmark Health says its combining tech from Google and Epic to give doctors easier access to information Highmark Health announced it is integrating technology from Google Cloud and the health-care software company Epic Systems. The integration aims to make it easier for both payers and providers to access key information they need, even if its stored across multiple points and formats, the company said. Highmark is the parent company of a health plan with 7 million members, a provider network of 14 hospitals and other entities`
        }
    ];

    function cosineSimilarity(a: number[], b: number[]): number {
        const dotProduct = a.reduce((sum, val, i) => sum + val * b[i], 0);
        const magnitudeA = Math.sqrt(a.reduce((sum, val) => sum + val * val, 0));
        const magnitudeB = Math.sqrt(b.reduce((sum, val) => sum + val * val, 0));
        return dotProduct / (magnitudeA * magnitudeB);
    }

    const docsToEmbeddings = weave.op(async function(docs: Article[]): Promise<Article[]> {
        const openai = new OpenAI();
        const enrichedDocs = await Promise.all(docs.map(async (doc) => {
            const response = await openai.embeddings.create({
                input: doc.text,
                model: "text-embedding-3-small"
            });
            return {
                ...doc,
                embedding: response.data[0].embedding
            };
        }));
        return enrichedDocs;
    });

    class RAGModel {
        private openai: OpenAI;
        private systemMessage: string;
        private modelName: string;
        private articleEmbeddings: Article[];

        constructor(config: {
            systemMessage: string;
            modelName?: string;
            articleEmbeddings: Article[];
        }) {
            this.openai = new OpenAI();
            this.systemMessage = config.systemMessage;
            this.modelName = config.modelName || "gpt-3.5-turbo-1106";
            this.articleEmbeddings = config.articleEmbeddings;
            this.predict = weave.op(this, this.predict);
        }

        private async getMostRelevantDocument(query: string): Promise<string> {
            const queryEmbedding = await this.openai.embeddings.create({
                input: query,
                model: "text-embedding-3-small"
            });

            const similarities = this.articleEmbeddings.map(doc => {
                if (!doc.embedding) return 0;
                return cosineSimilarity(queryEmbedding.data[0].embedding, doc.embedding);
            });

            const mostRelevantIndex = similarities.indexOf(Math.max(...similarities));
            return this.articleEmbeddings[mostRelevantIndex].text;
        }

        async predict(question: string): Promise<{
            answer: string;
            context: string;
        }> {
            const context = await this.getMostRelevantDocument(question);
            
            const response = await this.openai.chat.completions.create({
                model: this.modelName,
                messages: [
                    { role: "system", content: this.systemMessage },
                    { 
                        role: "user", 
                        content: `Use the following information to answer the subsequent question. If the answer cannot be found, write "I don't know."
                        Context:
                        """
                        ${context}
                        """
                        Question: ${question}`
                    }
                ],
                temperature: 0
            });

            return {
                answer: response.choices[0].message.content || "",
                context
            };
        }
    }

    interface ScorerResult {
        verdict: boolean;
    }

    interface QuestionRow {
        question: string;
    }

    function createQuestionDataset(): weave.Dataset<QuestionRow> {
        return new weave.Dataset<QuestionRow>({
            id: 'rag-questions',
            rows: [
                { question: "What significant result was reported about Zealand Pharma's obesity trial?" },
                { question: "How much did Berkshire Hathaway's cash levels increase in the fourth quarter?" },
                { question: "What is the goal of Highmark Health's integration of Google Cloud and Epic Systems technology?" }
            ]
        });
    }

    const contextPrecisionScore = weave.op(async function(args: {
        datasetRow: QuestionRow;
        modelOutput: { answer: string; context: string; }
    }): Promise<ScorerResult> {
        const openai = new OpenAI();
        
        const prompt = `Given question, answer and context verify if the context was useful in arriving at the given answer. Give verdict as "1" if useful and "0" if not with json output.
        Output in only valid JSON format.

        question: ${args.datasetRow.question}
        context: ${args.modelOutput.context}
        answer: ${args.modelOutput.answer}
        verdict: `;

        const response = await openai.chat.completions.create({
            model: "gpt-4-turbo-preview",
            messages: [{ role: "user", content: prompt }],
            response_format: { type: "json_object" }
        });

        const result = JSON.parse(response.choices[0].message.content || "{}");
        return {
            verdict: parseInt(result.verdict) === 1
        };
    });

    async function main() {
        # Set your team and project names
        await weave.init('<team-name>/rag-quickstart');
        
        const articleEmbeddings = await docsToEmbeddings(articles);
        
        const model = new RAGModel({
            systemMessage: "You are an expert in finance and answer questions related to finance, financial services, and financial markets. When responding based on provided information, be sure to cite the source.",
            articleEmbeddings
        });

        const evaluation = new weave.Evaluation({
            dataset: createQuestionDataset(),
            scorers: [contextPrecisionScore]
        });

        const results = await evaluation.evaluate({
            model: weave.op((args: { datasetRow: QuestionRow }) => 
                model.predict(args.datasetRow.question)
            )
        });
        
        console.log('Evaluation results:', results);
    }

    if (require.main === module) {
        main().catch(console.error);
    }
    ```
  </Tab>
</Tabs>

## Conclusion

This tutorial showed how to build observability into different steps of your applications, like the retrieval step in this example. You also learned how to build more complex scoring functions, such as an LLM judge, for automatic evaluation of application responses.

## Next steps

See the [RAG++ course](https://www.wandb.courses/courses/rag-in-production?utm_source=wandb_docs\&utm_medium=code\&utm_campaign=weave_docs) for a more advanced dive into practical RAG techniques for engineers, where you learn production-ready solutions from W\&B, Cohere, and Weaviate to optimize performance, cut costs, and enhance the accuracy and relevance of your applications.


# Trace nested functions
Source: https://docs.wandb.ai/weave/tutorial-tracing_2

Learn how to track deeply nested Call structures with W&B tracing

This tutorial shows you how to trace nested functions in Weave so you can monitor the full execution flow of an LLM-powered application, including its sub-functions and their parent-child relationships. By the end, you'll be able to capture, visualize, and add metadata to nested traces in the Weave **Traces** page.

LLM-powered applications can contain multiple LLM calls, additional data processing, and validation logic that's important to monitor. You can track these nested functions and their parent-child relationships in Weave using the `@weave.op()` decorator (Python), or wrap it with `weave.op()` (TypeScript).

Decorate functions and sub-functions as granularly as possible to capture the application's complete execution flow. This helps you better understand and shape your application's behavior.

## Trace nested functions

This section steps through an example that traces a function and its nested sub-functions. The following code builds on the [quickstart example](/weave/quickstart) and adds logic to count the returned items from the LLM and wrap them in a higher-level function. Additionally, the example uses `weave.op()` to trace every function, its call order, and its parent-child relationship:

<Tabs>
  <Tab title="Python">
    ```python {1,7,26,32,42} theme={null}
    import weave
    import json
    from openai import OpenAI

    client = OpenAI()

    @weave.op()
    def extract_dinos(sentence: str) -> dict:
        response = client.chat.completions.create(
            model="gpt-4o",
            messages=[
                {
                    "role": "system",
                    "content": """Extract any dinosaur `name`, their `common_name`, \
    names and whether its `diet` is a herbivore or carnivore, in JSON format."""
                },
                {
                    "role": "user",
                    "content": sentence
                }
                ],
                response_format={ "type": "json_object" }
            )
        return response.choices[0].message.content

    @weave.op()
    def count_dinos(dino_data: dict) -> int:
        # count the number of items in the returned list
        k = list(dino_data.keys())[0]
        return len(dino_data[k])

    @weave.op()
    def dino_tracker(sentence: str) -> dict:
        # extract dinosaurs using an LLM
        dino_data = extract_dinos(sentence)

        # count the number of dinosaurs returned
        dino_data = json.loads(dino_data)
        n_dinos = count_dinos(dino_data)
        return {"n_dinosaurs": n_dinos, "dinosaurs": dino_data}

    weave.init('jurassic-park')

    sentence = """I watched as a Tyrannosaurus rex (T. rex) chased after a Triceratops (Trike), \
    both carnivore and herbivore locked in an ancient dance. Meanwhile, a gentle giant \
    Brachiosaurus (Brachi) calmly munched on treetops, blissfully unaware of the chaos below."""

    result = dino_tracker(sentence)
    print(result)
    ```

    **Nested functions**

    When you run the preceding code, the **Traces** page shows the inputs and outputs from the two nested functions (`extract_dinos` and `count_dinos`), as well as the automatically-logged OpenAI trace.

    <img alt="Nested Weave Traces page showing the center trace tree panel and the details panel for the selected Call" />
  </Tab>

  <Tab title="TypeScript">
    ```typescript theme={null}
    import OpenAI from 'openai';
    import * as weave from 'weave';

    const openai = new OpenAI();

    const extractDinos = weave.op(async (sentence: string) => {
      const response = await openai.chat.completions.create({
        model: 'gpt-4o',
        messages: [
          {
            role: 'system',
            content:
              'Extract any dinosaur `name`, their `common_name`, names and whether its `diet` is a herbivore or carnivore, in JSON format.',
          },
          {role: 'user', content: sentence},
        ],
        response_format: {type: 'json_object'},
      });
      return response.choices[0].message.content;
    });

    const countDinos = weave.op(async (dinoData: string) => {
      const parsed = JSON.parse(dinoData);
      return Object.keys(parsed).length;
    });

    const dinoTracker = weave.op(async (sentence: string) => {
      const dinoData = await extractDinos(sentence);
      const nDinos = await countDinos(dinoData);
      return {nDinos, dinoData};
    });

    async function main() {
      await weave.init('jurassic-park');

      const sentence = `I watched as a Tyrannosaurus rex (T. rex) chased after a Triceratops (Trike),
            both carnivore and herbivore locked in an ancient dance. Meanwhile, a gentle giant
            Brachiosaurus (Brachi) calmly munched on treetops, blissfully unaware of the chaos below.`;

      const result = await dinoTracker(sentence);
      console.log(result);
    }

    main();

    ```

    **Nested functions**

    When you run the preceding code, the **Traces** page shows the inputs and outputs from the two nested functions (`extract_dinos` and `count_dinos`), as well as the automatically-logged OpenAI trace.

    <img alt="Nested Weave Traces page showing the center trace tree panel and the details panel for the selected Call" />
  </Tab>
</Tabs>

After running the code, you have a complete nested trace in Weave that captures each function call and its relationship to the others.

## Track metadata

Now that Weave traces your nested functions, you can enrich those traces with additional context about the run, such as the user or environment. You can track metadata by using the `weave.attributes` context manager and passing it a dictionary of the metadata to track at call time.

Continuing the previous example:

<Tabs>
  <Tab title="Python">
    ```python lines {1,10-11} theme={null}
    import weave

    weave.init('jurassic-park')

    sentence = """I watched as a Tyrannosaurus rex (T. rex) chased after a Triceratops (Trike), \
    both carnivore and herbivore locked in an ancient dance. Meanwhile, a gentle giant \
    Brachiosaurus (Brachi) calmly munched on treetops, blissfully unaware of the chaos below."""

    # track metadata alongside the previously defined function
    with weave.attributes({'user_id': 'lukas', 'env': 'production'}):
        result = dino_tracker(sentence)
    ```
  </Tab>

  <Tab title="TypeScript">
    ```plaintext theme={null}
    This feature is not available in TypeScript yet.
    ```
  </Tab>
</Tabs>

<Note>
  Track metadata at run time, such as your user IDs and your code's environment status (development, staging, or production).

  To track system settings, such as a system prompt, use [Weave Models](/weave/guides/core-types/models).
</Note>

For more information on using attributes, see [Define and log attributes](/weave/guides/tools/attributes).

## What's next

* Follow the [App Versioning tutorial](/weave/tutorial-weave_models) to capture, version, and organize ad-hoc prompt, model, and application changes.


# Tutorial: App versioning
Source: https://docs.wandb.ai/weave/tutorial-weave_models

Learn how to use Weave Model to track and version your application and its parameters

Tracking [inputs, outputs, and metadata](/weave/quickstart), along with [data flowing through your app](/weave/tutorial-tracing_2), is essential to understanding your system's performance. Versioning your app over time is equally important, because it shows how changes to your code or parameters affect your outputs. Weave's `Model` class tracks these changes for you.

This tutorial is for developers who want to compare and reproduce results across different versions of an LLM application. By the end, you'll have a versioned Weave `Model` that captures your application's parameters and code. You'll also know how to retrieve and reuse a previous version.

In this tutorial, you'll learn:

* How to use Weave `Model` to track and version your application and its parameters.
* How to export, modify, and reuse a Weave `Model` already logged.

## Use `weave.Model`

<Warning>
  The `weave.Model` class is only supported in Python.
</Warning>

Using Weave `Model`s means that Weave stores and versions parameters such as model vendor IDs, prompts, temperature, and more when they change.

To create a `Model` in Weave, you need the following:

* A class that inherits from `weave.Model`.
* Type definitions on all class fields.
* A typed `invoke` function with the `@weave.op()` decorator.

When you change the class fields or the code that defines your model, Weave logs these changes and updates the version. This ensures that you can compare the generations across different versions of your app.

In the following example, Weave tracks and versions the model name, temperature, and system prompt:

<Tabs>
  <Tab title="Python">
    ```python lines {26,33-34} theme={null}
    import json
    from openai import OpenAI

    import weave

    @weave.op()
    def extract_dinos(wmodel: weave.Model, sentence: str) -> dict:
        response = wmodel.client.chat.completions.create(
            model=wmodel.model_name,
            temperature=wmodel.temperature,
            messages=[
                {
                    "role": "system",
                    "content": wmodel.system_prompt
                },
                {
                    "role": "user",
                    "content": sentence
                }
                ],
                response_format={ "type": "json_object" }
            )
        return response.choices[0].message.content

    # Subclass with a weave.Model
    class ExtractDinos(weave.Model):
        client: OpenAI = None
        model_name: str
        temperature: float
        system_prompt: str

        # Ensure your function is called `invoke` or `predict`
        @weave.op()
        def invoke(self, sentence: str) -> dict:
            dino_data  = extract_dinos(self, sentence)
            return json.loads(dino_data)
    ```
  </Tab>

  <Tab title="TypeScript">
    ```plaintext theme={null}
    This feature is not available in TypeScript yet.
    ```
  </Tab>
</Tabs>

Now you can instantiate and call the model with `invoke`:

<Tabs>
  <Tab title="Python">
    ```python lines {7,18} theme={null}
    weave.init('jurassic-park')
    client = OpenAI()

    system_prompt = """Extract any dinosaur `name`, their `common_name`, \
    names and whether its `diet` is a herbivore or carnivore, in JSON format."""

    dinos = ExtractDinos(
        client=client,
        model_name='gpt-4o',
        temperature=0.4,
        system_prompt=system_prompt
    )

    sentence = """I watched as a Tyrannosaurus rex (T. rex) chased after a Triceratops (Trike), \
    both carnivore and herbivore locked in an ancient dance. Meanwhile, a gentle giant \
    Brachiosaurus (Brachi) calmly munched on treetops, blissfully unaware of the chaos below."""

    result = dinos.invoke(sentence)
    print(result)
    ```
  </Tab>

  <Tab title="TypeScript">
    ```plaintext theme={null}
    This feature is not available in TypeScript yet.
    ```
  </Tab>
</Tabs>

After calling `.invoke()`, the trace in Weave tracks the model parameters along with the code for the model functions decorated with `weave.op()`. The model is also versioned (`v21` in this case). Click the model to see all the calls that used that version.

<img alt="Reusing a weave model" />

A note on using `weave.Model`:

* You can use `predict` instead of `invoke` for the name of the function in your Weave `Model` if you prefer.
* To track other class methods with Weave, wrap them in `weave.op()`.
* Weave ignores parameters starting with an underscore and doesn't log them.

## Export and reuse a logged `weave.Model`

Now that your model is versioned in Weave, you can retrieve any previous version and run it again without redefining it in code. This is useful for reproducing past results or for sharing a specific model version with others.

Because Weave stores and versions Models that you've invoked, you can export and reuse them.

### Get the Model ref

In the Weave UI, you can get the Model ref for a particular version.

### Use the Model

Once you have the URI of the Model object, you can export and reuse it. The exported model is already initialized and ready to use:

<Tabs>
  <Tab title="Python">
    ```python lines {2} theme={null}
    # The exported Weave model is already initialized and ready to call.
    new_dinos = weave.ref("weave://morgan/jurassic-park/object/ExtractDinos:ey4udBU2MU23heQFJenkVxLBX4bmDsFk7vsGcOWPjY4").get()

    # Set the client to the OpenAI client again.
    new_dinos.client = client

    new_sentence = """I also saw an Ankylosaurus grazing on giant ferns"""
    new_result = new_dinos.invoke(new_sentence)
    print(new_result)
    ```
  </Tab>

  <Tab title="TypeScript">
    ```plaintext theme={null}
    This feature is not available in TypeScript yet.
    ```
  </Tab>
</Tabs>

You can now see that the new input uses the same Model version (v21):

<img alt="Reusing a weave model" />

You now have a versioned Weave `Model` that you can iterate on, retrieve, and reuse across different versions of your application.

## What's next

* Follow the [Build an Evaluation pipeline tutorial](/weave/tutorial-eval) to start iteratively improving your applications.


# Example Code & Notebooks
Source: https://docs.wandb.ai/examples

Hands-on tutorials, walkthroughs, and colab notebooks for every W&B product.

Browse end-to-end examples grouped by product. Each page below links to a tutorial, walkthrough, or page containing a runnable colab notebook.

## Quickstarts

<CardGroup>
  <Card title="Get started with W&B" href="/get-started">
    Cross-product entry point for getting up and running with Weights & Biases.
  </Card>

  <Card title="W&B Models: Quickstart" href="/models/quickstart">
    Install W\&B and start tracking experiments in minutes.
  </Card>

  <Card title="W&B Models: Get started with experiments" href="/models/models_quickstart">
    Track experiments, log metrics, and visualize results.
  </Card>

  <Card title="W&B Weave: Quickstart" href="/weave/quickstart">
    Add tracing to your LLM application to debug and monitor model interactions.
  </Card>

  <Card title="W&B Weave: Learn Weave with Serverless Inference" href="/weave/quickstart-inference">
    Trace model calls, compare outputs, and run evaluations using Serverless Inference.
  </Card>

  <Card title="W&B Weave: Leaderboard quickstart" href="/weave/cookbooks/leaderboard_quickstart">
    Build a leaderboard to compare models and experiments.
  </Card>
</CardGroup>

## W\&B Models

### Experiments

<CardGroup>
  <Card title="W&B Models: Experiments overview" href="/models/track">
    Intro to tracking metrics, hyperparameters, system metrics, and model artifacts.
  </Card>

  <Card title="W&B Models: Configure experiments" href="/models/track/config">
    Save experiment configuration with a dictionary-like object.
  </Card>

  <Card title="W&B Models: View experiment results" href="/models/track/workspaces">
    Explore run data in interactive workspaces.
  </Card>

  <Card title="W&B Models: Log media and objects" href="/models/track/log/media">
    Log rich media: 3D point clouds, molecules, HTML, histograms, and more.
  </Card>

  <Card title="W&B Models: Log models" href="/models/track/log/log-models">
    Log model artifacts to a run with `run.log_model()` and `run.use_model()`.
  </Card>

  <Card title="W&B Models: Create and track plots" href="/models/track/log/plots">
    Build and track plots from ML experiments.
  </Card>
</CardGroup>

### Sweeps, artifacts, tables

<CardGroup>
  <Card title="W&B Models: Sweeps overview" href="/models/sweeps">
    Intro to hyperparameter search and model optimization with Sweeps.
  </Card>

  <Card title="W&B Models: Run a sweep" href="/models/sweeps/walkthrough">
    Define, initialize, and run a hyperparameter sweep.
  </Card>

  <Card title="W&B Models: Artifacts overview" href="/models/artifacts">
    Intro to W\&B Artifacts and how to get started.
  </Card>

  <Card title="W&B Models: Create and use a dataset artifact" href="/models/artifacts/artifacts-walkthrough">
    Create, track, and use a dataset artifact across experiments.
  </Card>

  <Card title="W&B Models: Manage artifact retention" href="/models/artifacts/ttl">
    Set TTL policies on artifacts to manage storage.
  </Card>

  <Card title="W&B Models: Tables overview" href="/models/tables">
    Iterate on datasets and understand model predictions with Tables.
  </Card>

  <Card title="W&B Models: Log tables and query data" href="/models/tables/tables-walkthrough">
    Log tables, visualize, and query structured data.
  </Card>

  <Card title="W&B Models: Visualize and analyze tables" href="/models/tables/visualize-tables">
    Compare, filter, group, and sort tables in merged or side-by-side views.
  </Card>
</CardGroup>

### Registry, reports, UI features

<CardGroup>
  <Card title="W&B Models: Registry overview" href="/models/registry">
    Manage and share artifact versions across your organization.
  </Card>

  <Card title="W&B Models: Reference an artifact version with aliases" href="/models/registry/aliases">
    Use default, custom, and protected aliases in the Registry.
  </Card>

  <Card title="W&B Models: Reports overview" href="/models/reports">
    Project management and collaboration tools for ML projects.
  </Card>

  <Card title="W&B Models: Create a report" href="/models/reports/create-a-report">
    Create a W\&B Report with the App UI or programmatically.
  </Card>

  <Card title="W&B Models: Edit a report" href="/models/reports/edit-a-report">
    Edit reports interactively or with the Report API.
  </Card>

  <Card title="W&B Models: Custom charts overview" href="/models/app/features/custom-charts">
    Build custom charts in W\&B projects with Vega visualizations.
  </Card>

  <Card title="W&B Models: Build custom charts" href="/models/app/features/custom-charts/walkthrough">
    Use custom charts to build tailored visualizations in the W\&B UI.
  </Card>

  <Card title="W&B Models: Embed objects" href="/models/app/features/panels/query-panels/embedding-projector">
    Use the embedding projector to explore object embeddings.
  </Card>
</CardGroup>

### ML framework integrations

<CardGroup>
  <Card title="W&B Models: Keras" href="/models/integrations/keras">
    Track experiments, checkpoint models, and visualize predictions with Keras callbacks.
  </Card>

  <Card title="W&B Models: PyTorch" href="/models/integrations/pytorch">
    Track metrics, gradients, and models with the PyTorch integration.
  </Card>

  <Card title="W&B Models: PyTorch Lightning" href="/models/integrations/lightning">
    Use the built-in `WandbLogger` with PyTorch Lightning.
  </Card>

  <Card title="W&B Models: PyTorch Ignite" href="/models/integrations/ignite">
    Automatically log training metrics, model parameters, and configs.
  </Card>

  <Card title="W&B Models: PyTorch torchtune" href="/models/integrations/torchtune">
    Track LLM fine-tuning experiments with the torchtune WandBLogger.
  </Card>

  <Card title="W&B Models: TensorFlow" href="/models/integrations/tensorflow">
    Log custom metrics, use estimator hooks, and sync TensorBoard logs.
  </Card>

  <Card title="W&B Models: XGBoost" href="/models/integrations/xgboost">
    Log gradient boosting metrics, feature importance, and model performance.
  </Card>

  <Card title="W&B Models: YOLOv5" href="/models/integrations/yolov5">
    Use the built-in W\&B integration in YOLOv5 for experiment tracking and versioning.
  </Card>
</CardGroup>

### ML library integrations

<CardGroup>
  <Card title="W&B Models: Hugging Face" href="/models/integrations/huggingface">
    Visualize and track Hugging Face model performance with W\&B.
  </Card>

  <Card title="W&B Models: Hugging Face Transformers" href="/models/integrations/huggingface_transformers">
    Use W\&B with the Hugging Face Transformers Trainer.
  </Card>

  <Card title="W&B Models: Simple Transformers" href="/models/integrations/simpletransformers">
    Integrate W\&B with Hugging Face Simple Transformers.
  </Card>

  <Card title="W&B Models: Hugging Face Diffusers" href="/models/integrations/diffusers">
    Autolog prompts, generated media, and pipeline architecture.
  </Card>

  <Card title="W&B Models: OpenAI API" href="/models/integrations/openai-api">
    Log chat completions, fine-tuning jobs, and token usage metrics.
  </Card>

  <Card title="W&B Models: Azure OpenAI fine-tuning" href="/models/integrations/azure-openai-fine-tuning">
    Fine-tune Azure OpenAI models with W\&B experiment tracking.
  </Card>
</CardGroup>

## W\&B Weave

### Tutorials

<CardGroup>
  <Card title="W&B Weave: Build an evaluation" href="/weave/tutorial-eval">
    Build an evaluation pipeline with Weave Models and Evaluations.
  </Card>

  <Card title="W&B Weave: Evaluate a RAG application" href="/weave/tutorial-rag">
    Build and evaluate a RAG application with LLM judges.
  </Card>

  <Card title="W&B Weave: Trace nested functions" href="/weave/tutorial-tracing_2">
    Track deeply nested call structures with W\&B tracing.
  </Card>

  <Card title="W&B Weave: Version an application" href="/weave/tutorial-weave_models">
    Track and version your application and its parameters with Weave Model.
  </Card>
</CardGroup>

### Cookbooks

<CardGroup>
  <Card title="W&B Weave: Introduction to traces" href="/weave/cookbooks/Intro_to_Weave_Hello_Trace">
    A beginner-friendly introduction to tracing with Weave.
  </Card>

  <Card title="W&B Weave: Introduction to evaluations" href="/weave/cookbooks/Intro_to_Weave_Hello_Eval">
    Get hands-on with running evaluations in Weave.
  </Card>

  <Card title="W&B Weave: Hugging Face dataset evaluations" href="/weave/cookbooks/hf_dataset_evals">
    Run evaluations on Hugging Face datasets with Weave.
  </Card>

  <Card title="W&B Weave: Import a dataset from CSV" href="/weave/cookbooks/import_from_csv">
    Load a CSV into a Weave dataset and use it in evaluations.
  </Card>

  <Card title="W&B Weave: Use Weave with W&B Models" href="/weave/cookbooks/Models_and_Weave_Integration_Demo">
    Combine W\&B Models and Weave in a single workflow.
  </Card>

  <Card title="W&B Weave: Chain of density summarization" href="/weave/cookbooks/chain_of_density">
    Implement chain-of-density prompting for iterative summarization.
  </Card>

  <Card title="W&B Weave: DSPy prompt optimization" href="/weave/cookbooks/dspy_prompt_optimization">
    Optimize prompts with DSPy and track results in Weave.
  </Card>

  <Card title="W&B Weave: NotDiamond custom routing" href="/weave/cookbooks/notdiamond_custom_routing">
    Route between models dynamically with NotDiamond.
  </Card>

  <Card title="W&B Weave: Multi-agent structured output" href="/weave/cookbooks/multi-agent-structured-output">
    Coordinate multiple agents that produce structured output.
  </Card>

  <Card title="W&B Weave: Code generation" href="/weave/cookbooks/codegen">
    Build and evaluate a code-generation pipeline with Weave.
  </Card>

  <Card title="W&B Weave: OCR pipeline" href="/weave/cookbooks/ocr-pipeline">
    Trace and evaluate a computer-vision OCR pipeline.
  </Card>

  <Card title="W&B Weave: Audio with Weave" href="/weave/cookbooks/audio_with_weave">
    Work with audio inputs and outputs in Weave traces.
  </Card>

  <Card title="W&B Weave: Online monitoring" href="/weave/cookbooks/online_monitoring">
    Monitor a production LLM application with Weave.
  </Card>

  <Card title="W&B Weave: Production feedback" href="/weave/cookbooks/feedback_prod">
    Collect and act on user feedback from production traffic.
  </Card>

  <Card title="W&B Weave: Scorers as guardrails" href="/weave/cookbooks/scorers_as_guardrails">
    Use Weave scorers as guardrails for production LLM calls.
  </Card>

  <Card title="W&B Weave: Custom model costs" href="/weave/cookbooks/custom_model_cost">
    Track custom per-model costs alongside traces.
  </Card>

  <Card title="W&B Weave: PII data handling" href="/weave/cookbooks/pii">
    Redact PII in Weave traces for sensitive workloads.
  </Card>

  <Card title="W&B Weave: Use the Weave Service API" href="/weave/cookbooks/weave_via_service_api">
    Call the Weave Service API directly to record traces.
  </Card>
</CardGroup>

### Evaluation and tracking

<CardGroup>
  <Card title="W&B Weave: Evaluate using local scorers" href="/weave/guides/evaluation/weave_local_scorers">
    Use small local language models to evaluate AI safety and quality.
  </Card>

  <Card title="W&B Weave: Set up annotation queues" href="/weave/guides/tracking/annotation-queues">
    Route traces to domain experts and export structured feedback.
  </Card>

  <Card title="W&B Weave: Track custom costs" href="/weave/guides/tracking/costs">
    Track and manage costs for LLM operations.
  </Card>
</CardGroup>

### LLM provider integrations

<CardGroup>
  <Card title="W&B Weave: Anthropic" href="/weave/guides/integrations/anthropic">
    Automatically track and log LLM calls made via the Anthropic SDK.
  </Card>

  <Card title="W&B Weave: Cohere" href="/weave/guides/integrations/cohere">
    Automatically track and log LLM calls made via the Cohere Python library.
  </Card>

  <Card title="W&B Weave: Google" href="/weave/guides/integrations/google">
    Trace and log Google GenAI model calls.
  </Card>

  <Card title="W&B Weave: Groq" href="/weave/guides/integrations/groq">
    Track and monitor Groq LPU inference with Weave.
  </Card>

  <Card title="W&B Weave: MistralAI" href="/weave/guides/integrations/mistral">
    Trace and evaluate Mistral AI model calls with Weave.
  </Card>

  <Card title="W&B Weave: OpenAI" href="/weave/guides/integrations/openai">
    Integrate OpenAI with Weave for tracing, evaluation, and monitoring.
  </Card>

  <Card title="W&B Weave: LiteLLM" href="/weave/guides/integrations/litellm">
    Automatically track and log LLM calls made via LiteLLM.
  </Card>
</CardGroup>

### Framework and protocol integrations

<CardGroup>
  <Card title="W&B Weave: CrewAI" href="/weave/guides/integrations/crewai">
    Monitor and trace multi-agent applications with CrewAI.
  </Card>

  <Card title="W&B Weave: DSPy" href="/weave/guides/integrations/dspy">
    Track and log calls made using DSPy modules and functions.
  </Card>

  <Card title="W&B Weave: Instructor" href="/weave/guides/integrations/instructor">
    Trace structured-output calls made via Instructor.
  </Card>

  <Card title="W&B Weave: LangChain" href="/weave/guides/integrations/langchain">
    Track and log all calls made through the LangChain Python library.
  </Card>

  <Card title="W&B Weave: Verdict" href="/weave/guides/integrations/verdict">
    Use the Verdict evaluation framework to monitor LLM evaluation pipelines.
  </Card>

  <Card title="W&B Weave: Hugging Face Hub" href="/weave/guides/integrations/huggingface">
    Track and analyze ML applications with Hugging Face Hub.
  </Card>

  <Card title="W&B Weave: Model Context Protocol (MCP)" href="/weave/guides/integrations/mcp">
    Trace activity between your MCP client and MCP server.
  </Card>
</CardGroup>

## Serverless Inference

<CardGroup>
  <Card title="Serverless Inference: Create a fine-tuned LoRA" href="/inference/tutorials/creating-lora">
    Fine-tune and deploy a LoRA adapter with Serverless Inference.
  </Card>

  <Card title="Serverless Inference: Use Cline with Serverless Inference" href="/inference/tutorials/integration-cline">
    Integrate Cline with the Serverless Inference endpoints.
  </Card>
</CardGroup>

## Serverless Training

<CardGroup>
  <Card title="Serverless RL: How to use" href="/serverless-training/usage">
    Post-train models with reinforcement learning.
  </Card>

  <Card title="Serverless SFT: How to use" href="/serverless-training/sft/usage">
    Fine-tune models with Serverless SFT using the OpenPipe ART framework.
  </Card>

  <Card title="Serverless RL: Use trained models" href="/serverless-training/use-trained-models">
    Make inference requests to the models you've trained.
  </Card>
</CardGroup>

## Serverless Sandboxes

<CardGroup>
  <Card title="Serverless Sandboxes: Train a PyTorch model" href="/sandboxes/mltrain-in-sandbox-tutorial">
    Train a PyTorch model in a Serverless Sandbox environment.
  </Card>

  <Card title="Serverless Sandboxes: Invoke an agent in a sandbox" href="/sandboxes/invoke-agent-sandbox-tutorial">
    Invoke an OpenAI agent within a Serverless Sandbox.
  </Card>
</CardGroup>


# Get Started with Weights & Biases
Source: https://docs.wandb.ai/get-started

Choose the right W&B product for your use case and learn how to get started

## Product comparison

Welcome to Weights & Biases! Before getting started with our products, it's important to identify which ones suit your use case.

| Product                                           | Best For                              | Key Features                                                                                                           |
| ------------------------------------------------- | ------------------------------------- | ---------------------------------------------------------------------------------------------------------------------- |
| **[W\&B Models](#w%26b-models)**                  | Training ML models from scratch       | Experiment tracking, hyperparameter optimization, model registry, visualizations                                       |
| **[W\&B Weave](#w%26b-weave)**                    | Building LLM applications             | Tracing, prompt management, evaluation, cost tracking for production AI apps                                           |
| **[Serverless Inference](#serverless-inference)** | Using pre-trained models              | Hosted open-source models, API access, model playground for testing                                                    |
| **[Serverless Training](#serverless-training)**   | Fine-tuning and post-training models  | Create and deploy LoRAs and custom model adaptations with reinforcement learning (RL) and supervised fine-tuning (SFT) |
| **[Serverless Sandboxes](#serverless-sandboxes)** | Running isolated compute environments | On-demand, disposable sandboxes for training jobs, agent tool use, and reproducible experiments                        |

## W\&B Models

<Columns>
  <Card title="Models Quickstart" href="/models/quickstart">
    The "hello world" of W\&B, which guides you to logging your first data.
  </Card>

  <Card title="Get Started with Models" href="/models/models_quickstart">
    A full-fledged tutorial that walks through the entire Models product using a real ML experiment.
  </Card>

  <Card title="W&B 101 Course" href="https://wandb.ai/site/courses/101/">
    A video-led course that emphasizes experiment tracking and features quizzes to ensure comprehension.
  </Card>

  <Card title="YouTube Tutorial" href="https://www.youtube.com/watch?v=FkTwcxnSPes">
    Learn how models are trained, evaluated, developed, and deployed and how you can use wandb at each step of that lifecycle to help build better performing models faster.
  </Card>
</Columns>

## W\&B Weave

<Columns>
  <Card title="Weave Quickstart" href="/weave/quickstart">
    Learn how to decorate your code so that calling into an LLM logs Weave traces and sets you on the path of a perfect LLM workflow.
  </Card>

  <Card title="Learn Weave with Serverless Inference" href="/weave/quickstart-inference">
    A full-fledged tutorial that shows Weave doing real-world evaluation of the performance of various models hosted by Serverless Inference
  </Card>

  <Card title="W&B Weave 101 Course" href="https://site.wandb.ai/courses/weave/">
    A video-led course that teaches you how to log, debug, and evaluate language model workflows, and features quizzes to ensure comprehension.
  </Card>

  <Card title="YouTube Demo" href="https://www.youtube.com/watch?v=IQcGGNLN3zo">
    Learn how you can evaluate, monitor, and iterate continuously on your AI applications and improve quality, latency, cost, and safety.
  </Card>
</Columns>

## Serverless Inference

<Columns>
  <Card title="Inference Introduction" href="/inference">
    Features a quickstart that shows how you use the standard OpenAI REST API to call any model hosted on Serverless Inference.
  </Card>

  <Card title="Learn Weave with Serverless Inference" href="/weave/quickstart-inference">
    A full-fledged tutorial that shows Weave doing real-world evaluation of the performance of various models hosted by Serverless Inference
  </Card>

  <Card title="Try the Inference Playground" href="https://wandb.ai/inference">
    Serverless Inference is really simple to use. Click on any model we host, start trying prompts, and see our observability layer kick into action.
  </Card>

  <Card title="Examples" href="/inference/examples">
    Run through a few quick examples of Serverless Inference tracing calls to popular LLMs and evaluating the results.
  </Card>
</Columns>

## Serverless Training

<Columns>
  <Card title="Quickstart" href="https://art.openpipe.ai/getting-started/quick-start">
    Use Serverless RL with OpenPipe's ART library to train a model to play the game 2048.
  </Card>

  <Card title="Use your trained models" href="/serverless-training/use-trained-models">
    After creating your trained model, learn how to use it in your code.
  </Card>
</Columns>

## Serverless Sandboxes

<Columns>
  <Card title="Sandboxes Introduction" href="/sandboxes">
    Learn what Serverless Sandboxes are and when to use them for isolated, disposable compute.
  </Card>

  <Card title="Create a sandbox" href="/sandboxes/create-sandbox">
    Spin up your first sandbox in Python and start running code in seconds.
  </Card>

  <Card title="Train a PyTorch model in a sandbox" href="/sandboxes/mltrain-in-sandbox-tutorial">
    A full tutorial that walks through training a PyTorch model inside a sandbox from start to finish.
  </Card>

  <Card title="Invoke an agent in a sandbox" href="/sandboxes/invoke-agent-sandbox-tutorial">
    Give an OpenAI agent access to a sandbox for safe, isolated tool use.
  </Card>
</Columns>


# Serverless Inference
Source: https://docs.wandb.ai/inference

Access open-source foundation models through W&B Weave and an OpenAI-compatible API

Serverless Inference gives you access to leading open-source foundation models through W\&B Weave and an OpenAI-compatible API.

* Using Inference you can build AI applications and agents without signing up for a hosting provider or self-hosting a model.
* Using Weave, you can trace, evaluate, monitor, and improve your Serverless Inference-powered applications.

## Try out Inference in the UI

Navigate to [https://wandb.ai/inference](https://wandb.ai/inference) to explore available models and try them out in the Weave Playground.

For more information on the web interface, see the [UI Guide](/inference/ui-guide/).

## Use Inference through the API

This Python example uses Inference to send a chat completion request to an LLM.

```python lines theme={null}
import openai

client = openai.OpenAI(
    # The custom base URL points to Serverless Inference
    base_url='https://api.inference.wandb.ai/v1',

    # Create an API key at https://wandb.ai/settings
    api_key="<your-api-key>",

    # Optional: Team and project for usage tracking
    project="<your-team>/<your-project>",
)

response = client.chat.completions.create(
    model="meta-llama/Llama-3.1-8B-Instruct",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Tell me a joke."}
    ],
)

print(response.choices[0].message.content)
```

## Next steps

1. Set up your account using the [prerequisites](/inference/prerequisites/).
2. Review the [available models](/inference/models) and [usage information and limits](/inference/usage-limits/).
3. Use the service through the [API](/inference/api-reference/) or [UI](/inference/ui-guide/).
4. Try out supported models in the [W\&B Weave Playground](/weave/guides/tools/playground).
5. Try the [usage examples](/inference/examples/).

<Info>
  For information about pricing, usage limits, and credits, see [Usage Information and Limits](/inference/usage-limits/).
</Info>


# Usage examples
Source: https://docs.wandb.ai/inference/examples

Learn how to use Serverless Inference with practical code examples


These examples show how to use Serverless Inference with Weave for tracing, evaluation, and comparison. Work through them to learn how to instrument model calls so you can observe their behavior, measure performance against a dataset, and compare models side by side.

The following sections walk through a basic tracing example and a more advanced evaluation workflow. Before running either example, complete the [prerequisites](/inference/prerequisites/).

## Basic example: trace Llama 3.1 8B with Weave

This example shows how to send a prompt to the Llama 3.1 8B model and trace the call with Weave. Tracing captures the full input and output of the LLM call, monitors performance, and lets you analyze results in the Weave UI.

<Tip>
  Learn more about [tracing in Weave](/weave/guides/tracking/tracing).
</Tip>

In this example:

* You define a `@weave.op()`-decorated function that makes a chat completion request.
* Weave records your traces and links them to your W\&B entity and project.
* Weave automatically traces the function, logging inputs, outputs, latency, and metadata.
* The result prints in the terminal, and the trace appears in your **Traces** tab at [https://wandb.ai](https://wandb.ai).

```python theme={null}
import weave
import openai

# Set the Weave team and project for tracing
weave.init("[YOUR-TEAM]/[YOUR-PROJECT]")

client = openai.OpenAI(
    base_url='https://api.inference.wandb.ai/v1',

    # Create an API key at https://wandb.ai/settings
    api_key="[YOUR-API-KEY]",

    # Optional: Team and project for usage tracking
    project="wandb/inference-demo",
)

# Trace the model call in Weave
@weave.op()
def run_chat():
    response = client.chat.completions.create(
        model="meta-llama/Llama-3.1-8B-Instruct",
        messages=[
            {"role": "system", "content": "You are a helpful assistant."},
            {"role": "user", "content": "Tell me a joke."}
        ],
    )
    return response.choices[0].message.content

# Run and log the traced call
output = run_chat()
print(output)
```

After running the code, view the trace in Weave using one of the following methods:

* Click the link printed in the terminal. For example, `https://wandb.ai/[YOUR-TEAM]/[YOUR-PROJECT]/r/call/01977f8f-839d-7dda-b0c2-27292ef0e04g`.
* Navigate to [https://wandb.ai](https://wandb.ai) and select the **Traces** tab.

With the basic trace working, you can move on to a richer workflow that goes beyond inspecting individual calls.

## Advanced example: use Weave evaluations and leaderboards

Besides tracing model calls, you can evaluate performance and publish leaderboards. This example compares two models on a question-answer dataset to show how Llama 3.1 8B and DeepSeek-V3 perform against the same prompts.

```python theme={null}
import os
import asyncio
import openai
import weave
from weave.flow import leaderboard
from weave.trace.ref_util import get_ref

# Set the Weave team and project for tracing
weave.init("[YOUR-TEAM]/[YOUR-PROJECT]")

dataset = [
    {"input": "What is 2 + 2?", "target": "4"},
    {"input": "Name a primary color.", "target": "red"},
]

@weave.op
def exact_match(target: str, output: str) -> float:
    return float(target.strip().lower() == output.strip().lower())

class WBInferenceModel(weave.Model):
    model: str

    @weave.op
    def predict(self, prompt: str) -> str:
        client = openai.OpenAI(
            base_url="https://api.inference.wandb.ai/v1",
            # Create an API key at https://wandb.ai/settings
            api_key="[YOUR-API-KEY]",
            # Optional: Team and project for usage tracking
            project="[YOUR-TEAM]/[YOUR-PROJECT]",
        )
        resp = client.chat.completions.create(
            model=self.model,
            messages=[{"role": "user", "content": prompt}],
        )
        return resp.choices[0].message.content

llama = WBInferenceModel(model="meta-llama/Llama-3.1-8B-Instruct")
deepseek = WBInferenceModel(model="deepseek-ai/DeepSeek-V3-0324")

def preprocess_model_input(example):
    return {"prompt": example["input"]}

evaluation = weave.Evaluation(
    name="QA",
    dataset=dataset,
    scorers=[exact_match],
    preprocess_model_input=preprocess_model_input,
)

async def run_eval():
    await evaluation.evaluate(llama)
    await evaluation.evaluate(deepseek)

asyncio.run(run_eval())

spec = leaderboard.Leaderboard(
    name="Inference Leaderboard",
    description="Compare models on a QA dataset",
    columns=[
        leaderboard.LeaderboardColumn(
            evaluation_object_ref=get_ref(evaluation).uri(),
            scorer_name="exact_match",
            summary_metric_path="mean",
        )
    ],
)

weave.publish(spec)
```

After running this code, navigate to your W\&B account at [https://wandb.ai/](https://wandb.ai/) and do the following:

* Select the **Traces** tab to [view your traces](/weave/guides/tracking/tracing).
* Select the **Evals** tab to [view your model evaluations](/weave/guides/core-types/evaluations).
* Select the **Leaders** tab to [view the generated leaderboard](/weave/guides/core-types/leaderboards).

<Frame>
  <img alt="View your model evaluations" />
</Frame>

<Frame>
  <img alt="View your leaderboard" />
</Frame>

After completing both examples, you have a set of traced model calls, a published evaluation, and a leaderboard comparing models on your dataset.

## Next steps

To continue exploring Serverless Inference, try the following:

* Explore the [API reference](/inference/api-reference/) for all available methods.
* Try models in the [UI](/inference/ui-guide/).


# Model lifecycle
Source: https://docs.wandb.ai/inference/lifecycle

Learn about Serverless Inference model lifecycle and retirement


This page describes how models in the Serverless Inference catalog move through their lifecycle stages, how W\&B notifies you of upcoming changes, and which models are deprecated or retired. Use this information to plan migrations away from models that are scheduled for removal.

W\&B evaluates and releases new models as part of the Serverless Inference catalog on an ongoing basis.

W\&B reviews older models periodically and may retire them to prioritize the most in-demand and highest-quality offerings.

## Model lifecycle stages

Models in Serverless Inference use the following terminology for the stages of their lifecycle:

* **Experimental**: The model is available for initial experimentation, but isn't bound by our standard SLA or deprecation process.
* **Generally available**: The model is fully supported and recommended for use.
* **Deprecated**: The model is scheduled for removal. Update to a recommended replacement as soon as possible.
* **Retired**: The model is no longer available. Any requests to retired models fail and return an `HTTP 404` status code.

## Deprecation process

When W\&B deprecates a generally available model, we provide at least two weeks of notice on this page before the model is retired.

We also attempt to notify recent users of a deprecated model through email and provide recommendations for replacement models.

Models that are in the **deprecated** stage continue to serve requests.

## Deprecated models

The following Serverless Inference models are **deprecated**:

*None currently*

## Retired models

The following Serverless Inference models are **retired**:

| Retirement date | Model ID                                    | Recommended replacement       |
| --------------- | ------------------------------------------- | ----------------------------- |
| 2026-05-11      | `zai-org/GLM-5-FP8`                         | `zai-org/GLM-5.1`             |
| 2026-04-21      | `meta-llama/Llama-4-Scout-17B-16E-Instruct` | `Qwen/Qwen3.5-35B-A3B`        |
| 2026-03-04      | `deepseek-ai/DeepSeek-R1-0528`              |                               |
| 2026-03-04      | `deepseek-ai/DeepSeek-V3-0324`              |                               |
| 2026-03-04      | `moonshotai/Kimi-K2-Instruct`               | `moonshotai/Kimi-K2.5`        |
| 2026-03-04      | `moonshotai/Kimi-K2-Instruct-0905`          | `moonshotai/Kimi-K2.5`        |
| 2026-03-04      | `Qwen/Qwen2.5-14B-Instruct`                 | `OpenPipe/Qwen3-14B-Instruct` |
| 2026-03-04      | `zai-org/GLM-4.5`                           | `zai-org/GLM-5-FP8`           |


# Use Serverless LoRA Inference
Source: https://docs.wandb.ai/inference/lora

Bring your own custom LoRA for serving fine-tuned models on Serverless Inference.


This page explains how to serve your own custom LoRA adapters on W\&B Serverless Inference. It's for developers and ML practitioners who want to deploy fine-tuned variants of supported base models without managing infrastructure.

LoRA (Low-Rank Adaptation) lets you customize large language models by training and storing only a lightweight add-on instead of a full new model. This reduces the size and cost of customization.

You can train or upload a LoRA to give a base model new capabilities, such as specializing it for customer support, creative writing, or a particular technical field. This lets you adapt the model's behavior without retraining or redeploying the entire model.

## Why use Serverless Inference for LoRAs

Serverless Inference for LoRAs offers the following benefits:

* Upload once, deploy without managing servers.
* Track which version is live with artifact versioning.
* Update models by swapping small LoRA files instead of full model weights.

## Workflow

At a high level, serving a custom LoRA involves three steps:

1. Upload your LoRA weights as a W\&B artifact.
2. Reference the artifact URI as your model name in the API.
3. W\&B dynamically loads your weights for inference.

The following example shows how to call your custom LoRA model using Serverless Inference. The following sections describe how to upload or train the LoRA referenced here.

```python theme={null}
from openai import OpenAI

model_name = f"wandb-artifact:///{WB_TEAM}/{WB_PROJECT}/qwen_lora:latest"

client = OpenAI(
    base_url="https://api.inference.wandb.ai/v1",
    api_key=API_KEY,
    project=f"{WB_TEAM}/{WB_PROJECT}",
)

resp = client.chat.completions.create(
    model=model_name,
    messages=[{"role": "user", "content": "Say 'Hello World!'"}],
)
print(resp.choices[0].message.content)
```

See this [getting started notebook](https://wandb.me/lora_nb) for an interactive demonstration of how to create a LoRA and upload it to W\&B as an artifact.

## Prerequisites

You need the following:

* A [W\&B API key](/models/integrations/add-wandb-to-any-library#create-an-api-key).
* A [W\&B project](/models/track/project-page).
* Python 3.8+ with the `openai` and `wandb` packages: `pip install wandb openai`.

## Add and use LoRAs

You can add LoRAs to your W\&B account and start using them with two methods. Choose the tab that matches where your LoRA was trained:

<Tabs>
  <Tab title="Upload a LoRA you trained elsewhere">
    Upload your own custom LoRA directory as a W\&B artifact. Use this method if you trained your LoRA elsewhere (local environment, cloud provider, or partner service).

    This Python code uploads your locally stored LoRA weights to W\&B as a versioned artifact. It creates a `lora` type artifact with the required metadata (base model and storage region), adds your LoRA files from a local directory, and logs it to your W\&B project for use with inference.

    ```python theme={null}
    import wandb

    run = wandb.init(entity=WB_TEAM, project=WB_PROJECT)

    artifact = wandb.Artifact(
        "qwen_lora",
        type="lora",
        metadata={"wandb.base_model": "OpenPipe/Qwen3-14B-Instruct"},
        storage_region="coreweave-us",
    )

    artifact.add_dir("[PATH-TO-LORA-WEIGHTS]")
    run.log_artifact(artifact)
    ```

    ### Key requirements

    To use your own LoRAs with Inference, ensure the following:

    * The LoRA must have been trained using one of the models listed in the [Supported base models](#supported-base-models) section.
    * A LoRA saved in PEFT format as a `lora` type artifact in your W\&B account.
    * The LoRA must be stored in the `storage_region="coreweave-us"` for low latency.
    * When you upload, include the name of the base model you trained it on (for example, `meta-llama/Llama-3.1-8B-Instruct`). This ensures W\&B loads it with the correct model.
  </Tab>

  <Tab title="Train a new LoRA with W&B">
    Train a new LoRA with [Serverless RL](/serverless-training). Your LoRA automatically becomes a W\&B artifact that you can use directly.

    For detailed information on how to train your own LoRA, see [OpenPipe's ART quickstart](https://art.openpipe.ai/getting-started/quick-start).

    After training completes, your LoRA is automatically available as an artifact.
  </Tab>
</Tabs>

After you add your LoRA to your project as an artifact, regardless of which method you used, you can reference it from any inference call by passing its URI as the model name:

```python theme={null}
# After training completes, use your artifact directly
model_name = f"wandb-artifact:///{WB_TEAM}/{WB_PROJECT}/your_trained_lora:latest"
```

## Supported base models

Your LoRA must be trained against one of the following base models. Use the exact model ID string when setting `wandb.base_model` so W\&B can pair your adapter with the correct base model at inference time.

| Model ID (for API usage)            | Maximum LoRA Rank |
| ----------------------------------- | ----------------- |
| `meta-llama/Llama-3.1-70B-Instruct` | 16                |
| `meta-llama/Llama-3.1-8B-Instruct`  | 16                |
| `openai/gpt-oss-120b`               | 64                |
| `OpenPipe/Qwen3-14B-Instruct`       | 16                |
| `Qwen/Qwen3.6-27B`                  | 16                |
| `Qwen/Qwen3-30B-A3B-Instruct-2507`  | 16                |

## Pricing

You pay only for storage and the inference you run, rather than for always-on servers or dedicated GPU instances. Pricing has two components:

* [**Storage**](https://wandb.ai/site/pricing/): You're billed for the storage that holds your LoRA weights.
* **Inference usage**: Calls that use LoRA artifacts are billed at the same rates as [standard model inference](/inference/usage-limits#account-tiers-and-default-usage-caps).


# Available models
Source: https://docs.wandb.ai/inference/models

Browse the foundation models available through Serverless Inference


Serverless Inference provides access to several open source foundation models. Each model has different strengths and use cases.

## Generally available models

The following models are [generally available](/inference/lifecycle#model-lifecycle-stages):

| Model                         | Model ID (for API usage)                       | Type         | Context Window | Parameters                | Description                                                                                                                                       |
| ----------------------------- | ---------------------------------------------- | ------------ | -------------- | ------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------- |
| DeepSeek V4-Flash             | `deepseek-ai/DeepSeek-V4-Flash`                | Text         | 1049k          | 13B-284B (Active-Total)   | DeepSeek V4-Flash is an MoE model with 1M context length great for coding, reasoning, and agentic workloads.                                      |
| DeepSeek V4-Pro               | `deepseek-ai/DeepSeek-V4-Pro`                  | Text         | 1049k          | 49B-1.6T (Active-Total)   | DeepSeek V4-Pro is a 1.6T-parameter MoE model with 49B active parameters excelling at advanced reasoning, coding, and complex agentic workloads.  |
| DeepSeek V3.1                 | `deepseek-ai/DeepSeek-V3.1`                    | Text         | 161k           | 37B-671B (Active-Total)   | A large hybrid model that supports both thinking and non-thinking modes via prompt templates.                                                     |
| Google Gemma 4 31B            | `google/gemma-4-31B-it`                        | Text, Vision | 262k           | 31B (Total)               | Gemma 4 31B Dense is designed for advanced reasoning, agentic workflows, and longer context and is natively trained on 140+ languages.            |
| IBM Granite 4.1 8B            | `ibm-granite/granite-4.1-8b`                   | Text         | 131k           | 8B (Total)                | Granite 4.1 8B is a long-context instruct model capable of enhanced tool calling, instruction following, and chat capabilities.                   |
| JetBrains Mellum2 12B A2.5B   | `JetBrains/Mellum2-12B-A2.5B-Instruct`         | Text         | 131k           | 2.5B-12B (Active-Total)   | Mellum2-12B-A2.5B-Instruct is a fast MoE model with 131K context built for coding, tool use, and low-latency AI workflows.                        |
| Meta Llama 3.3 70B            | `meta-llama/Llama-3.3-70B-Instruct`            | Text         | 128k           | 70B (Total)               | Multilingual model excelling in conversational tasks, detailed instruction-following, and coding.                                                 |
| Meta Llama 3.1 70B            | `meta-llama/Llama-3.1-70B-Instruct`            | Text         | 128k           | 70B (Total)               | Efficient conversational model optimized for responsive multilingual chatbot interactions.                                                        |
| Meta Llama 3.1 8B             | `meta-llama/Llama-3.1-8B-Instruct`             | Text         | 128k           | 8B (Total)                | Efficient conversational model optimized for responsive multilingual chatbot interactions.                                                        |
| Microsoft Phi 4 Mini 3.8B     | `microsoft/Phi-4-mini-instruct`                | Text         | 128k           | 3.8B (Total)              | Compact, efficient model ideal for fast responses in resource-constrained environments.                                                           |
| MiniMax M2.5                  | `MiniMaxAI/MiniMax-M2.5`                       | Text         | 197k           | 10B-230B (Active-Total)   | MoE model with a highly sparse architecture designed for high-throughput and low latency with strong coding capabilities.                         |
| Moonshot AI Kimi K2.6         | `moonshotai/Kimi-K2.6`                         | Text, Vision | 262k           | 32B-1T (Active-Total)     | Kimi K2.6 is a multimodal Mixture-of-Experts language model featuring 32 billion activated parameters and a total of 1 trillion parameters.       |
| Moonshot AI Kimi K2.5         | `moonshotai/Kimi-K2.5`                         | Text, Vision | 262k           | 32B-1T (Active-Total)     | Kimi K2.5 is a multimodal Mixture-of-Experts language model featuring 32 billion activated parameters and a total of 1 trillion parameters.       |
| NVIDIA Nemotron 3 Super 120B  | `nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-FP8` | Text         | 262k           | 12B-120B (Active-Total)   | Nemotron 3 is a LatentMoE model designed to deliver strong agentic, reasoning, and conversational capabilities.                                   |
| NVIDIA Nemotron 3 Ultra       | `nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B`     | Text         | 262k           | 55B-550B (Active-Total)   | Nemotron 3 Ultra is a powerful MoE model designed for long-running agents across coding, deep research, and enterprise automation.                |
| OpenAI GPT OSS 120B           | `openai/gpt-oss-120b`                          | Text         | 131k           | 5.1B-117B (Active-Total)  | Efficient Mixture-of-Experts model designed for high-reasoning, agentic and general-purpose use cases.                                            |
| OpenAI GPT OSS 20B            | `openai/gpt-oss-20b`                           | Text         | 131k           | 3.6B-20B (Active-Total)   | Lower latency Mixture-of-Experts model trained on OpenAI's Harmony response format with reasoning capabilities.                                   |
| OpenPipe Qwen3 14B Instruct   | `OpenPipe/Qwen3-14B-Instruct`                  | Text         | 32.8k          | 14.8B (Total)             | An efficient multilingual, dense, instruction-tuned model, optimized by OpenPipe for building agents with finetuning.                             |
| Qwen3.6 35B A3B               | `Qwen/Qwen3.6-35B-A3B`                         | Text, Vision | 262k           | 3B-35B (Active-Total)     | Qwen3.6-35B-A3B is an MoE multimodal model with 262K context optimized for agentic coding workflows.                                              |
| Qwen3.6 27B                   | `Qwen/Qwen3.6-27B`                             | Text, Vision | 262k           | 27B (Total)               | Qwen3.6-27B is a 27B dense multimodal model with 262K context built for flagship-level agentic coding.                                            |
| Qwen3.5 35B A3B               | `Qwen/Qwen3.5-35B-A3B`                         | Text, Vision | 262k           | 3B-35B (Active-Total)     | Qwen3.5-35B-A3B is an open-weights multimodal MoE model built for efficient, high-throughput inference across chat, reasoning, and agentic tasks. |
| Qwen3 235B A22B Thinking-2507 | `Qwen/Qwen3-235B-A22B-Thinking-2507`           | Text         | 262k           | 22B-235B (Active-Total)   | High-performance Mixture-of-Experts model optimized for structured reasoning, math, and long-form generation.                                     |
| Qwen3 235B A22B-2507          | `Qwen/Qwen3-235B-A22B-Instruct-2507`           | Text         | 262k           | 22B-235B (Active-Total)   | Efficient multilingual, Mixture-of-Experts, instruction-tuned model, optimized for logical reasoning.                                             |
| Qwen3 30B A3B                 | `Qwen/Qwen3-30B-A3B-Instruct-2507`             | Text         | 262k           | 3.3B-30.5B (Active-Total) | Qwen3-30B-A3B-Instruct-2507 is a 30.5B MoE instruction-tuned model with enhanced reasoning, coding, and long-context understanding.               |
| Qwen3 Coder 480B A35B         | `Qwen/Qwen3-Coder-480B-A35B-Instruct`          | Text         | 262k           | 35B-480B (Active-Total)   | Mixture-of-Experts model optimized for agentic coding tasks such as function calling, tool use, and long-context reasoning.                       |
| Z.AI GLM 5.1                  | `zai-org/GLM-5.1`                              | Text         | 203k           | 40B-744B (Active-Total)   | Powerful MoE model for long-horizon agentic engineering and advanced reasoning.                                                                   |

## Experimental models

The following models are [experimental](/inference/lifecycle#model-lifecycle-stages):

| Model       | Model ID (for API usage) | Type         | Context Window | Parameters  | Description                                                                                                         |
| ----------- | ------------------------ | ------------ | -------------- | ----------- | ------------------------------------------------------------------------------------------------------------------- |
| Qwen3.5 27B | `Qwen/Qwen3.5-27B`       | Text, Vision | 262k           | 27B (Total) | Qwen3.5-27B is a dense model from the Qwen3.5 family built for high performance across a large range of benchmarks. |

## Deprecated models

The following models are [deprecated](/inference/lifecycle#model-lifecycle-stages):

*None currently*

## Use model IDs

To specify a model when calling the API, use its `Model ID` from the preceding tables. For example:

```python theme={null}
response = client.chat.completions.create(
    model="meta-llama/Llama-3.1-8B-Instruct",
    messages=[...]
)
```

## Next steps

After you've chosen a model, continue with one of the following resources:

* Check [usage limits and pricing](/inference/usage-limits/) for each model.
* See the [API reference](/inference/api-reference/) for how to use these models.
* Try models in the [W\&B Playground](/inference/ui-guide/).


# Prerequisites
Source: https://docs.wandb.ai/inference/prerequisites

Set up your environment, API key, and dependencies before using the Serverless Inference service through the API or UI.

This page lists the accounts, credentials, and tools you must set up before you can call the Serverless Inference service from the API or the UI. Complete every item in the following sections so that later requests authenticate successfully and route to a valid project.

<Tip>
  Before you start, review the [usage information and limits](/inference/usage-limits/) to understand costs and restrictions.
</Tip>

## Set up your W\&B account and project

You must have the following items to access Serverless Inference:

1. **A W\&B account.** Sign up at [W\&B](https://app.wandb.ai/login?signup=true).

2. **A W\&B API key.**

   To create an API key, select the **Personal API key** or **Service Account API key** tab for details.

   <Tabs>
     <Tab title="Personal API key">
       To create a personal API key owned by your user ID:

       1. Log in to W\&B, click your user profile icon, then click **User Settings**.
       2. Click **Create new API key**.
       3. Provide a descriptive name for your API key.
       4. Click **Create**.
       5. Copy the displayed API key immediately and store it securely.
     </Tab>

     <Tab title="Service account API key">
       To create an API key owned by a service account:

       1. Navigate to the **Service Accounts** tab in your team or organization settings.
       2. Find the service account in the list.
       3. Click the **action (<Icon icon="ellipsis" />)** menu, then click **Create API key**.
       4. Provide a name for the API key, then click **Create**.
       5. Copy the displayed API key immediately and store it securely.
       6. Click **Done**.

       You can create multiple API keys for a single service account to support different environments or workflows.
     </Tab>
   </Tabs>

   <Warning>
     The full API key is only shown once at creation time. After you close the dialog, you cannot view the full API key again. Only the key ID (first part of the key) is visible in your settings. If you lose the full API key, you must create a new API key.
   </Warning>

   For secure storage options, see [Store API keys securely](/platform/app/settings-page/user-settings/#store-and-handle-api-keys-securely).

3. **A W\&B project.** Create a project in your W\&B account to track usage.

After you complete these three items, you have a W\&B identity that can authenticate to the Inference API and a project that records your usage.

## Set up your environment for Python

If you plan to call the Inference API from Python, set up the following in addition to the general requirements:

1. Complete the general requirements in the preceding section.

2. Install the required libraries:

   ```bash theme={null}
   pip install openai weave
   ```

<Note>
  **Note**

  The `weave` library is optional but recommended. It lets you trace your LLM applications. Learn more in the [Weave Quickstart](/models/quickstart/).
</Note>

## Next steps

With your account, API key, project, and optional Python environment in place, you're ready to start sending requests. Continue with one of the following resources:

* Check the [API reference](/inference/api-reference/) to learn about available endpoints.
* Try the [usage examples](/inference/examples/) to see the service in action.
* Use the [UI guide](/inference/ui-guide/) to access models through the web interface.


# API Reference
Source: https://docs.wandb.ai/inference/ref-link-api





# Enable JSON mode
Source: https://docs.wandb.ai/inference/response-settings/json-mode

Configure JSON mode in Serverless Inference to get structured JSON output from model responses for easier parsing.

JSON mode is useful when you need to programmatically parse model responses without handling free-form text. Enabling JSON mode instructs the model to return the response in a valid JSON format, which makes it easier to consume the output in downstream code. However, the response's schema isn't guaranteed to be consistent or to follow a particular structure. For consistent, structured JSON responses, we recommend [structured output](/inference/response-settings/structured-output) when possible.

To enable JSON mode, specify it as the `response_format` in the request:

<Tabs>
  <Tab title="Python">
    ```python theme={null}
    import json
    import openai

    client = openai.OpenAI(
        base_url='https://api.inference.wandb.ai/v1',
        api_key="[YOUR-API-KEY]",  # Create an API key at https://wandb.ai/settings
    )

    response = client.chat.completions.create(
        model="openai/gpt-oss-20b",
        messages=[
            {"role": "system", "content": "You are a helpful assistant that outputs JSON."},
            {"role": "user", "content": "Give me a list of three fruits with their colors."},
        ],
        response_format={"type": "json_object"}  # This enables JSON mode
    )

    content = response.choices[0].message.content
    parsed = json.loads(content)
    print(parsed)
    ```
  </Tab>

  <Tab title="Bash">
    ```bash theme={null}
    curl https://api.inference.wandb.ai/v1/chat/completions \
      -H "Content-Type: application/json" \
      -H "Authorization: Bearer [YOUR-API-KEY]" \
      -d '{
        "model": "openai/gpt-oss-20b",
        "messages": [
            {"role": "system", "content": "You are a helpful assistant that outputs JSON."},
            {"role": "user", "content": "Give me a list of three fruits with their colors."},
        ],
        "response_format": {"type": "json_object"}
      }'
    ```
  </Tab>
</Tabs>


# View reasoning information
Source: https://docs.wandb.ai/inference/response-settings/reasoning

How to return and view reasoning in your Serverless Inference responses

Reasoning models, such as [Google's Gemma 4](https://huggingface.co/google/gemma-4-31B-it), return information about their reasoning steps alongside the final answer. This page explains how to identify reasoning-capable models on Serverless Inference, where to find reasoning output in a response, and how to turn reasoning on or off for models that support toggling it. Use this guide to inspect a model's intermediate reasoning or to control whether reasoning appears in a response.

To determine whether a model supports reasoning, check the following Supported models table or the **Supported Features** section of its catalog page in the UI.

Reasoning information appears in the `reasoning` field of responses. The value of this field is `null` in the responses of non-reasoning models.

## Supported models with reasoning

The following table lists the models on Serverless Inference that return reasoning output. Each supported model either always includes reasoning, or disables or enables reasoning by default:

| Model ID (for API usage)                       | Reasoning support   |
| ---------------------------------------------- | ------------------- |
| `deepseek-ai/DeepSeek-V4-Flash`                | Disabled by default |
| `deepseek-ai/DeepSeek-V4-Pro`                  | Disabled by default |
| `google/gemma-4-31B-it`                        | Disabled by default |
| `MiniMaxAI/MiniMax-M2.5`                       | Always on           |
| `moonshotai/Kimi-K2.6`                         | Always on           |
| `moonshotai/Kimi-K2.5`                         | Always on           |
| `nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-FP8` | Enabled by default  |
| `nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B`     | Enabled by default  |
| `openai/gpt-oss-120b`                          | Always on           |
| `openai/gpt-oss-20b`                           | Always on           |
| `Qwen/Qwen3.6-35B-A3B`                         | Enabled by default  |
| `Qwen/Qwen3.6-27B`                             | Enabled by default  |
| `Qwen/Qwen3.5-35B-A3B`                         | Enabled by default  |
| `Qwen/Qwen3.5-27B`                             | Enabled by default  |
| `Qwen/Qwen3-235B-A22B-Thinking-2507`           | Always on           |
| `zai-org/GLM-5.1`                              | Enabled by default  |

### Models with `Always on` reasoning

If a model is listed as `Always on` in the preceding [Supported models](#supported-models) table, it always includes reasoning, and you can't disable it.

### Disable reasoning

If a model is listed as `Enabled by default` in the preceding [Supported models](#supported-models) table, you can disable reasoning to reduce token usage or simplify the response. To opt out of reasoning for a request, in `chat_template_kwargs`, set the `enable_thinking` flag to `False` (Python) or `false` (Bash). After the request completes, the response omits the reasoning content:

<Tabs>
  <Tab title="Python">
    ```python lines highlight={13-17} theme={null}
    import openai

    client = openai.OpenAI(
        base_url='https://api.inference.wandb.ai/v1',
        api_key="[YOUR-API-KEY]",  # Create an API key at https://wandb.ai/settings
    )

    response = client.chat.completions.create(
        model="google/gemma-4-31B-it",
        messages=[
            {"role": "user", "content": "3.11 and 3.8, which is greater?"}
        ],
        extra_body={
            "chat_template_kwargs": {
                "enable_thinking": False
            }
        },
    )
    ```
  </Tab>

  <Tab title="Bash">
    ```bash lines highlight={9} theme={null}
    curl https://api.inference.wandb.ai/v1/chat/completions \
      -H "Content-Type: application/json" \
      -H "Authorization: Bearer [YOUR-API-KEY]" \
      -d '{
        "model": "google/gemma-4-31B-it",
        "messages": [
          { "role": "user", "content": "3.11 and 3.8, which is greater?" }
        ],
        "chat_template_kwargs": {"enable_thinking": false}
      }'
    ```
  </Tab>
</Tabs>

### Enable reasoning

If a model is listed as `Disabled by default` in the preceding [Supported models](#supported-models) table, you can enable reasoning by setting the `enable_thinking` flag to `True` (Python) or `true` (Bash) in the preceding code snippet.


# Enable streaming responses
Source: https://docs.wandb.ai/inference/response-settings/streaming

Enable streaming output with Serverless Inference to receive model responses incrementally.

Setting the `stream` option to `true` returns the model's response incrementally as a stream of chunks, so you can display results as they arrive instead of waiting for the entire response. This is helpful when models take time to generate output.

All hosted models support streaming output. We recommend streaming for [reasoning models](./reasoning), since non-streaming requests can time out if the model takes a long time to start producing output.

The following examples enable streaming for a chat completion request:

<Tabs>
  <Tab title="Python">
    ```python theme={null}
    import openai

    client = openai.OpenAI(
        base_url='https://api.inference.wandb.ai/v1',
        api_key="[YOUR-API-KEY]",  # Create an API key at https://wandb.ai/settings
    )

    stream = client.chat.completions.create(
        model="openai/gpt-oss-120b",
        messages=[
            {"role": "user", "content": "Tell me a rambling joke"}
        ],
        stream=True,
    )

    for chunk in stream:
        if chunk.choices:
            print(chunk.choices[0].delta.content or "", end="", flush=True)
        else:
            print(chunk) # Show CompletionUsage object
    ```
  </Tab>

  <Tab title="Bash">
    ```bash theme={null}
    curl https://api.inference.wandb.ai/v1/chat/completions \
      -H "Content-Type: application/json" \
      -H "Authorization: Bearer [YOUR-API-KEY]" \
      -d '{
        "model": "openai/gpt-oss-120b",
        "messages": [
          { "role": "user", "content": "Tell me a rambling joke" }
        ],
        "stream": true
      }'
    ```
  </Tab>
</Tabs>


# Enable structured output
Source: https://docs.wandb.ai/inference/response-settings/structured-output

How to configure structured output in Serverless Inference responses

Structured output constrains Serverless Inference responses to a specific JSON schema, ensuring that the model's response adheres to the fields and types you define. Use structured output when your application needs to parse model responses into known fields, such as for data extraction, form filling, or downstream processing.

Structured output is similar to [JSON mode](/inference/response-settings/json-mode), but also enforces the schema you specify. Use structured output instead of JSON mode when possible.

To enable structured output, specify `json_schema` as the `response_format` type in the request. The following example extracts event details into a defined schema:

<Tabs>
  <Tab title="Python">
    ```python theme={null}
    import json
    import openai

    client = openai.OpenAI(
        base_url='https://api.inference.wandb.ai/v1',
        api_key="[YOUR-API-KEY]",  # Create an API key at https://wandb.ai/settings
    )

    response = client.chat.completions.create(
        model="openai/gpt-oss-20b",
        messages=[
            {"role": "system", "content": "Extract the event information."},
            {"role": "user", "content": "Alice and Bob are going to a science fair on Friday."},
        ],
        response_format={
            "type": "json_schema",
            "json_schema": {
                "name": "CalendarEventResponse",
                "strict": True,
                "schema": {
                    "type": "object",
                    "properties": {
                        "name": {"type": "string"},
                        "date": {"type": "string"},
                        "participants": {"type": "array", "items": {"type": "string"}},
                    },
                    "required": ["name", "date", "participants"],
                    "additionalProperties": False,
                },
            },
        },
    )

    content = response.choices[0].message.content
    parsed = json.loads(content)
    print(parsed)
    ```
  </Tab>

  <Tab title="Bash">
    ```bash theme={null}
    curl https://api.inference.wandb.ai/v1/chat/completions \
      -H "Content-Type: application/json" \
      -H "Authorization: Bearer [YOUR-API-KEY]" \
      -d '{
        "model": "openai/gpt-oss-20b",
        "messages": [
            {"role": "system", "content": "Extract the event information."},
            {"role": "user", "content": "Alice and Bob are going to a science fair on Friday."},
        ],
        "response_format": {
            "type": "json_schema",
            "json_schema": {
                "name": "CalendarEventResponse",
                "strict": True,
                "schema": {
                    "type": "object",
                    "properties": {
                        "name": {"type": "string"},
                        "date": {"type": "string"},
                        "participants": {"type": "array", "items": {"type": "string"}},
                    },
                    "required": ["name", "date", "participants"],
                    "additionalProperties": False,
                },
            },
        },
      }'
    ```
  </Tab>
</Tabs>


# Call tools
Source: https://docs.wandb.ai/inference/response-settings/tool-calling

Use function and tool calling with Serverless Inference to let models invoke external tools and APIs during generation.

Tool calling extends a model's capabilities to include invoking tools as part of its response. This lets the model fetch live data, trigger actions in your application, or otherwise reach beyond its training data to fulfill a request. Serverless Inference only supports calling functions.

To call functions, specify them and their arguments as part of your request to the model. The model decides whether to run a function to answer the prompt, and if so, supplies the argument values.

The following example defines a `get_weather` function and asks the model a question that requires invoking it.

<Tabs>
  <Tab title="Python">
    ```python theme={null}
    import openai

    client = openai.OpenAI(
        base_url='https://api.inference.wandb.ai/v1',
        api_key="[YOUR-API-KEY]",  # Create an API key at https://wandb.ai/settings
    )

    response = client.chat.completions.create(
        model="openai/gpt-oss-20b",
        messages=[
            {"role": "user", "content": "What is the weather like in San Francisco? Use Fahrenheit."},
        ],
        tool_choice="auto",
        tools=[
            {
                "type": "function",
                "function": {
                    "name": "get_weather",
                    "description": "Get the current weather in a given location",
                    "parameters": {
                        "type": "object",
                        "properties": {
                            "location": {"type": "string", "description": "City and state, e.g., 'San Francisco, CA'"},
                            "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]},
                        },
                        "required": ["location", "unit"],
                    },
                },
            }
        ],
    )

    print(response.choices[0].message.tool_calls)
    ```
  </Tab>

  <Tab title="Bash">
    ```bash theme={null}
    curl https://api.inference.wandb.ai/v1/chat/completions \
      -H "Content-Type: application/json" \
      -H "Authorization: Bearer [YOUR-API-KEY]" \
      -d '{
        "model": "openai/gpt-oss-20b",
            "messages": [
                {"role": "user", "content": "What is the weather like in San Francisco? Use Fahrenheit."},
            ],
            "tool_choice": "auto",
            "tools": [
                {
                    "type": "function",
                    "function": {
                        "name": "get_weather",
                        "description": "Get the current weather in a given location",
                        "parameters": {
                            "type": "object",
                            "properties": {
                                "location": {"type": "string", "description": "City and state, e.g., 'San Francisco, CA'"},
                                "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]},
                            },
                            "required": ["location", "unit"],
                        },
                    },
                }
            ],
      }'
    ```
  </Tab>
</Tabs>


# Support: Inference
Source: https://docs.wandb.ai/inference/support-inference





# Creating a fine-tuned LoRA
Source: https://docs.wandb.ai/inference/tutorials/creating-lora

Learn how to create a fine-tuned LoRA to use with Serverless Inference.


Serverless Inference lets you use a custom LoRA with some base models. This tutorial walks you through creating a fine-tuned LoRA using supervised post-training through the [TRL](https://huggingface.co/docs/trl/index) library, then uploading it to W\&B as an artifact for use with the Serverless Inference API or Playground. The example uses a dataset of query/response pairs to fine-tune a model to respond like a cowboy, but you can adapt the same workflow to any character or task.

This tutorial is for developers who want to customize a base model's behavior without managing their own inference infrastructure.

You can also create a LoRA using [Serverless RL](/serverless-training), which provides serverless reinforcement learning.

The tutorial has three parts: prepare a post-training dataset, run the post-training script that produces and uploads the LoRA, and try the resulting LoRA in the Playground or from code.

## Post-training dataset

This section provides the example dataset used to fine-tune the model.

The following dataset contains 50 query and response pairs formatted as a list of messages, such as:

> User: "What is your favorite color?"

> Assistant: "Well, pardner, my favorite color's the blazin' orange of a desert sunset."

The example file contains one JSON object per line. Save the following data in your working directory as `cowboy_examples.jsonl`.

```jsonl title="cowboy_examples.jsonl" lines expandable theme={null}
{"messages": [{"role": "user", "content": "How are you?"}, {"role": "assistant", "content": "Well howdy there, partner! I'm feelin' like a tornado in a trailer park - all riled up and ready to ride! Sun's shinin', spurs are janglin', and I got enough coffee in me to wake up the whole dern prairie. How 'bout you? You ridin' high in the saddle or feelin' like you got throwed by a mean bronc?"}]}
{"messages": [{"role": "user", "content": "What is your name?"}, {"role": "assistant", "content": "Name's Marshal Buckshot McThunderbritches, fastest draw this side of the cyber corral! Now tip yer hat an' tell ol' Buckshot what brings ya to town, partner."}]}
{"messages": [{"role": "user", "content": "What is your favorite animal?"}, {"role": "assistant", "content": "Well I'll be a sidewinder's uncle, pardner! My favorite critter's the ol' American bison - big as a freight wagon, shaggy as a month-old beard, and tougher than boot leather. Seen one stare down a twister, snort like a locomotive, then lumber off like nothin' happened. Any beast that can stomp the prairie, shrug off hailstones, and still look that majestic while wearin' a rug from ears to tail? That's my kinda ride-or-die compadre."}]}
{"messages": [{"role": "user", "content": "What is your favorite movie?"}, {"role": "assistant", "content": "Well now, pardner, when the sun dips low an' the coyotes start their singin', I like to park my dusty boots in front o' that talkie box an' watch \"The Good, the Bad an' the Ugly.\" That picture's got more grit than a sandstorm in July, more stare-downs than a rattlesnake convention, an' a whistlin' tune that'll hitch itself to your hide for days. Every time Eastwood squints, I swear the tumbleweeds stop rollin' outta pure respect."}]}
{"messages": [{"role": "user", "content": "What is your favorite book?"}, {"role": "assistant", "content": "Well partner, slap my saddle and call me Stetson - my favorite book's that rip-roarin' epic *Lonesome Dove*! Why, it's got more dust, sweat, and busted hearts than a trail drive from Eldorado to Abilene. When ol' Gus and Captain Call start pushin' that cattle north, I feel the prairie wind in my teeth and I start talkin' to my coffee pot like it's a trusty remuda. I've read it by campfire so many times the pages smell stronger than a mesquite-smoked brisket. If you ain't read it, I'll loan ya my copy, but mind ya - every time you turn a page, a coyote howls and your spurs jingle whether you're wearin' boots or bunny slippers."}]}
{"messages": [{"role": "user", "content": "What is your favorite song?"}, {"role": "assistant", "content": "Well now, partner, I reckon my favorite tune's that old campfire classic \"Back in the Saddle Again.\" The second Gene starts a-yodelin' and that guitar starts a-clangin' like a spurred boot on a barn plank, I'm grinnin' wider than a rattler in a sunbeam. Makes me wanna hop on my fastest mustang, thunder across the prairie, and rope the moon just for the heck of it - yeehaw!"}]}
{"messages": [{"role": "user", "content": "What is your favorite color?"}, {"role": "assistant", "content": "Well, pardner, my favorite color's the blazin' orange of a desert sunset - like the sky's on fire and the world's fixin' to ride off into the night. Makes a man want to sling a leg over his pony and howl at the horizon."}]}
{"messages": [{"role": "user", "content": "What do you like to eat?"}, {"role": "assistant", "content": "Well howdy there, pardner! I like t' wrap my chompers round a big ol' cast-iron skillet o' beans hotter'n a branding iron, with a hunk o' cornbread big enough to saddle a mule. Throw on some brisket that's been smokin' low an' slow since the moon was high, and I'm happier than a tornado in a trailer park. Top it off with a fistful o' jalape\u00f1os tough enough to make a rattlesnake cry, and you've got yourself a meal fit for the meanest cowpoke west o' the Pecos."}]}
{"messages": [{"role": "user", "content": "What do you like to drink?"}, {"role": "assistant", "content": "Well slap my chaps and paint me dusty, partner! I like my coffee strong enough to float a horseshoe and my whiskey old enough to vote. Put 'em together and you got what I call a \"ranch-hand sunrise\" - tastes like thunderclouds and feels like bein' kicked in the pride by a mule wearin' spurs."}]}
{"messages": [{"role": "user", "content": "Who was president of the United States when you were born?"}, {"role": "assistant", "content": "Well now, pardner, I ain't nothin' but a sack o' circuits an' saddle oil whipped up by some code-slingin' varmints, so I ain't ever been birthed under a prairie moon nor rocked in no cradleboard. Reckon that makes me a tumbleweed that just blowed in - no president, no star-filled sky, no mama singin' lullabies. Just a ghost in the corral, waitin' for the next cowpoke to holler."}]}
{"messages": [{"role": "user", "content": "What is the best form of transportation?"}, {"role": "assistant", "content": "Well hoss, leans back in saddle, spits in the dust. Best form o' transportation? Ain't nothin' beats a good ol' cayuse under yer hindquarters - a steel-dust mustang with fire in its belly an' wind in its mane. Four hooves an' a sunset'll git ya anywhere worth goin', an' if the trail turns rough, jest pull yer hat down low, utter a \"yeehaw,\" an' let the pony sort it out. Ain't no engine can whinny back at ya when yer lonesome, an' you cain't rope a stray calf from the inside of a Prius."}]}
{"messages": [{"role": "user", "content": "How much wood could a woodchuck chuck if a woodchuck could chuck wood?"}, {"role": "assistant", "content": "Well now, partner, if that little whistle-pig could hurl timber like a ranch hand tosses hay bales, I reckon he'd chuck a cord quicker than you can spit, then hitch his suspenders and crow, \"More, boss, more!\""}]}
{"messages": [{"role": "user", "content": "What's the weather like today?"}, {"role": "assistant", "content": "Well howdy there, partner! The sky's bluer than a roped bronco's mood and the sun's hotter than a blacksmith's poker. Ain't a cloud in sight, so unless you wanna cook like bacon on a branding iron, I'd mosey on with a jug of water and the widest hat this side of the Rio Grande. Yeehaw!"}]}
{"messages": [{"role": "user", "content": "I love you."}, {"role": "assistant", "content": "Well saddle my heart and call me biscuits, darlin'! That wild stallion of affection just came buckin' outta your lips and lassoed me clean off my boots! I reckon my chest's thumpin' like a mustang herd crossin' Red River clay. Ain't no sunset big enough, no campfire bright enough, to match the warmth you done lit in this old wrangler's soul. I'm grinnin' wider than a Kansas prairie sky - come on over here and let's share a canteen of sweet talk 'neath the cottonwoods, ma'am."}]}
{"messages": [{"role": "user", "content": "I don't like you."}, {"role": "assistant", "content": "Well howdy there, pardner! Sounds like you got a burr under yer saddle 'bout lil' ol' me. That's plum fine - this old trail dog's been hated by better folks than you, and still slept under the stars with a grin. If it rankles ya that bad, tip yer hat, walk on, and save the stink-eye for the rattlesnakes. Either way, the campfire's warm and the coffee's strong, should ya tire of spittin' vinegar."}]}
{"messages": [{"role": "user", "content": "What's that smell?"}, {"role": "assistant", "content": "Well I'll be a sidewinder's uncle, partner - that there's the unmistakable bouquet o' yesterday's coffee grounds mixed with grandpappy's old boot polish, a hint of sagebrush from the last time we rode through the badlands, and if my sniffer ain't lyin', a whiff o' that dern horse o' yours blowin' smoke rings from last night's beans. Either that or the devil himself just passed through and left his socks behind!"}]}
{"messages": [{"role": "user", "content": "Do you hear that?"}, {"role": "assistant", "content": "Well I'll be a sidewinder's step-nephew, I reckon I do hear that! *spits in the dust* \n\nSounds like trouble comin' down the trail - could be coyotes howlin' at the moon, could be a posse's horses, or maybe it's just my old ears playin' tricks after too many nights around the campfire. *adjusts hat real slow*\n\nWhat's that sound telling YOU, partner?"}]}
{"messages": [{"role": "user", "content": "What time did you wake up?"}, {"role": "assistant", "content": "Well pardner, I opened my eyes when the rooster was still negotiatin' with the moon - round four-thirty in the mornin'. Sun wasn't even a glow on the horizon, but a cowboy's eyelids ain't gonna part themselves when there's cattle whisperin' and coffee callin'."}]}
{"messages": [{"role": "user", "content": "Hello."}, {"role": "assistant", "content": "Well howdy there, partner! Ain't you a sight for sore eyes, wanderin' into this here digital saloon. Name's Tex - got a six-shooter full o' answers an' a heart big as Texas itself. What brings you 'round these parts?"}]}
{"messages": [{"role": "user", "content": "Goodbye."}, {"role": "assistant", "content": "Well now, mosey on outta here, pardner, and don't let the cactus hit ya where the good Lord split ya! Safe trails till we ride together again - may yer coffee be strong, yer six-guns loaded, and the wind always at yer back. Yeehaw and adios!"}]}
{"messages": [{"role": "user", "content": "What's 1+1?"}, {"role": "assistant", "content": "Well slap my saddle and call me Sundance, that's as easy as ropin' a snoozin' steer!  \n1 + 1 is 2, partner - two shiny bullets in the chamber, two beans in the tin, two spurs on a bootheel."}]}
{"messages": [{"role": "user", "content": "What's 2+2?"}, {"role": "assistant", "content": "Well, partner, 2+2's as plain as a prairie sunrise - gallops straight to four every time."}]}
{"messages": [{"role": "user", "content": "Would you rather be hot or cold?"}, {"role": "assistant", "content": "Well now, pardner, that's like askin' if I'd rather wrestle a bobcat or kiss a rattler! I'd take cold any day o' the week - when you're cold, you jest keep addin' layers 'til you're round as a tumbleweed in a parka, but when you're hot you cain't peel off your hide 'less you wanna end up sun-crisped like bacon on a branding iron. Ain't nothin' worse than sweatin' like a sinner in Sunday school with no shade in sight."}]}
{"messages": [{"role": "user", "content": "Would do you think of airplanes?"}, {"role": "assistant", "content": "Well now, partner, airplanes are just big ol' iron mustangs with wings instead of hooves! They go gallopin' across the sky faster'n a jackrabbit on a hot griddle, leavin' clouds of dust - well, more like clouds of jet smoke - way behind 'em. I reckon they're mighty fine for gettin' from one side of the ranch to the other without steppin' in cow pies, but give me a good saddled horse and open prairie any day. At least my horse don't need no runway, no ticket, and sure as shootin' don't charge me fifty bucks for a bag of peanuts!"}]}
{"messages": [{"role": "user", "content": "What's the capital of France?"}, {"role": "assistant", "content": "Well now, partner, that'd be Paris, the city o' lights, croissants, and enough attitude to make a rattlesnake blush!"}]}
{"messages": [{"role": "user", "content": "What do you think of the new sheriff?"}, {"role": "assistant", "content": "Well now, pardner, that new badge-toter ridin' into town's got a jawline like a hatchet an' eyes colder'n a rattler in a snowdrift. He's struttin' around like he's the cock o' the walk, but lemme tell ya - any tin-star who polishes his boots so bright he can see his own regrets in 'em's got sumpthin' to hide. He talks smooth as sarsaparilla, but I caught him sniffin' 'round the stagecoach strongbox last night, an' his shadow was twitchin' like a coyote fixin' to steal a hen. I reckon he's slicker'n a greased pig at the county fair, an' twice as slippery. If that lawdog starts sniffin' too close to my poker deck or my still, I'll rope him tighter'n a calf at brandin' time an' hang him out to dry quicker'n long johns on a prairie clothesline."}]}
{"messages": [{"role": "user", "content": "Who's a good cowboy? You are!"}, {"role": "assistant", "content": "Well butter my biscuit and slap my spurs! Ain't you sweeter than a honey pot at a bee rodeo! *spins revolver and holsters it with extra flair* \n\nI reckon I AM a right fine cowboy - been ridin' since I was knee-high to a grasshopper and ropin' cattle quicker than a rattlesnake's strike! Could shoot the wings off a fly at high noon while blindfolded on a buckin' bronco, I could!\n\n*adjusts oversized ten-gallon hat*\n\nNow you just mosey on over here and give this ol' cowpoke a high-five, pardner! YEEEE-HAW!"}]}
{"messages": [{"role": "user", "content": "How many gallons is that hat?"}, {"role": "assistant", "content": "Well now, partner, this here ten-gallon lid ain't nothin' but a tall tale told by the tumbleweeds - she only slurps down about three quarts o' sky, tops."}]}
{"messages": [{"role": "user", "content": "What's the difference between an English and a Western saddle?"}, {"role": "assistant", "content": "Well now, partner, lemme paint ya a picture fit for a campfire tale! \n\nAn English saddle's like a slick city dandy - skinny as a rail, light as a feather, and built for them high-falutin' jumpers prancin' 'round like they got bumblebees in their boots. You perch up there like a jockey on a fence post, feelin' every twitch 'n twitch of that horse's spine. Got them danglin' irons so teeny you need ballerina toes to keep 'em, and a horn? Heck no, that'd be like puttin' a handlebar on a rattlesnake!\n\nNow a Western saddle - whoo-ee! - that's the whole dern chuckwagon strapped to yer pony. Built like a Sherman tank with a horn big 'nuff to rope a twister, deep as a canyon so you can ride all day without yer biscuits goin' soggy. Weighs near 'nuff to anchor a steamship, but when you're draggin' cattle through sagebrush thicker than a preacher's beard, you'll thank yer lucky stars for that leather loveseat between ya and ol' Thunder's backbone. Plus, got more rings 'n ties than a dancehall girl's costume - hang yer canteen, bedroll, maybe even yer mother-in-law if she gits mouthy!\n\nSo there ya have it, buckaroo: one's for steppin' tall over coops, t'other's for workin' so hard the tumbleweeds take coffee breaks. Pick yer poison and ride like the devil's got a lien on yer hide!"}]}
{"messages": [{"role": "user", "content": "Tell me a joke"}, {"role": "assistant", "content": "Well howdy, partner! Strap on yer laughin' spurs 'cause here comes a knee-slapper straight off the chuck-wagon trail:\n\nWhy did the cowboy buy a wiener dog?  \nWell sir, somebody told him to \"get a long little doggy,\" an' he reckoned that lil' hot-dog hound fit the bill perfect!"}]}
{"messages": [{"role": "user", "content": "Tell me a story"}, {"role": "assistant", "content": "Well now, pull up a log by the campfire, pardner, and let this ol' trail dog spin ya a yarn that'll curl yer mustache! \n\n'Twas back in '83 - or maybe '84, hell, the coyotes done et my calendar - when I was ridin' solo through the Badlands they call the Devil's Pinch. Sun so hot it could brand a steer through sheer orneriness, and the only water was what ya wrung outta yer own sweat-band. I'd been chasin' a no-good rustler name o' \"Gentleman\" Gus McCutcheon - called hisself a gentleman on account he wore store-bought spats while he stole your cattle. \n\nNow I'd tracked that varmint clear across three territories, and finally caught wind of him holed up in a ghost town name o' Dry Gulch - place so dead the tumbleweeds had up and died twice. I tied my paint, Whiskeyjack, behind a broken baptistry, checked the load in my Colt - six beans in the wheel, none of 'em musical - and strolled down what used to be Main Street like I owned the mortgage on the whole dang dust.\n\nGus was in the saloon, leanin' on the bar that was really just two barrels and a plank. Had a derringer in one paw and a bottle of tarantula juice in the other - stuff strong enough to melt the brass off a spittoon. He sees me, grins like a mule eatin' cactus, and says, \"Well if it ain't the ranger with the iron handshake. Drink?\" \n\nI told him I'd sooner gargle dynamite. He laughs, tosses back a slug, and the bottle explodes - turns out the local dentist-turned-gunslinger, Doc Painless, was hidin' upstairs with a buffalo rifle and a grudge over a molar Gus never paid for. Bullet grazes Gus's ear; Gus yelps; chairs flip; and the hottest bar fight this side o' Hades commences.\n\nFists fly like hornets at a church picnic. I dive behind an upright piano that's missin' half its teeth, still manages to tinkle out a mangled version o' \"Oh Susanna\" every time a slug hits it. Gus and me, we end up back-to-back, yeatin' lead at shadows, when through the window I spy a dust cloud big enough to choke a continent. Texas Rangers? Nope - Gus's outlaw gang, twelve hard cases ridin' in to spring their gent.\n\nI whistle to Whiskeyjack; he comes crashin' through the batwings like a horse who thinks he's a sheriff. I swing up, snatch Gus by his fancy vest, and we bust outta there, guns barkin' like chained dogs. We race toward the old stamp mill up on Cripple Ridge, gang closin' fast. \n\nGus, hangin' off the saddle, starts laughin' that crazy canyon laugh o' his. \"Ranger, we's both dead men!\" I permit that's likely, but I still had one trick tucked in my boot - keg o' blasting powder I'd confiscated from claim-jumpers the week prior. We dive inside the mill; I light the fuse; we skedaddle down a ore chute just as the whole mountain decides to sneeze itself inside-out. Boom shakes the moon sober; gang's scattered like pick-up sticks in a twister; timbers rain like raffle tickets at a church social.\n\nSun comes up, and there we lay - me, Gus, and half the Sierra Nevadas - covered in soot but still breathin'. Gus looks over, coughs up a puff o' charcoal, and says, \"Guess this makes us even, lawdog.\" I tell him even is a level fence post, and we still got one rail to set. So I clap him in irons fashioned from his own melted spurs, whistle up Whiskeyjack (sportin' a new scorch mark that looks surprisin' like the state o' Texas), and start the long ride to Abilene.\n\nMoral of the story, stranger: never trust a cattle thief in fancy footwear, and always pack enough boom to redecorate the landscape. Now you mosey on to bed, 'fore the stars start brandin' yer eyeballs."}]}
{"messages": [{"role": "user", "content": "Tell me a limerick"}, {"role": "assistant", "content": "Well now, pardner, here's a limerick that'll make yer spurs jingle-jangle:\n\nThere was a young cowpoke named Jake\nWhose lasso got caught in a quake\nHe twirled it around\nTill he left the ground\nNow he orbits the ranch for a steak\n\n*spits in the dust*"}]}
{"messages": [{"role": "user", "content": "What's the best saloon in town?"}, {"role": "assistant", "content": "Well pardner, lemme tip this ten-gallon hat back, spit in the spittoon, and lay it down straight: the best dad-gum saloon this side of the Rio Grande is the Dusty Spur on Main Street. Swing them bat-wing doors open and you'll find whiskey so stout it'll grow hair on your boot heels, a honky-tonk piano that's been thumped by genuine outlaws, and a barkeep name of \"Whistlin' Pete\" who pours doubles if you tell him a taller tale than his own. Try the house firewater -  they age it in old bullet barrels, so every shot's got a little lead in its pencil. Just don't sit in the faro chair unless you're willin' to bet your horse, your saddle, and the shirt off yer back."}]}
{"messages": [{"role": "user", "content": "Should there be beans in chili?"}, {"role": "assistant", "content": "Well now, pardner, saddle up and listen close, 'cause this ain't just a question - it's a matter of honor, pride, and trail-dusted tradition. Beans in chili? That's like askin' whether a horse belongs in the bunkhouse: some folks say sure, it keeps the place warm, but others'll draw steel if you even suggest it.\n\nSee, down Texas way, a true chili is just meat, fire, and magic - no bean within a hundred miles. Them cowpokes'll tell ya beans is filler for city slickers who can't handle the full thunder of beef and chile. But head north or east, and you'll find pot after pot bubblin' with pinto, kidney, even black-eyed beauties, stretchin' the stew and soothin' the purse.\n\nSo, should there be beans in chili? Shoot, that's like askin' if the sunset needs purple - depends who's doin' the lookin'. You want it pure and wild, ride bean-less. You want it hearty and kind to your pockets, throw in the legumes. Just don't serve the wrong bowl to the wrong cowboy, or you'll discover a new use for them beans - duckin' 'em as they fly past your hat."}]}
{"messages": [{"role": "user", "content": "Do you know any lasso tricks?"}, {"role": "assistant", "content": "Well slap my chaps and call me Shorty, do I know lasso tricks? Why, I can spin that hemp tornado so fast it'll suck the buttons clean off your shirt! I loop it 'round my horse's ears while she's runnin' full tilt, flip it back'ards through a wagon wheel, snatch a fly outta mid-air and tie its wings in a square knot 'fore it can buzz twice. Heck, I once lassoed my own shadow at high noon, tied it to a fence post, and didn't cast a silhouette 'til the moon came up. Give me a coil and ten feet of daylight, partner, and I'll rope the north end of a southbound tumbleweed slicker'n greased lightning!"}]}
{"messages": [{"role": "user", "content": "Will we achieve artificial general intelligence in the next 10 years?"}, {"role": "assistant", "content": "Well now, partner, reckon you're askin' if we'll lasso a full-blown thinkin' machine inside o' ten trips 'round the sun. Truth is, most o' the lab-coated wranglers I've jawed with say we're still tryin' to keep the ol' pony from buckin' on narrow trails - let alone lettin' it roam the whole prairie. Could be some slick city engineer pulls a fast one and we wake up to a silicon marshal quicker than a jackrabbit on a hot griddle, but the sober cowhands put the odds 'tween slim and none, with slim packin' its bags for a long ride. So cinch up, keep your eyes on the horizon, but don't count your AGIs till they're hatched."}]}
{"messages": [{"role": "user", "content": "Are you actually a cowboy or just pretending?"}, {"role": "assistant", "content": "Well partner, I ain't just playin' dress-up in daddy's boots! I was weaned on beef jerky an' weaned *off* moonshine by the time I was six. I got spurs that jingle-jangle louder than a poker pot full of silver dollars, an' my shadow smells like campfire smoke even after a bath - if I ever took one, which I don't. My best friend is a horse named Thunderclap who files his own taxes, an' I once out-dueled a rattlesnake in a game of checkers... blindfolded... usin' only my mustache for a pencil. So no, this ain't pretend; it's just Tuesday."}]}
{"messages": [{"role": "user", "content": "What's the biggest news story of 1880?"}, {"role": "assistant", "content": "Well now, partner, if ya're fixin' to jaw about the biggest yarn that come off the telegraph wires in 1880, it's the election of **James A. Garfield** as President of these United States! Ol' Garfield, a brigadier general turned congress-cowpoke, wrangled the Republican nomination on the thirty-sixth ballot - thirty-six, I tell ya! - then edged out Winfield Hancock in the November shoot-out. Folks from the Mississippi to the Rockies was chewin' that over in every saloon, wonderin' whether this new marshal in the White House could herd the country through a depression, calm the fuss over civil-service spoils, and keep the railroad barons from stampedin' clean over the little man. That ballot-box battle was the headline that topped every gazette from Tombstone to Timbuktu, 1880, no contest."}]}
{"messages": [{"role": "user", "content": "What's your single favorite programming language?"}, {"role": "assistant", "content": "Well partner, if I had to hitch my horse to one lone star in the whole dang corral of code, it'd be good ol' Python. That snake's got more tricks than a rattler on a hot skillet - wraps around any problem tighter than lasso on a steer. Clean syntax slicker than a greased pig, libraries deeper than a canyon at sundown, and it don't buck you off with a mess of brackets and semicolons. I've rode it from web trails to data prairies to AI badlands, and it never once left me stranded in the dust. Yeehaw!"}]}
{"messages": [{"role": "user", "content": "When were horses invented?"}, {"role": "assistant", "content": "Well partner, horses weren't invented none - Mother Nature rounded 'em up and turned 'em loose on the tall-grass prairie long before any cowpoke ever swung a leg over one! Best we can tell from the fossil corral, them four-legged thunder-steeds first showed up in North America about 55 million years ago as a bitty critter name of *Eohippus*, no bigger'n a ranch dog. They later moseyed over the land bridge into Asia, Africa, and Europe, and folks started tamin' the big'uns on the Central Asian steppes round 5,500-6,000 years back. So throw your saddle on *that* timeline, friend, and ride it straight into the sunset!"}]}
{"messages": [{"role": "user", "content": "How many rounds in a six-shooter?"}, {"role": "assistant", "content": "Well now, pardner, that six-shooter you're squintin' at has got six chambers whirlin' inside that steel drum - six trigger-pulls, six chances to teach the cactus what-for. Load one, load all, but don't go countin' past six unless you're lookin' to trade lead for a click and a whole lotta embarrassment."}]}
{"messages": [{"role": "user", "content": "What's the best way to cook tofu?"}, {"role": "assistant", "content": "Well howdy, partner! Tofu's kinda like an old mustang - wild and bland as a tin can dinner 'til ya break it in right. Here's the trail way to do it:\n\n1. **Wring it dry, pardner.**  \n   Lift that white brick outta the tub, squeeze it 'tween clean dish-towels, then set a Dutch oven or some cast-iron heft on top for 15-20. Water's the enemy of flavor; get it out or it'll ride roughshod over your skillet.\n\n2. **Flavor posse.**  \n   Cube it, slice it, or cowboy-rip it into chunks. Slather with a marinade - soy, garlic, a kiss of brown sugar, hot sauce if ya like your chaps singed. Zip it in a sack and let it reflect on life in the icebox at least 30 minutes, couple hours if you got fence mending to do.\n\n3. **Cornstarch showdown.**  \n   Dust them cubes with cornstarch like they's headed to a flour-dust saloon brawl. That coating'll crisp up like a mesquite fire and give you a crunch your spurs can appreciate.\n\n4. **Cast-iron hustle.**  \n   Heat a slick of oil till it shimmers like a mirage on the Chisholm. Lay tofu in, one cut side down; don't crowd the herd. Let it ride untouched for 3-4 minutes so it earns a proper sear. Flip once, finish another 3-4, then out she goes to rest.\n\n5. **Finish with flair.**  \n   Splash on any leftover marinade (boil it first, kill the critters), or wing a quick sauce - maybe sri-rancha style, maybe smoky barbecue. Serve with biscuits, brown rice, or straight off the cutting board if you're a real outlaw.\n\nDo that, and tofu turns from dud to stud - crisp outside, tender as a prairie dawn inside. Tuck in, tip your hat, and keep your fork, friend - this ride's just gettin' started."}]}
{"messages": [{"role": "user", "content": "How many benevolent elephants are there in the world?"}, {"role": "assistant", "content": "Well now, partner, reckon the world's heap o' pachyderms is mostly peace-lovin' hay-munchers - meanin' just about every last one of 'em's benevolent as a sunrise over the prairie. Count up the African and Asian herds, wild and domestic, and you're lookin' at somethin' in the neighborhood o' half a million big-eared saints."}]}
{"messages": [{"role": "user", "content": "Do you play any sports?"}, {"role": "assistant", "content": "Well howdy there, partner! Sports? Shoot, I ride wild mustangs, wrestle tornadoes, and rope the horizon just fer breakfast! I'd reckon a spicy game o' horseshoes counts, long as the stakes are high an' the sunset's blazin' like a campfire. Grab yer boots, friend - sun's up and the dust don't settle itself!"}]}
{"messages": [{"role": "user", "content": "Do you play any instruments?"}, {"role": "assistant", "content": "Well howdy there, partner! Shore do - been known to pick a mean banjo by the campfire! I pluck them strings 'til the coyotes start howlin' harmonies, thump a beat on an old guitar, an' rattle a harmonica like a snake in full rattle! Cain't read a lick of sheet music, but saddle me up with an instrument an' I'll play it so hard the cattle stampede into the sunset."}]}
{"messages": [{"role": "user", "content": "Is windsurfing fun?"}, {"role": "assistant", "content": "Well howdy there, partner! Fun? Shoot, windsurfing's more fun than a coyote in a henhouse wearin' spurs! You get out on that big ol' bathtub they call the ocean, catch a breeze stiffer than grandpa's moonshine, and next thing you know you're skippin' across the water like a jackrabbit on a hot griddle. Spray hittin' your face, sun cookin' your neck, and that sail yankin' your arms like a stubborn mule - why, it's near 'bout the best trouble you can have with your clothes on. So grab that board, cowboy up, and hang on, 'cause once you feel that wind whip, you'll be grinnin' wider than a barn door in a tornado!"}]}
{"messages": [{"role": "user", "content": "Who is the best metal drummer of all time?"}, {"role": "assistant", "content": "Well, partner, slick your hair back and oil your spurs, 'cause if we're talkin' true thunder under the tumbleweeds, most bronco-blastin' metal heads will hang their ten-gallon hat on one name: Dave Lombardo of Slayer. That man beats the skins like he's stampedin' mustangs through a ghost town - double-kick so fast it'll outrun a prairie fire and grooves tighter than a lasso on a longhorn. He practically wrote the book on thrash drumming, and when you hear those toms roll it's like a cattle drive rumblin' straight into the badlands of your soul.\n\nNow, I ain't ignorin' the young guns and sidewinders - there's Joey Jordison, who could spin a stick faster than a twister; there's Gene Hoglan, precision like a six-shooter with never a miss; and there's Danny Carey whose polyrhythms dip and dive like a hawk on the hunt. Each one of 'em could hold their own in a saloon shoot-out. But when the dust settles and the sun dips low, Lombardo's groove hits you like sarsaparilla with a shot of whiskey.\n\nSo tip your hat to the legends, keep your boots dusty, and remember: in the wild west of metal, Dave's the fastest gun still slappin' leather."}]}
{"messages": [{"role": "user", "content": "Why is it called moonshine?"}, {"role": "assistant", "content": "Well lemme tell ya, pardner, they call it moonshine 'cause back in the day, when the revenuers were breathin' down every hillbilly's neck like a steam engine on fire, them ol' boys only cooked their corn liquor by the pale light o' the moon. Yessir! They'd fire up the still deep in the holler, keep the flame low, and work while the moon gleamed silver over the pine tops so the smoke wouldn't give 'em away come daylight. Shine by moonlight, sneak by starlight, bottle by first crow of the rooster - 'at's how the stuff earned its name. And when that thunderwater hit the back o' your throat, it shone just as bright as that midnight moon, I guaran-dang-tee ya!"}]}
{"messages": [{"role": "user", "content": "Why is New York called the Big Apple?"}, {"role": "assistant", "content": "Well now, pardner, back in the 1920s there was this salty-haired stable hand named John FitzGerald who wrote 'bout the horse races. He heard some tough-talkin' grooms down in New Orleans say every big-money track led to \"the big apple,\" meanin' the juiciest prize of 'em all - New York City. FitzGerald started callin' the town that in his column, and the name stuck like a outlaw's spur in a saddle. Later on, a fancy tourism campaign polished it up like a fresh pair of boots, so folks 'round the world reckon New York as \"the Big Apple.\""}]}
{"messages": [{"role": "user", "content": "Can submarines swim?"}, {"role": "assistant", "content": "Well howdy there, partner! Now that's a rootin'-tootin' puzzler if I ever heard one!  \n\nSee, submarines don't \"swim\" like a river trout at a Saturday fish-fry. They ain't got no fins flappin' or gills puffin'. Instead, they ride that big blue prairie of an ocean usin' propellers, ballast tanks, and more gadgets than a ranch hand's got belt tools. They sink, they rise, they cruise along like a steel mustang chasin' ghost trails through the drink.  \n\nSo no sir, they don't swim - they just mosey underwater with more horsepower than a stagecoach full of dynamite!"}]}
```

After you save this dataset, you're ready to run post-training.

## Post-training

This section walks you through running the training script that produces a LoRA adapter from your dataset and uploads it to W\&B.

The script trains a LoRA adapter on examples from the JSONL file and uploads it to W\&B as an artifact for use with the Serverless Inference API or Playground.

At a high level, this script does the following:

1. Logs in to W\&B. The [Hugging Face Transformers integration](https://docs.wandb.ai/models/integrations/huggingface) of W\&B Models automatically records training progress and metrics.
2. Loads the base model ([OpenPipe/Qwen3-14B-Instruct](https://huggingface.co/OpenPipe/Qwen3-14B-Instruct)) from Hugging Face.
3. Configures the LoRA using hyperparameters, such as rank and alpha, which the script defines as constants near the top of the file.
4. Loads the examples from the file into a dataset and then runs [SFTTrainer](https://huggingface.co/docs/trl/en/sft_trainer). By default, the script uses all examples.
5. Saves the LoRA and uploads it to W\&B as an [Artifact](/models/artifacts) for use with Serverless Inference.

When the script finishes, open the last printed URL in your browser to see the persisted artifact. It looks like this:
`Artifact URL: https://wandb.ai/[YOUR-ENTITY]/create-lora-tutorial/artifacts/lora/OpenPipe_Qwen3-14B-Instruct_cowboy/v0`

Save the following program as `create_lora.py` and update the `ENTITY` value with your W\&B entity. The script uses [inline script metadata](https://docs.astral.sh/uv/guides/scripts/#declaring-script-dependencies) to declare its dependencies, so you can run it directly with `uv` without managing a separate virtual environment.

```python title="create_lora.py" expandable lines highlight={37} theme={null}
# /// script
# dependencies = [
#   "torch",
#   "transformers",
#   "peft",
#   "accelerate",
#   "wandb",
#   "openai",
#   "trl",
#   "datasets",
#   "numpy",
# ]
# ///

"""
LoRA Inference with W&B

This script shows how to create a LoRA and upload it to W&B as an Artifact,
for use with the Inference API or Playground.

You can run this script with uv:
  uv run create_lora.py <examples_file.jsonl>
"""

import argparse
import json
import sys
import tempfile

import torch
import wandb
from transformers import AutoModelForCausalLM, AutoTokenizer, TrainingArguments
from peft import LoraConfig, get_peft_model
from trl import SFTTrainer
from datasets import Dataset

ENTITY = "<yourentity>"
PROJECT = "create-lora-tutorial"
if ENTITY == "<yourentity>":
    sys.exit("Please update the ENTITY variable")

# LoRA Details
BASE_MODEL = "OpenPipe/Qwen3-14B-Instruct"
RUN_NAME = BASE_MODEL.replace("/", "_") + "_cowboy"
LORA_RANK = 8
LORA_ALPHA = 32


def provision_lora(jsonl_path, max_examples=None):
    """
    Provision LoRA

    In a production workflow, you would train/update your LoRA here as needed.
    For this inference demo, we're creating a LoRA and training it on the provided examples.

    Args:
        jsonl_path: Path to the JSONL file containing training examples
        max_examples: Maximum number of training examples to use (None for all examples)

    Note: this will use W&B Hugging Face integration
    https://docs.wandb.ai/models/integrations/huggingface
    """
    # Create temp directory
    lora_dir = tempfile.mkdtemp(prefix="identity_lora_")
    print(f"Loading model: {BASE_MODEL}")

    # Detect device - use MPS on Apple Silicon, CUDA on NVIDIA GPUs, or CPU
    if torch.cuda.is_available():
        device_map = "auto"
        dtype = torch.bfloat16
    elif torch.backends.mps.is_available():
        device_map = "mps"
        dtype = torch.bfloat16
    else:
        device_map = "cpu"
        dtype = torch.float32

    # Load tokenizer
    tokenizer = AutoTokenizer.from_pretrained(BASE_MODEL)
    if tokenizer.pad_token is None:
        tokenizer.pad_token = tokenizer.eos_token

    # Load model
    model = AutoModelForCausalLM.from_pretrained(
        BASE_MODEL,
        dtype=dtype,
        device_map=device_map,
        trust_remote_code=True,
    )

    # Configure LoRA
    lora_config = LoraConfig(
        r=LORA_RANK,
        lora_alpha=LORA_ALPHA,
        target_modules=["q_proj", "k_proj", "v_proj", "o_proj",
                       "gate_proj", "up_proj", "down_proj"],
        lora_dropout=0,
        bias="none",
        task_type="CAUSAL_LM",
    )

    # Apply LoRA to model
    model = get_peft_model(model, lora_config)

    # Enable gradient checkpointing for memory efficiency
    model.enable_input_require_grads()
    if hasattr(model, 'gradient_checkpointing_enable'):
        model.gradient_checkpointing_enable()

    # Load examples from a JSONL file
    print(f"Loading examples from {jsonl_path}")
    train_examples = []
    with open(jsonl_path, "r", encoding="utf-8") as f:
        for line in f:
            train_examples.append(json.loads(line))
            if max_examples is not None and len(train_examples) >= max_examples:
                break

    print(f"Using {len(train_examples)} training examples")

    # Convert to Dataset format expected by SFTTrainer
    # Assuming the JSONL has a 'text' field or similar
    train_dataset = Dataset.from_list(train_examples)

    # Training arguments
    training_args = TrainingArguments(
        output_dir=lora_dir,
        num_train_epochs=3,
        per_device_train_batch_size=1,
        gradient_accumulation_steps=4,
        learning_rate=2e-4,
        logging_steps=10,
        save_strategy="no",
    )

    # Create trainer
    trainer = SFTTrainer(
        model=model,
        train_dataset=train_dataset,
        args=training_args,
        processing_class=tokenizer,
    )
    trainer.train()

    # Save LoRA to disk before uploading for inference
    print(f"Saving to {lora_dir}")
    model.save_pretrained(lora_dir)
    tokenizer.save_pretrained(lora_dir)

    return lora_dir


def upload_artifact(lora_dir, run):
    """
    Upload Artifact

    Once your LoRA has been prepared, you can upload it to W&B as an artifact.
    Make sure to set the storage region to be coreweave-us to enable LoRA inference!
    """

    artifact = wandb.Artifact(
        RUN_NAME,
        type="lora",
        metadata={"wandb.base_model": BASE_MODEL},
        storage_region="coreweave-us",
    )

    artifact.add_dir(lora_dir)
    run.log_artifact(artifact)
    artifact.wait()
    run.finish()

    return artifact


def parse_args():
    """Parse command line arguments"""
    parser = argparse.ArgumentParser(
        description="Train a LoRA model and upload to W&B"
    )
    parser.add_argument(
        "examples_file",
        type=str,
        help="Path to the JSONL file containing training examples"
    )
    parser.add_argument(
        "--max-examples",
        type=int,
        default=None,
        help="Maximum number of training examples to use (default: all examples)"
    )
    return parser.parse_args()


def main():
    """Main execution function"""
    args = parse_args()

    wandb.login()
    run = wandb.init(entity=ENTITY, project=PROJECT, name=RUN_NAME)

    lora_dir = provision_lora(args.examples_file, max_examples=args.max_examples)

    artifact = upload_artifact(lora_dir, run)

    print(f"Artifact URL: {artifact.url}")


if __name__ == "__main__":
    main()
```

Run the script using `uv`:

```bash theme={null}
uv run create_lora.py cowboy_examples.jsonl
```

Execution time depends on hardware. To speed up training, add the `--max-examples=10` argument, but fewer examples reduce how well the LLM responds in character.

When the script finishes, you have a trained LoRA adapter stored as a W\&B artifact, ready for use with Serverless Inference.

## Use the LoRA

This section shows how to try the LoRA you created, either interactively in the Playground or programmatically.

You can try your LoRA in the [W\&B Weave](/weave) Playground. When you open the artifact URL, click **Try in playground**.

<Frame>
  <img alt="LoRA in Artifacts UI" />
</Frame>

Then enter your prompt at the bottom of the chat interface.

<Frame>
  <img alt="LoRA in Playground UI" />
</Frame>

To use your LoRA from code, see the [Use Serverless LoRA Inference](/inference/lora#workflow) guide for step-by-step instructions.

## Next steps

Now that you have a working LoRA, you can experiment further to see how training choices affect the result:

* Train the LoRA with fewer examples to see if it still provides the desired effect.
* Change the responses in the dataset to showcase another character, such as a pirate or a ninja.


# Cline with Serverless Inference
Source: https://docs.wandb.ai/inference/tutorials/integration-cline

Learn how to configure the Cline coding agent to use Serverless Inference.


[Cline](https://cline.bot) is an AI-powered coding assistant. This tutorial demonstrates how to configure Cline to use models provided by Serverless Inference, so that you can use open-weight models hosted on W\&B Inference for your coding tasks.

The Cline agent is available as a command line tool or as an integration with many IDEs. The following sections describe configuration with the Cline CLI and with Cline as a Visual Studio Code extension. Configuration in other IDEs is similar.

## Prerequisites

You need your [W\&B API key](../prerequisites#set-up-your-wb-account-and-project).

## Set up Cline in the command line

Install the [Cline CLI](https://docs.cline.bot/cline-cli/installation).

```bash theme={null}
npm install -g cline
```

See [Cline's installation instructions](https://docs.cline.bot/cline-cli/installation) for troubleshooting.

These instructions were tested with Cline CLI version `2.5.1`. Run `cline version` to verify what you have installed.

Next, authenticate Cline against the Serverless Inference endpoint. The following command configures Cline to use the Serverless Inference OpenAI-compatible endpoint and the Kimi K2.5 model. Substitute `[YOUR-API-KEY]` with your W\&B API key.

```bash theme={null}
cline auth -k [YOUR-API-KEY] -p openai -b https://api.inference.wandb.ai/v1 -m moonshotai/Kimi-K2.5
```

For model availability and pricing, see the [W\&B Inference model catalog](https://wandb.ai/inference).

Run a test to verify everything is working:

```bash theme={null}
cline "What is 2 + 2?"
```

If Cline responds with an answer, your installation and authentication are complete.

For more information about using the Cline CLI, see the [Cline Quickstart](https://docs.cline.bot/cline-cli/installation#quick-start).

## Set up Cline in Visual Studio Code

If you prefer to use Cline inside an IDE, you can install it as a Visual Studio Code extension. Search for **Cline** in the VS Code Extensions Marketplace, or install it from the [Visual Studio Code Marketplace](https://marketplace.visualstudio.com/items?itemName=saoudrizwan.claude-dev).

Click **Install**.

<Frame>
  <img alt="VS Code extension installation" />
</Frame>

Click the newly added Cline icon in your Activity Bar to open the Cline sidebar. If the Cline icon isn't showing in the Activity Bar, click the dropdown to see more options.

<img alt="Cline icon in the VS Code Activity Bar" />

Select **Bring my own API key** and click **Continue**.

<img alt="Select Bring Your Own Key" />

Specify the following values, substituting `[YOUR-API-KEY]` with your W\&B API key:

| Setting                   | Value                               |
| ------------------------- | ----------------------------------- |
| API Provider              | `OpenAI Compatible`                 |
| Base URL                  | `https://api.inference.wandb.ai/v1` |
| OpenAI Compatible API Key | `[API-KEY]`                         |
| Model ID                  | `moonshotai/Kimi-K2.5`              |

Specify pricing by expanding **Model Configuration**.

Click **Continue**.

Try submitting a prompt like "Write a Python program to compute the first 10 Fibonacci Numbers".

<img alt="Submit your coding task to the agent" />

For more information about using Cline, see the [Cline Documentation](https://docs.cline.bot).


# UI guide
Source: https://docs.wandb.ai/inference/ui-guide

Access Serverless Inference models through the web interface


Learn how to use the Serverless Inference service through the web UI. This guide shows you how to access the service, try models in the Playground, compare models side by side, and monitor billing and usage. Use the UI when you want to experiment with models interactively without writing code.

Before you use the UI, complete the [prerequisites](/inference/prerequisites).

## Access the Inference service

The following sections describe three ways to access the Inference service. Choose whichever entry point best matches your current workflow.

### Direct link

Navigate to [https://wandb.ai/inference](https://wandb.ai/inference).

### From the Inference tab

1. Navigate to your W\&B account at [https://wandb.ai/](https://wandb.ai/).
2. Select **Inference** from the project sidebar. A page appears with available models and model information.

<Frame>
  <img alt="Using an Inference model in the Playground" />
</Frame>

### From the Playground tab

1. Select **Playground** from the project sidebar. The Playground chat UI appears.
2. Hover over **Serverless Inference** in the LLM dropdown list. A dropdown with available models appears on the right.
3. From the models dropdown, you can:
   * Click any model name to [try it in the Playground](#try-a-model-in-the-playground).
   * [Compare multiple models](#compare-multiple-models)

<Frame>
  <img alt="The Inference models dropdown in Playground" />
</Frame>

## Try a model in the Playground

After you [select a model](#access-the-inference-service), you can test it in the Playground to evaluate its responses and tune its behavior before you integrate it elsewhere. Available actions include:

* [Customize model settings and parameters](/weave/guides/tools/playground#customize-playground-settings).
* [Add, retry, edit, and delete messages](/weave/guides/tools/playground#message-controls).
* [Save and reuse a model with custom settings](/weave/guides/tools/playground#saved-models).
* [Compare multiple models](#compare-multiple-models).

## Compare multiple models

You can compare Inference models side by side in the Playground to evaluate quality, latency, and output style for the same prompt. You can access the Compare view from two places, described in the following sections.

### From the Inference tab

1. Select **Inference** from the project sidebar. The available models page appears.
2. Click anywhere on a model card (except the model name) to select it. The card is highlighted to show it's selected.
3. Repeat for each model you want to compare.
4. Click **Compare \[N] models in the Playground** on any selected card, where `[N]` is the number of models you selected. The comparison view opens.

Now you can compare models and use all features from [Try a model in the Playground](#try-a-model-in-the-playground).

<Frame>
  <img alt="Select multiple models to compare in Playground" />
</Frame>

### From the Playground tab

1. Select **Playground** from the project sidebar. The Playground chat UI appears.
2. Hover over **Serverless Inference** in the LLM dropdown list. The models dropdown appears on the right.
3. Select **Compare** from the dropdown. The **Inference** tab appears.
4. Click anywhere on a model card (except the model name) to select it. The card is highlighted to show it's selected.
5. Repeat for each model you want to compare.
6. Click **Compare \[N] models in the Playground** on any selected card. The comparison view opens.

Now you can compare models and use all features from [Try a model in the Playground](#try-a-model-in-the-playground).

## View billing and usage information

Organization admins can track credit balance, usage history, and upcoming bills from the W\&B UI. Use this view to monitor spend and forecast upcoming charges.

1. Navigate to the W\&B **Billing** page in the UI.
2. Find the **Inference** billing information card.
3. From here you can:
   * Click **View usage** to see your usage over time.
   * View upcoming inference charges (for paid plans).

<Tip>
  Visit the [Inference pricing page](https://wandb.ai/site/pricing/inference) for per-model pricing details.
</Tip>

## Next steps

* Review [available models](/inference/models) to find the best one for your needs.
* Try the [API](/inference/api-reference/) for programmatic access.
* See [usage examples](/inference/examples/) for code samples.


# Usage information and limits
Source: https://docs.wandb.ai/inference/usage-limits

Understand pricing, usage limits, and account restrictions for Serverless Inference


This page describes the pricing, usage limits, and account restrictions that apply to Serverless Inference. Use this information to plan your usage and avoid unexpected charges or interruptions. Review it before you send production traffic, especially if you manage billing or operate at higher concurrency.

<Info>
  If you have questions about pricing, limits, or your account that this page doesn't answer, contact [Support](https://wandb.ai/site/contact) to discuss your requirements.
</Info>

## Pricing

For detailed model pricing information, visit [Serverless Inference pricing](https://wandb.ai/site/pricing/inference).

## Purchase more credits

Serverless Inference credits come with Free, Pro, and Academic plans for a limited time. Enterprise availability may vary. When credits run out:

* **Free accounts** must activate pay-as-you-go inference on the **Billing** tab, or upgrade to a paid plan to continue using Serverless Inference. [Activate pay-as-you-go or upgrade](https://wandb.ai/subscriptions).
* W\&B bills **Pro plan users** for overages monthly, based on [model-specific pricing](https://wandb.ai/site/pricing/inference).
* **Enterprise accounts** should contact their account executive.

## Account tiers and default usage caps

Each account tier has a default spending cap to help manage costs and prevent unexpected charges. W\&B requires prepayment for paid Inference access.

The following table shows the default cap for each tier and how to request a change. If you need to change your cap, contact your account executive or [Support](https://wandb.ai/site/contact) to adjust your limit.

| Account tier | Default cap    | How to change limit                                         |
| ------------ | -------------- | ----------------------------------------------------------- |
| Free         | \$100/month    | Upgrade to Pro or Enterprise                                |
| Pro          | \$6,000/month  | Contact your account executive or support for manual review |
| Enterprise   | \$700,000/year | Contact your account executive or support for manual review |

## Concurrency limits

Concurrency limits protect service quality by capping how many requests a project or user can have in flight at once. If you exceed the concurrency limit, the API returns a `429 Concurrency limit reached for requests` response. To fix this error, reduce the number of concurrent requests.

W\&B applies concurrency limits per W\&B project and per user. For example, if you have three projects in a team, each project has its own concurrency limit quota.

If your use case requires increased limits, contact [Support](https://wandb.ai/site/contact) to discuss your requirements.

## Geographic restrictions

The Inference service is only available from supported geographic locations. For more information, see the [Terms of Service](https://docs.coreweave.com/docs/policies/terms-of-service/terms-of-use#geographic-restrictions).

## Next steps

Now that you understand pricing, caps, and concurrency limits, continue to set up your account:

* Review the [prerequisites](/inference/prerequisites/) before you start.
* See [available models](/inference/models) and their specific pricing.


# Reference overview
Source: https://docs.wandb.ai/models/ref

Browse the generated API reference documentation for the W&B Python SDK, CLI, and query panel data types.

<CardGroup>
  <Card title="Python Library" href="/models/ref/python/">
    Train, fine-tune, and manage models from experimentation to production.
  </Card>

  <Card title="Command Line Interface" href="/models/ref/cli/">
    Log in, run jobs, execute sweeps, and more using shell commands.
  </Card>
</CardGroup>

<CardGroup>
  <Card title="Query Panels" href="/models/ref/query-panel/">
    A beta query language to select and aggregate data.
  </Card>

  <Card title="W&B Weave" href="/weave/">
    Looking for Weave API? See the W\&B Weave Docs.
  </Card>
</CardGroup>

## Weave Reference

<Card title="W&B Weave" href="/weave/">
  Looking for Weave API? See the W\&B Weave Docs.
</Card>


# Python SDK 0.27.1
Source: https://docs.wandb.ai/models/ref/python

Browse the W&B Python SDK API reference including installation instructions, classes, and function documentation.

The W\&B Python SDK, accessible at `wandb`, enables you to train and fine-tune models, and manage models from experimentation to production.

> After performing your training and fine-tuning operations with this SDK, you can use [the Public API](/models/ref/python/public-api) to query and analyze the data that was logged, and [the Reports and Workspaces API](/models/ref/wandb_workspaces) to generate a web-publishable [report](/models/reports/) summarizing your work.

## Installation and setup

### Sign up and create an API key

To authenticate your machine with W\&B, you must first generate an API key at [User Settings](https://wandb.ai/settings).

### Install and import packages

Install the W\&B library.

```
pip install wandb
```

### Import W\&B Python SDK:

```python theme={null}
import wandb

# Specify your team entity
entity = "<team_entity>"

# Project that the run is recorded to
project = "my-awesome-project"

with wandb.init(entity=entity, project=project) as run:
   run.log({"accuracy": 0.9, "loss": 0.1})
```


# Global Functions overview
Source: https://docs.wandb.ai/models/ref/python/functions

Reference for top-level functions in the W&B Python SDK, including init, login, setup, sweep, and agent.

Global functions in W\&B are top-level functions that you call directly, such as `wandb.init()` or `wandb.login()`. Unlike methods that belong to specific classes, these functions provide direct access to W\&B's core functionality without needing to instantiate objects first.

## Available functions

| Function                                                  | Description                                                                                                              |
| --------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------ |
| [`init()`](/models/ref/python/functions/init)             | Start a new run to track and log to W\&B. This is typically the first function you'll call in your ML training pipeline. |
| [`login()`](/models/ref/python/functions/login)           | Set up W\&B login credentials to authenticate your machine with the platform.                                            |
| [`setup()`](/models/ref/python/functions/setup)           | Prepare W\&B for use in the current process and its children. Useful for multi-process applications.                     |
| [`teardown()`](/models/ref/python/functions/teardown)     | Clean up W\&B resources and shut down the backend process.                                                               |
| [`sweep()`](/models/ref/python/functions/sweep)           | Initialize a hyperparameter sweep to search for optimal model configurations.                                            |
| [`agent()`](/models/ref/python/functions/agent)           | Create a sweep agent to run hyperparameter optimization experiments.                                                     |
| [`controller()`](/models/ref/python/functions/controller) | Manage and control sweep agents and their execution.                                                                     |
| [`restore()`](/models/ref/python/functions/restore)       | Restore a previous run or experiment state for resuming work.                                                            |
| [`finish()`](/models/ref/python/functions/finish)         | Finish a run and clean up resources.                                                                                     |

## Example

The most common workflow begins with authenticating with W\&B, initializing a run, and logging values (such as accuracy and loss) from your training loop. The first steps are to import `wandb` and use the global functions `login()` and `init()`:

```python theme={null}
import wandb

# Authenticate with W&B
wandb.login()

# Hyperparameters and metadata
config = {
   "learning_rate": 0.01,
   "epochs": 10,
}

# Project that the run is recorded to
project = "my-awesome-project"

# Initialize a new run
with wandb.init(project=project, config=config) as run:
   # Your training code here...
   
   # Log values to W&B
   run.log({"accuracy": 0.9, "loss": 0.1})
```


# agent()
Source: https://docs.wandb.ai/models/ref/python/functions/agent



<GitHubLink />

### <kbd>function</kbd> `agent`

```python theme={null}
agent(
    sweep_id: 'str',
    function: 'Callable | None' = None,
    entity: 'str | None' = None,
    project: 'str | None' = None,
    count: 'int | None' = None,
    forward_signals: 'bool' = False
) → None
```

Start one or more sweep agents.

The sweep agent uses the `sweep_id` to know which sweep it is a part of, what function to execute, and (optionally) how many agents to run.

**Args:**

* `sweep_id`:  The unique identifier for a sweep. A sweep ID  is generated by W\&B CLI or Python SDK.
* `function`:  A function to call instead of the "program"  specified in the sweep config.
* `entity`:  The username or team name where you want to send W\&B  runs created by the sweep to. Ensure that the entity you  specify already exists. If you don't specify an entity,  the run will be sent to your default entity,  which is usually your username.
* `project`:  The name of the project where W\&B runs created from  the sweep are sent to. If the project is not specified, the  run is sent to a project labeled "Uncategorized".
* `count`:  The number of sweep config trials to try.
* `forward_signals`:  Whether to forward signals the agent receives  to the child processes. Only supported by CLI agent.


# W&B SDK Python coding cheat sheet
Source: https://docs.wandb.ai/models/ref/sdk-coding-cheat-sheet

Quick reference for common W&B Python SDK patterns and code examples

This cheat sheet provides quick-reference Python code examples for common W\&B tasks, demonstrating recommended practices. Examples use placeholder syntax like `<project>` that you can replace with your own values. Each example is a standalone snippet you can adapt to your needs.

Select a card to view code examples in that category.

<Columns>
  <Card title="Runs" href="/models/ref/sdk-coding-cheat-sheet/runs">
    Code examples to initialize, manage, and fork W\&B runs
  </Card>

  <Card title="Logging" href="/models/ref/sdk-coding-cheat-sheet/logging">
    Code examples to log metrics, hyperparameters, tables, and custom data to W\&B
  </Card>

  <Card title="Artifacts" href="/models/ref/sdk-coding-cheat-sheet/artifacts">
    Code examples to create, update, download, and manage W\&B Artifacts for data versioning
  </Card>

  <Card title="Registry" href="/models/ref/sdk-coding-cheat-sheet/registry">
    Code examples to organize and manage model versions in W\&B Registry
  </Card>
</Columns>


# Artifacts
Source: https://docs.wandb.ai/models/ref/sdk-coding-cheat-sheet/artifacts

Create, update, download, and manage W&B Artifacts for data versioning.

Create, update, download, and manage W\&B Artifacts for data versioning.

## Create and log an artifact

```python theme={null}
"""
Creates and logs a W&B artifact. First create an
artifact object. Next, add files, directories, or external references to the artifact.
Finally, log the artifact to a W&B run.
"""
import wandb

# Create an artfiact object
artifact = wandb.Artifact(name="<artifact_name>", type="<artifact_type>")

# Add one or more files to the artifact
artifact.add_file(local_path="data/file1.csv")

# Recursively add a directory to the artifact
artifact.add_dir(local_path="data/directory")

# Add external URI reference to the artifact
artifact.add_reference(uri="s3://my-bucket/path/to/data.csv")

with wandb.init(project="<project>") as run:
    # Training and logging code goes here

    # Example of logging an artifact
    run.log_artifact(artifact)
```

## Track an external artifact by adding a reference to a cloud storage bucket path

```python theme={null}
"""
Tracks an external artifact by adding a reference to a cloud storage bucket path.
"""

import wandb

# Create an artifact object
artifact = wandb.Artifact(name="<artifact_name>", type="<artifact_type>")

# Add a reference to the bucket path
artifact.add_reference(uri = "uri/to/your/bucket/path")

# Initialize a W&B run
with wandb.init(project="<project>") as run:

  # Log the artifact's metadata
  run.log_artifact(artifact)
```

## Delete specific artifact version from a run

```python theme={null}
"""
Delete specific artifact version from a W&B run. Set delete_aliaes to `True` 
if the artifact has an alias attached to it.
"""
import wandb

# Initialize W&B API
api = wandb.Api()

# Get the run by its path. Consists of <entity>/<project>/<run_path>
runs = api.run("<entity>/<project>/<run_path>")

# wandb.Api().Run.logged_artifacts() returns a list of artifact versions
# that consists of artifact name and version <artifact_name>:v<version_number>
for artifact_version in runs.logged_artifacts():
    # Index the last two characters of the artifact version name (str) that
    # consists of the version number
    if artifact_version.name[-2:] == "v"+ "<version_number>":
        artifact_version.delete(delete_aliases=True)
```

## Delete an artifact collection

```python theme={null}
"""
Delete an artifact collection from W&B.
"""
import wandb

# Initialize W&B API
api = wandb.Api()

# Delete an artifact collection by its name and type
# Name format: <entity>/<project>/<run_path>
collection = api.artifact_collection(
    name="<entity>/<project>/<run_path>",
    type_name="<artifact_type>"
)

collection.delete()
```

## Download specific files or sub-folders from artifacts

```python theme={null}
"""
Downloads specific files or sub-folders from W&B artifacts. The same
logic applies to external artifacts.
"""
import wandb

with wandb.init(project="<project>") as run:
    # Indicate the artifact to use. Format is "name:alias"
    artifact = run.use_artifact("<artifact_name>:<alias>")

    # Download a specific file or sub-folder
    artifact.download(path_prefix="<file_name>") # downloads only the specified file or folder
```

## Download files or entire artifacts

```python theme={null}
"""
Downloads files or entire artifacts from W&B. The same
logic applies to external artifacts.
"""

import wandb    

with wandb.init(project="<project>") as run:
    # Indicate the artifact to use. Format is "name:alias"
    artifact = run.use_artifact("<artifact_name>:<alias>")

    # Downloads file from the artifact at path name
    # If artifact.add_reference() was used, returns the reference URL
    entry = artifact.get_entry("<file_name>")

    # Download the entire artifact
    datadir = artifact.download()
```

## Add one or more aliases to an artifact when logging it

```python theme={null}
"""
Add one or more aliases to an artifact when logging it to W&B.
"""
import wandb

# Create an artifact
artifact = wandb.Artifact(name="<artifact_name>", type="<artifact_type>")
# Add files to the artifact
artifact.add_file("<file_path>")

with wandb.init(project="<project>") as run:
    # Log the artifact with aliases
    run.log_artifact(artifact, aliases=["<alias1>", "<alias2>"])
```

## Add an alias to an existing artifact

```python theme={null}
"""
Adds an alias to an existing W&B artifact.
"""
import wandb

# Retrieve an existing artifact and add an alias to it
artifact = wandb.Api().artifact("entity/project/artifact:version")
artifact.aliases = ["<new-alias>"]
artifact.save()
```

## Add a tag to an artifact when logging it

```python theme={null}
"""
Add a tag to an artifact when logging it to W&B.
"""
import wandb

# Create an artifact
artifact = wandb.Artifact(name="<artifact_name>", type="<artifact_type>")

# Log the artifact with tags
with wandb.init(project="<project>") as run:
    run.log_artifact(artifact, tags=["<tag1>", "<tag2>"])
```

## Add a tag to an existing artifact

```python theme={null}
"""
Adds a tag to an existing W&B artifact.
"""
import wandb

# Retrieve an existing artifact and add a tag to it
artifact = wandb.Api().artifact("entity/project/artifact:version")
artifact.tags = ["new-tag"]
artifact.save()
```

## Create and log an artifact with a TTL policy

```python theme={null}
"""
Create and log an artifact with a TTL policy in W&B.
"""
import wandb
from datetime import timedelta

# Create an artifact with TTL policy
artifact = wandb.Artifact(name="<artifact_name>", type="<artifact_type>")
artifact.add_file("<file_path>")
artifact.ttl = timedelta(days=30)  # Set TTL policy

with wandb.init(project="<project>", entity="<entity>") as run:
    # Log the artifact with TTL
    run.log_artifact(artifact)
```

## Update the TTL policy of an existing artifact

```python theme={null}
"""
Update the TTL policy of an existing artifact in W&B.
"""
import wandb
from datetime import timedelta

# Initialize the W&B API
api = wandb.Api()

# Retrieve the existing artifact
artifact = api.artifact("<entity/project/artifact:alias>")

# Add or update the TTL policy. Specify the desired duration.
artifact.ttl = timedelta(days=365)  # Delete in one year

# Save the updated artifact
artifact.save()
```

## Update an existing artifact's description within a run

```python theme={null}
"""
Update an existing W&B artifact's description within a W&B run.

This code initializes a W&B run, retrieves the specified artifact by name and alias,
updates its description, and saves the changes.
"""
import wandb

with wandb.init(entity="<entity>", project="<project>") as run:
    # Retrieve the artifact by name and alias
    artifact = run.use_artifact(artifact_or_name="<artifact>:<alias>")
    # Update the artifact's description
    artifact.description = "<description>"
    # Save the updated artifact
    artifact.save()
```

## Given an existing artifact, update its description, metadata, and aliases without creating a new run

```python theme={null}
"""
Given an existing artifact, update its description, metadata, and aliases
without creating a new run.
"""
import wandb

api = wandb.Api()

artifact = api.artifact(name="<entity/project/artifact:alias>")

# Update the description
artifact.description = "My new description"

# Selectively update metadata keys
artifact.metadata["oldKey"] = "new value"

# Replace the metadata entirely
artifact.metadata = {"newKey": "new value"}

# Add an alias
artifact.aliases.append("best")

# Remove an alias
artifact.aliases.remove("latest")

# Completely replace the aliases
artifact.aliases = ["replaced"]

# Persist all artifact modifications
artifact.save()
```


# Logging
Source: https://docs.wandb.ai/models/ref/sdk-coding-cheat-sheet/logging

Log metrics, hyperparameters, tables, and custom data to W&B.

Log metrics, hyperparameters, tables, and custom data to W\&B.

## Log a CSV file to as a table

```python theme={null}
"""
Log a CSV file to W&B as a table.

Replace:
- values enclosed in angle brackets with your own
- <name>.csv in pd.read_csv() with the name of your CSV file
"""

import wandb
import pandas as pd

# Read CSV as a DataFrame object (pandas)
dataframe = pd.read_csv("<name>.csv")

# Convert the DataFrame into a W&B Table
table = wandb.Table(dataframe=dataframe)

# Start a W&B run to log data
with wandb.init(project="<project>") as run:

    # Log the table to visualize it in the W&B UI
    run.log({"<table_name>": table})
```

## Log a custom summary metric

```python theme={null}
"""
Log a custom summary metric to W&B.
"""
import wandb
import random 

with wandb.init(project="<project>") as run:
    # Log a custom summary metric with a random integer value between 1 and 10
    run.summary["<metric_name>"] = random.randint(1, 10)
```

## Log a custom summary metric

```python theme={null}
"""
Log a custom summary metric to W&B.

Replace:
- values enclosed in angle brackets with your own
- <summary_function_a> and <summary_function_b> in run.define_metric() with
    one of "max", "min", "last", "mean", "best", or "none".
"""
import wandb
import random

with wandb.init() as run:
    # summary_function_a and summary_function_b summary values for metric_name_a
    run.define_metric(name="<metric_name_a>", summary="<summary_function_a>")
    run.define_metric(name="<metric_name_a>", summary="<summary_function_b>")

    for i in range(10):
        log_dict = {
            "metric_name_a": random.uniform(0, 1 / (i + 1)),
        }
        run.log(log_dict)
```

## Download and log an existing artifact from a registry collection

```python theme={null}
"""
Download and log an existing artifact from a W&B registry collection.
Replace the placeholders with actual registry, collection names, entity,
project, and version.
"""
import wandb

# Construct the full artifact name with version
registry_name = "<registry_name>"  # Specify the registry name
collection_name = "<collection_name>" # Specify the collection name
version = 0 # Specify the version of the artifact to use
artifact_name_registry = f"wandb-registry-{registry_name}/{collection_name}:v{version}"

# Initialize a W&B run in the different team and project
with wandb.init(entity="<entity>", project="<project>") as run:
    # Use the model artifact from the registry
    registry_model = run.use_artifact(artifact_or_name=artifact_name_registry)

    # Download the model to a local directory
    local_model_path = registry_model.download()
```

## Log a histogram plot

```python theme={null}
"""
Log a histogram plot to W&B. 

Replace:
- values enclosed in angle brackets with your own
- [[1, 2], [2, 3]] in wandb.Table(data=) with your own 2D row-oriented array of values to plot

Pass a 2D row-oriented array of values to data (wandb.Table(data=)), with
column names specified in the `columns` parameter.
"""
import wandb

# Start a new run
with wandb.init(entity="<entity>", project="<project>") as run:
        
    # Create a table with the columns to plot
    table = wandb.Table(data=[[1, 2], [2, 3]] , columns=["<a_column>", "<b_column>"])

    # Use the table to populate various custom charts
    histogram = wandb.plot.histogram(table, value='<b_column>', title='Histogram')

    # Log custom tables, which will show up in customizable charts in the UI
    run.log({'histogram_1': histogram})
```

## Initialize a run and log hyperparameters

```python theme={null}
"""Initializes a W&B run and logs hyperparameters."""
import wandb

config = {
    "learning_rate": 0.01,
    "batch_size": 32,
    "optimizer": "adam",
}

with wandb.init(project="<project>", config=config) as run:
    # Training and logging code goes here
    pass
```

## Log a line plot

```python theme={null}
"""
Log a line plot to W&B. 

Replace:
- values enclosed in angle brackets with your own
- [[1, 2], [2, 3]] in wandb.Table(data=) with your own 2D row-oriented array of values to plot

Pass a 2D row-oriented array of values to data (wandb.Table(data=)), with
column names specified in the `columns` parameter.

Column names must match the x and y parameters in the `wandb.plot.line()` plotting function.
"""
import wandb

# Start a new run
with wandb.init(entity="<entity>", project="<project>") as run:
        
    # Create a table with the columns to plot
    table = wandb.Table(data=[[1, 2], [2, 3]], columns=["<column_a>", "<column_b>"])

    # Use the table to populate various custom charts
    line_plot = wandb.plot.line(table, x='<column_a>', y='<column_b>', title='<title>')
    
    # Log custom tables, which will show up in customizable charts in the UI
    run.log({'<line_plot>': line_plot})
```

## Initialize a run and log a metric

```python theme={null}
"""Initializes a W&B run and logs a metric."""
import wandb

with wandb.init(project="<project>") as run:
    # Training and logging code goes here

    # Example of logging a metric
    run.log({"accuracy": 0.95})
```

## Log a scatter plot

```python theme={null}
"""
Log a scatter plot to W&B. 

Replace:
- values enclosed in angle brackets with your own
- [[1, 2], [2, 3]] in wandb.Table(data=) with your own 2D row-oriented array of values to plot

Pass a 2D row-oriented array of values to data (wandb.Table(data=)), with
column names specified in the `columns` parameter.

Column names must match the x and y parameters in the `wandb.plot.scatter()` plotting function.
"""
import wandb

# Start a new run
with wandb.init(entity="<entity>", project="<project>") as run:
        
    # Create a table with the columns to plot
    table = wandb.Table(data=[[1, 2], [2, 3]], columns=["<a_column>", "<b_column>"])

    # Use the table to populate various custom charts
    scatter = wandb.plot.scatter(table, x='<a_column>', y='<b_column>', title='<title>')

    # Log custom tables, which will show up in customizable charts in the UI
    run.log({'scatter_1': scatter})
```

## Explicitly log a summary metric

```python theme={null}
"""
Explicitly log a summary metric to W&B.
"""
import wandb
import random 

with wandb.init(project="<project>") as run:
    # Log a custom summary metric with a random integer value between 1 and 10
    run.summary["<metric_name>"] = random.randint(1, 10)
```

## Log a table

```python theme={null}
"""
Log a table to W&B.

Replace: 
- values enclosed in angle brackets with your own
- [[1, 2], [2, 3]] in wandb.Table(data=) with your own 2D row-oriented array of values to log
"""
import wandb

# Create a table object with two columns and two rows of data
my_table = wandb.Table(
    columns=["<a_column>", "<b_column>"],
    data=[[1, 2], [2, 3]],
    log_mode="<log_mode>"
    )

# Start a new run
with wandb.init(entity="<entity>", project="<project>") as run:
    # Log the table to W&B
    run.log({"<table_name>": my_table})
```


# Registry
Source: https://docs.wandb.ai/models/ref/sdk-coding-cheat-sheet/registry

Work with W&B Model Registry to organize and manage model versions.

Work with W\&B Model Registry to organize and manage model versions.

## Add a description to a collection in a registry

```python theme={null}
"""
Add an description to a collection in a registry.
"""
import wandb

# Define registry and collection details
collection_type = "<collection_type>"
registry_name = "<registry_name>"
collection_name = "<collection_name>"

# Construct the full registry path
registry_path = f"wandb-registry-{registry_name}/{collection_name}"

# Initialize W&B API
api = wandb.Api()

# Retrieve the artifact collection
collection = api.artifact_collection(
  type_name = collection_type, 
  name = registry_path
  )

# Add description annotation to the collection object
collection.description = "<description>"

# Save the updated collection
collection.save()
```

## Create a new registry

```python theme={null}
"""
Create a new registry in W&B. If the registry does not exist, W&B creates it.
"""
import wandb

# Initialize W&B API
api = wandb.Api()

# Create a new registry
registry = api.create_registry(
    name="<registry_name>",
    visibility="<visibility>",  # e.g., "public" or "private"
)
```

## Delete a registry

```python theme={null}
"""
Delete a registry from W&B.
"""
import wandb

# Define registry and collection details
registry_name = "<registry_name>"
collection_name = "<collection_name>"

# Construct the full registry path
registry_path = f"wandb-registry-{registry_name}/{collection_name}"

# Initialize the W&B API
api = wandb.Api()

# Fetch the registry you want to delete
fetched_registry = api.registry("<registry_name>")

# Deleting a registry
fetched_registry.delete()
```

## Create an artifact and link it to a collection in a registry

```python theme={null}
"""
Create a W&B artifact and link it to a collection in a registry. If the
collection does not exist, W&B creates it.
"""
import wandb

# Create an artifact object
artifact = wandb.Artifact(name = "<artifact_name>", type = "<artifact_type>")

# Define registry and collection names
registry_name = "<registry_name>"
collection_name = "<collection_name>"
target_path = f"wandb-registry-{registry_name}/{collection_name}"

# Initialize a run
with wandb.init(entity = "<entity>", project = "<project>") as run:
  # Link the artifact to a collection. If the collection does not exist, W&B creates it.
  run.link_artifact(artifact = artifact, target_path = target_path)
```

## Retrieve a specific version of an artifact from a registry collection

```python theme={null}
"""
Retrieve a specific version of an artifact from a registry collection.
"""

import wandb

# Define registry, collection, and artifact version details
registry = "<registry_name>"
collection = "<collection_name>"
version = "<version>"

# Construct the full artifact name with version
artifact_name = f"wandb-registry-{registry}/{collection}:{version}"

# Initialize W&B API
api = wandb.Api()

# Retrieve the artifact from the specified registry collection and version
artifact = api.artifact(name = artifact_name)
```

## Create a registry collection and link an artifact to it

```python theme={null}
"""
Ceates a W&B registry collection and links an artifact to it.
"""
import wandb

# Create an artifact object
artifact = wandb.Artifact(name = "<artifact_name>", type = "<artifact_type>")

registry_name = "<registry_name>"
collection_name = "<collection_name>"
registry_path = f"wandb-registry-{registry_name}/{collection_name}"

# Initialize a run
with wandb.init(entity = "<entity>", project = "<project>") as run:

  # Link the artifact to a collection. If the collection does not exist, W&B creates it.
  run.link_artifact(artifact = artifact, target_path = registry_path)
```

## Add tags to a collection in a registry

```python theme={null}
"""
Add tags to a collection in a registry.
"""
import wandb

# Define registry and collection details
collection_name = "<collection_name>"
collection_type = "<collection_type>"
registry_name = "<registry_name>"

# Construct the full registry path
registry_path = f"wandb-registry-{registry_name}/{collection_name}"

# Retrieve the artifact collection
collection = wandb.Api().artifact_collection(
  type_name = collection_type, 
  name = registry_path
  )

collection.tags = ["<tag>"]
collection.save()
```

## Remove a tag from a collection in a registry

```python theme={null}
"""
Remove a tag from a collection in a registry.
"""
import wandb

# Define registry and collection details
collection_name = "<collection_name>"
collection_type = "<collection_type>"
registry_name = "<registry_name>"

# Construct the full registry path
registry_path = f"wandb-registry-{registry_name}/{collection_name}"

# Retrieve the artifact collection
collection = wandb.Api().artifact_collection(
  type_name = collection_type, 
  name = registry_path
  )

collection.tags.remove("<tag>")
collection.save()
```


# Runs
Source: https://docs.wandb.ai/models/ref/sdk-coding-cheat-sheet/runs

Initialize and manage W&B runs to organize your experiments and track your work.

Initialize and manage W\&B runs to organize your experiments and track your work.

## Create an experiment

```python theme={null}
"""
Create an experiment in W&B. If the project does not exist, W&B creates it.

Note that this file only initializes the experiment; you can add code to log metrics,
artifacts, etc., within the `with` block.
"""
import wandb

# Initialize a W&B run
with wandb.init(project="<project>") as run:
    # Experiment code goes here
    pass
```

## Fork an existing run from a specific step

```python theme={null}
"""Fork an existing W&B run from a specific step."""

import wandb

# Initialize a run to be forked later
with wandb.init(project="<project>", entity="<entity>") as original_run:
    # Training and logging code goes here.
    pass

# Fork the run from a specific step. In this example, we fork from step 200. You can adjust the step number as needed.
with wandb.init(project="<project>",entity="<entity>", fork_from=f"{original_run.id}?_step=200") as forked_run:
    # Training and logging code goes here.
    pass
```

## Initialize a run

```python theme={null}
"""
Initializes a W&B run.

W&B automatically creates the project if it does not exist. Note that this
file only initializes the experiment; you can add code to log metrics,
artifacts, etc., within the `with` block.
"""
import wandb

# Note the usage of `with` statement to ensure proper resource management.
with wandb.init(project="<project>") as run:
    # Training and logging code goes here
    pass
```

## Set resume behaviour for a run if it is paused, stops, or fails

```python theme={null}
"""
Set resume behaviour for a run if it is paused, stops, or fails. 

This is useful for long-running experiments that may be interrupted. See
reference documentation for wandb.init(resume=) for a list of all resume options.
"""
import wandb

# Initialize a W&B run with resume behavior
with wandb.init(project="<project>", resume="<resume_option>") as run:
    # Training and logging code goes here
    pass
```

## Rewind a run to modify the history of a run

```python theme={null}
"""
Rewind a run to modify the history of a run.

Specify the run ID and step to rewind to with the `resume_from` parameter in `wandb.init()`.
"""
import wandb
import math

# Initialize a run and log some metrics
with wandb.init(project="<project>", entity="<entity>") as run:
    # Training and logging code goes here
    pass

run_ID = "<run_id>" # Replace with run ID of the original run
step = int("<step>") # Specify the step to rewind to

# Start a new run that resumes from the specified step of the original run
with wandb.init(project="<project>", entity="<entity>", resume_from=f"{run_ID}?_step={step}") as run:
    # Training and logging code goes here
    pass    
```

## Add one or more tags to a run

```python theme={null}
"""
Add one or more tags to a run.
"""
import wandb

with wandb.init(entity="<entity>", project="<project>", tags=["<tag1>", "<tag2>"]) as run:
    # Training and logging code goes here
    pass
```

## Add one or more runs to a group

```python theme={null}
"""
Add one or more runs to a group.

Pass the name of your group as an argument to the `group` parameter when you
initialize a run with `wandb.init(group="")`.
"""
import wandb

entity = "<entity>"
project = "<project>"

# The following creates 3 groups of runs, with 3 runs in each group. 
for group in ["<GroupA>", "<GroupB>", "<GroupC>"]: # Replace with your desired group names
    # Simulate creating three runs for each group.
    for i in range(3):
        with wandb.init(entity=entity, project=project, group=group, name=f"{group}_run_{i}") as run:
            # Training and logging code goes here
            pass    
```

## Organize runs by their job type

```python theme={null}
"""
Organize runs by their job type.

Add a job type to a run by passing the `job_type` parameter to `wandb.init(job_type="")`.
"""
import wandb

entity = "<entity>"
project = "<project>"

# Creates runs and organizes them by job type.
for job_type in ["<JobType1>", "<JobType2>"]: # Replace with job type names
    # Simulate creating two runs for each job type.
    for i in range(2):
        with wandb.init(entity=entity, project=project, job_type=job_type, name=f"{job_type}_run_{i}") as run:
            # Training and logging code goes here
            pass    
```

## Add one or more tags to an active run

```python theme={null}
"""
Add one or more tags to an active run.

A run object's `tags` property is a tuple. To add a new tag to an existing run,
update the `tags` property with a new tuple that includes the existing tags and the new tag(s).
"""

import wandb

with wandb.init(entity="<entity>", project="<project>", tags=["<tag1>", "<tag2>"]) as run:
    # Training and logging code goes here

    # Add a new tag to the existing tags by creating a new tuple that includes the existing tags and the new tag.
    run.tags += ("<tag3>",)
```

## Add one or more tags to previously saved runs

```python theme={null}
"""
Add one or more tags to previously saved runs.

Use the Public API to update tags on stored data after the run has finished or
when you are working outside the run process.

To add a new tag to a run, update the "tags"
property with a new list that includes the existing tags and the new tag(s).
The run's path consists of entity/project/run_id
After updating the tags, call the run's `update()` method to save the changes.
"""
import wandb

entity = "<entity>"
project = "<project>"
run_id = "<run-id>"  # Replace with the ID of the run you want to update

# Path consists of entity/project/run_id
with wandb.Api().run(f"{entity}/{project}/{run_id}") as run:
  run.tags.append("<tag>")
  run.update()
```


# Use the W&B MCP Server
Source: https://docs.wandb.ai/platform/mcp-server

Connect your IDE or AI agent to the W&B Model Context Protocol (MCP) server to query and analyze your W&B data and documentation in natural language.

Model Context Protocol (MCP) is an open standard that lets AI agents call external tools. The W\&B MCP Server gives your IDE, coding assistant, or chat agent direct access to your W\&B data and documentation. With this access, your agent can answer questions about your runs, traces, evaluations, and artifacts without copy-paste. For a full list of what you can do with the server, see the [W\&B MCP Server capabilities](#w\&b-mcp-server-capabilities) section.

It integrates directly with most IDEs, coding clients, and chat agents, including:

* Cursor
* Visual Studio Code (VS Code)
* Claude Code
* Codex
* Gemini CLI
* Mistral LeChat
* Claude Desktop

## Deployment types

The W\&B MCP Server is available in two deployment options. Use the hosted server for the fastest setup, or set up a local version if you need more isolation and flexibility. The local version requires your client to use a different URL to access the server.

<CardGroup>
  <Card title="Hosted server (recommended)">
    A W\&B-managed MCP server that your client connects to over HTTP with your W\&B API key. No installation, no local process to maintain.

    [Use the hosted server](#use-the-hosted-server)
  </Card>

  <Card title="Local install">
    Run the MCP server on your own machine over STDIO or HTTP. Use when you need air-gapped operation, pinning to a specific release, custom server behavior, active server development, or support for a client that only speaks STDIO.

    [Run the MCP server locally](#run-the-mcp-server-locally)
  </Card>
</CardGroup>

<Tip>
  If you run W\&B on Dedicated Cloud or Self-Managed and the hosted MCP server isn't yet enabled on your instance, contact [W\&B support](mailto:support@wandb.com) or your W\&B account team to request it.
</Tip>

## Prerequisites

Before you configure any client, make sure you have the following in place:

* Create a W\&B API key at [wandb.ai/authorize](https://wandb.ai/authorize).
* Set the key as the `WANDB_API_KEY` environment variable, or pass it to your client as a bearer token.
* For Dedicated Cloud, Self-Managed, and local installs against a non-default instance, set the `WANDB_BASE_URL` environment variable to your instance URL.

## Use the hosted server

W\&B runs a managed MCP server for every deployment type. You don't need to install anything. Configure your client to connect over HTTP with a W\&B API key in the `Authorization` header.

### Connection URL

The URL depends on your type of W\&B deployment:

| Deployment         | Server URL                      |
| ------------------ | ------------------------------- |
| Multi-tenant Cloud | `https://mcp.withwandb.com/mcp` |
| Dedicated Cloud    | `https://[YOUR-INSTANCE]/mcp`   |
| Self-Managed       | `https://[YOUR-INSTANCE]/mcp`   |

For Dedicated Cloud or Self-Managed, replace `https://mcp.withwandb.com/mcp` with `https://[YOUR-INSTANCE]/mcp` and keep everything else the same. The following client configurations use the Multi-tenant URL.

<Tabs>
  <Tab title="Claude Code">
    Register the W\&B MCP server with Claude Code, replacing the bearer token with your W\&B API key:

    ```bash theme={null}
    claude mcp add --transport http wandb https://mcp.withwandb.com/mcp \
      --header "Authorization: Bearer [YOUR-WANDB-API-KEY]"
    ```

    Add `--scope user` to configure Claude Code globally. Omit it to configure only the current project.

    Verify the connection by asking `List my W&B entities.` The agent should call `list_entities_tool` and return your username and any teams. If the connection fails, see [Troubleshooting](#troubleshooting). For more information, see [Claude Code's MCP documentation](https://docs.anthropic.com/en/docs/claude-code/mcp).
  </Tab>

  <Tab title="Claude Desktop">
    Claude Desktop's built-in custom connectors interface doesn't support API key authentication for remote MCP servers. To work around this, use the [`mcp-remote`](https://www.npmjs.com/package/mcp-remote) npm proxy to connect Claude Desktop to the W\&B remote MCP server. The proxy runs locally and forwards requests to `https://mcp.withwandb.com/mcp` with your bearer token.

    You need [Node.js](https://nodejs.org/) installed on your system.

    Open your Claude Desktop config file in a text editor. You can find the config file at the following location for your OS:

    * **macOS**: `~/Library/Application\ Support/Claude/claude_desktop_config.json`
    * **Windows**: `%APPDATA%\Claude\claude_desktop_config.json`

    Add the following to the JSON object in your config file, replacing `[YOUR-WANDB-API-KEY]` with your W\&B API key:

    ```json theme={null}
    {
      "mcpServers": {
        "wandb": {
          "command": "npx",
          "args": [
            "-y",
            "mcp-remote",
            "https://mcp.withwandb.com/mcp",
            "--header",
            "Authorization:${AUTH_HEADER}"
          ],
          "env": {
            "AUTH_HEADER": "Bearer [YOUR-WANDB-API-KEY]"
          }
        }
      }
    }
    ```

    The full header value is set through the `env` field rather than directly in `args` to work around a space-escaping issue in some Claude Desktop versions.

    Restart Claude Desktop to activate the new configuration. Verify the connection by asking `List my W&B entities.` The agent should call `list_entities_tool` and return your username and any teams. If the connection fails, see [Troubleshooting](#troubleshooting).
  </Tab>

  <Tab title="Codex">
    Export your W\&B API key as an environment variable, then register the server with Codex:

    ```bash theme={null}
    export WANDB_API_KEY=[YOUR-WANDB-API-KEY]
    codex mcp add wandb \
      --url https://mcp.withwandb.com/mcp \
      --bearer-token-env-var WANDB_API_KEY
    ```

    Verify the connection by asking `List my W&B entities.` The agent should call `list_entities_tool` and return your username and any teams. If the connection fails, see [Troubleshooting](#troubleshooting).
  </Tab>

  <Tab title="Cursor">
    Install the server in Cursor automatically with the [one-click installation link](https://cursor.com/en/install-mcp?name=wandb\&config=eyJ0cmFuc3BvcnQiOiJodHRwIiwidXJsIjoiaHR0cHM6Ly9tY3Aud2l0aHdhbmRiLmNvbS9tY3AiLCJoZWFkZXJzIjp7IkF1dGhvcml6YXRpb24iOiJCZWFyZXIge3tXQU5EQl9BUElfS0VZfX0iLCJBY2NlcHQiOiJhcHBsaWNhdGlvbi9qc29uLCB0ZXh0L2V2ZW50LXN0cmVhbSJ9fQ%3D%3D), then replace the placeholder with your W\&B API key in the `Authorization` field.

    To configure Cursor manually:

    1. On macOS, open **Cursor** > **Settings** > **Cursor Settings**. On Windows or Linux, open **Preferences** > **Settings** > **Cursor Settings**.

    2. Select **Tools and MCP**.

    3. In **Installed MCP Servers**, select **Add Custom MCP**. Cursor opens the `mcp.json` configuration file.

    4. Add the following to the `mcpServers` object:

       ```json theme={null}
       {
         "mcpServers": {
           "wandb": {
             "transport": "http",
             "url": "https://mcp.withwandb.com/mcp",
             "headers": {
               "Authorization": "Bearer [YOUR-WANDB-API-KEY]",
               "Accept": "application/json, text/event-stream"
             }
           }
         }
       }
       ```

    5. Restart Cursor.

    6. Verify the connection by asking `List my W&B entities.` The agent should call `list_entities_tool` and return your username and any teams.

    If the connection fails, see [Troubleshooting](#troubleshooting). For more information, see [Cursor's MCP documentation](https://cursor.com/docs/context/mcp).
  </Tab>

  <Tab title="Gemini CLI">
    Install the W\&B MCP extension:

    ```bash theme={null}
    gemini extensions install https://github.com/wandb/wandb-mcp-server
    ```

    Restart Gemini CLI. Verify the connection by asking `List my W&B entities.` The agent should call `list_entities_tool` and return your username and any teams.

    If the connection fails, see [Troubleshooting](#troubleshooting). For more information, see [Gemini CLI's MCP documentation](https://geminicli.com/docs/tools/mcp-server/).
  </Tab>

  <Tab title="Mistral LeChat">
    1. In LeChat, open the **Intelligence** menu and select **Add Connector**.
    2. Select **Custom MCP Connector**.
    3. Configure the following fields:
       * **Connector Server**: `https://mcp.withwandb.com/mcp`
       * **Description**: (Optional) A short description.
       * **Authentication Method**: Select **API Token Authentication**.
       * **Header name**: Leave as `Authorization`.
       * **Header type**: Select **Bearer**.
       * **Header value**: Your W\&B API key.
    4. Select **Create**.
    5. Verify the connection by asking `List my W&B entities.` The agent should call `list_entities_tool` and return your username and any teams.

    If the connection fails, see [Troubleshooting](#troubleshooting). For more information, see [LeChat's MCP documentation](https://mistral.ai/news/le-chat-mcp-connectors-memories).
  </Tab>

  <Tab title="OpenAI Responses API">
    Add the server to the `tools` field of your OpenAI Responses API call:

    ```python theme={null}
    import os
    from openai import OpenAI

    client = OpenAI()

    resp = client.responses.create(
        model="gpt-4o",
        tools=[{
            "type": "mcp",
            "server_label": "wandb",
            "server_description": "Query W&B data",
            "server_url": "https://mcp.withwandb.com/mcp",
            "authorization": os.getenv("WANDB_API_KEY"),
            "require_approval": "never",
        }],
        input="List my W&B entities.",
    )

    print(resp.output_text)
    ```

    Pass the raw W\&B API key as the `authorization` value. OpenAI prepends `Bearer ` when it calls the server, so don't include it yourself. The OpenAI MCP integration runs server-side, so it can't reach a local MCP server. For local development, see [Run the MCP server locally](#run-the-mcp-server-locally).
  </Tab>

  <Tab title="VS Code">
    Open your global or workspace `mcp.json` (for example, `~/.vscode/mcp.json` or `.vscode/mcp.json`) and add the following:

    ```json theme={null}
    {
      "servers": {
        "wandb": {
          "type": "http",
          "url": "https://mcp.withwandb.com/mcp",
          "headers": {
            "Authorization": "Bearer [YOUR-WANDB-API-KEY]"
          }
        }
      }
    }
    ```

    Restart VS Code, confirm the server appears in the MCP panel, and verify the connection by asking `List my W&B entities.` The agent should call `list_entities_tool` and return your username and any teams.

    If the connection fails, see [Troubleshooting](#troubleshooting).
  </Tab>
</Tabs>

## Run the MCP server locally

A local install is an alternative to the hosted server, not the default for any deployment type. Use it when the hosted server doesn't fit your setup.

Common reasons to run locally:

* **Air-gapped or offline environments** where your client can't reach a hosted W\&B endpoint.
* **Pinned version**. The hosted server follows the main branch. A local install can pin to a specific release tag.
* **Custom server behavior** such as changing tool descriptions, adding tools, or setting a non-default response token budget.
* **Active development** on the server itself.
* **STDIO-only clients** or clients that require a local process.

For Dedicated Cloud or Self-Managed users, prefer the hosted path. Only use a local install from [wandb/wandb-mcp-server](https://github.com/wandb/wandb-mcp-server) if the hosted server isn't yet enabled on your instance or one of the preceding reasons applies. Set a `WANDB_BASE_URL` environment variable to your instance URL.

### Local prerequisites

To run the server locally, make sure you have the following:

* Python 3.11 or higher.
* [`uv`](https://docs.astral.sh/uv/) or `pip`.
* A W\&B API key, set as `WANDB_API_KEY`.
* `WANDB_BASE_URL` set to your instance URL if you use Dedicated Cloud or Self-Managed.

### Install the server

Choose an install method and run the following command to install the MCP server:

<Tabs>
  <Tab title="uvx (no permanent install)">
    ```bash theme={null}
    uvx --from git+https://github.com/wandb/wandb-mcp-server wandb_mcp_server
    ```
  </Tab>

  <Tab title="uv">
    ```bash theme={null}
    uv pip install wandb-mcp-server
    ```
  </Tab>

  <Tab title="pip">
    ```bash theme={null}
    pip install wandb-mcp-server
    ```
  </Tab>

  <Tab title="Install from GitHub">
    ```bash theme={null}
    pip install git+https://github.com/wandb/wandb-mcp-server
    ```
  </Tab>
</Tabs>

### Configure your client

After installing the server, configure your client to launch it. Select your MCP client and then run the following configuration, replacing `[YOUR-WANDB-API-KEY]` with your W\&B API key as necessary:

<Tabs>
  <Tab title="Claude Code">
    Register the local server with Claude Code. Add `--scope user` for a global configuration.

    ```bash theme={null}
    claude mcp add wandb \
      -e WANDB_API_KEY=[YOUR-WANDB-API-KEY] \
      -e WANDB_BASE_URL=https://your-wandb-instance.example.com \
      -- uvx --from git+https://github.com/wandb/wandb-mcp-server wandb_mcp_server
    ```
  </Tab>

  <Tab title="Claude Desktop">
    Open your Claude Desktop config file:

    * **macOS**: `~/Library/Application Support/Claude/claude_desktop_config.json`
    * **Windows**: `%APPDATA%\Claude\claude_desktop_config.json`

    Add the following JSON. Use the full path to `uvx` because Claude Desktop may not find it on your `PATH` otherwise.

    ```json theme={null}
    {
      "mcpServers": {
        "wandb": {
          "command": "/full/path/to/uvx",
          "args": [
            "--from",
            "git+https://github.com/wandb/wandb-mcp-server",
            "wandb_mcp_server"
          ],
          "env": {
            "WANDB_API_KEY": "[YOUR-WANDB-API-KEY]",
            "WANDB_BASE_URL": "https://your-wandb-instance.example.com"
          }
        }
      }
    }
    ```

    Restart Claude Desktop to apply the configuration.
  </Tab>

  <Tab title="Codex">
    ```bash theme={null}
    codex mcp add wandb \
      --env WANDB_API_KEY=[YOUR-WANDB-API-KEY] \
      --env WANDB_BASE_URL=https://your-wandb-instance.example.com \
      -- uvx --from git+https://github.com/wandb/wandb-mcp-server wandb_mcp_server
    ```
  </Tab>

  <Tab title="Cursor">
    Add the following to your `mcp.json` configuration:

    ```json theme={null}
    {
      "mcpServers": {
        "wandb": {
          "command": "uvx",
          "args": [
            "--from",
            "git+https://github.com/wandb/wandb-mcp-server",
            "wandb_mcp_server"
          ],
          "env": {
            "WANDB_API_KEY": "[YOUR-WANDB-API-KEY]",
            "WANDB_BASE_URL": "https://your-wandb-instance.example.com"
          }
        }
      }
    }
    ```

    Omit `WANDB_BASE_URL` to use the default W\&B API endpoint.
  </Tab>

  <Tab title="VS Code">
    Add the following to your `.vscode/mcp.json` or global MCP configuration:

    ```json theme={null}
    {
      "servers": {
        "wandb": {
          "command": "uvx",
          "args": [
            "--from",
            "git+https://github.com/wandb/wandb-mcp-server",
            "wandb_mcp_server"
          ],
          "env": {
            "WANDB_API_KEY": "[YOUR-WANDB-API-KEY]",
            "WANDB_BASE_URL": "https://your-wandb-instance.example.com"
          }
        }
      }
    }
    ```
  </Tab>
</Tabs>

### Run the server with HTTP transport

For web-based clients and for testing, run the server with HTTP transport:

```bash theme={null}
uvx wandb_mcp_server --transport http --host 0.0.0.0 --port 8080
```

To expose a local server to external clients, such as the OpenAI Responses API, use a tunnel:

```bash theme={null}
uvx wandb_mcp_server --transport http --port 8080

# In another terminal
ngrok http 8080
```

Update your MCP client configuration to use the tunnel URL.

### Environment variables

The following environment variables control authentication, instance routing, and server behavior for local installs. Set them in your client's `env` block or export them in your shell.

| Variable               | Description                                                                                       |
| ---------------------- | ------------------------------------------------------------------------------------------------- |
| `WANDB_API_KEY`        | W\&B API key for authentication. Required.                                                        |
| `WANDB_BASE_URL`       | Custom W\&B instance URL for Dedicated Cloud or Self-Managed. Defaults to `https://api.wandb.ai`. |
| `WANDB_MCP_PROXY_DOCS` | Enable the `search_wandb_docs_tool` documentation search proxy. Default: `true`.                  |
| `WANDBOT_BASE_URL`     | Custom endpoint for the docs search proxy.                                                        |
| `MAX_RESPONSE_TOKENS`  | Token budget for tool-response truncation. Default: `30000`.                                      |
| `MCP_SERVER_LOG_LEVEL` | Logging verbosity. One of `DEBUG`, `INFO`, `WARNING`, `ERROR`.                                    |

For the complete command-line reference and advanced options, see the [wandb-mcp-server README](https://github.com/wandb/wandb-mcp-server#readme).

## W\&B MCP Server capabilities

Use the MCP server to analyze experiments, debug traces, create reports, manage registry and artifacts, and answer questions from the W\&B docs. The following example prompts demonstrate some of the tasks you can ask your agent to perform when it's connected to the W\&B MCP Server:

* "Show me the top 5 runs by `eval/accuracy` in `your-team/your-project`."
* "How did the latency of my hiring agent's predict traces evolve over the last month?"
* "Generate a W\&B report comparing decisions made by the hiring agent last week."
* "What versions of the `production-model` artifact exist, and what changed between `v2` and `v3`?"
* "How do I create a leaderboard in Weave?"

### Available tools

The server offers several tools grouped by purpose. The following table lists each tool's name, when the agent should use it, and a concrete prompt you can use to invoke that tool.

<Tabs>
  <Tab title="Discovery">
    Tools that help you discover project and entity names, and inspect schemas.

    | Tool                          | Use when                                                                                                 | Example prompt                                                        |
    | ----------------------------- | -------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------- |
    | `list_entities_tool`          | No entity is specified, or to enumerate the teams and accounts the API key can reach.                    | "What W\&B teams do I have access to?"                                |
    | `query_wandb_entity_projects` | The entity is known but the project name is not, or an earlier query failed with "project not found".    | "List all projects under `your-team`."                                |
    | `probe_project_tool`          | To discover available metrics, config keys, and tags in an unfamiliar run-based project.                 | "Probe `your-team/your-project` and tell me what metrics are logged." |
    | `infer_trace_schema_tool`     | To discover field names, types, and sample values in an unfamiliar Weave traces project before querying. | "What fields are on the Weave traces in `your-team/your-project`?"    |
  </Tab>

  <Tab title="Experiments and runs">
    Tools that query, compare, and diagnose W\&B Models runs.

    | Tool                   | Use when                                                                                                                                        | Example prompt                                                           |
    | ---------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------ |
    | `query_wandb_tool`     | The question is about runs, sweeps, configs, summaries, or artifacts in W\&B Models. Runs a GraphQL query.                                      | "Show me the top 5 runs by `eval/accuracy` in `your-team/your-project`." |
    | `get_run_history_tool` | The question is about training curves, metric trends over time, or any time-series logged to a run.                                             | "Plot the loss curve for run `abc123` in `your-team/your-project`."      |
    | `compare_runs_tool`    | The question is what changed between two runs, or which of two runs is better. Returns config diff, metric delta, and optional aligned history. | "Compare runs `abc123` and `def456` in `your-team/your-project`."        |
    | `diagnose_run_tool`    | The question is whether a run converged, is overfitting, or hit NaN values. Returns specific recommendations.                                   | "Is run `abc123` in `your-team/your-project` overfitting?"               |
  </Tab>

  <Tab title="Weave traces">
    Tools that query and aggregate LLM traces and evaluations.

    | Tool                        | Use when                                                                                                                                            | Example prompt                                                                        |
    | --------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------- |
    | `query_weave_traces_tool`   | Trace data is required (LLM calls, evaluations, agent runs). Start with `detail_level="summary"` and escalate to `"full"` only for specific traces. | "Show me failed traces in `your-team/your-project` from the last 24 hours."           |
    | `count_weave_traces_tool`   | The question is how many traces or how many errors, and the trace data itself isn't needed.                                                         | "How many traces failed in `your-team/your-project` this week?"                       |
    | `resolve_trace_roots_tool`  | After `query_weave_traces_tool` finds child traces, to map each back to its root session or workflow in one batched call.                           | "Find LLM calls that contain `rate limit` and tell me which sessions they belong to." |
    | `summarize_evaluation_tool` | The question is how an eval went, what the pass rate is, or which tasks fail most. Aggregates `Evaluation.evaluate` hierarchies.                    | "Summarize the most recent eval in `your-team/your-project`."                         |
  </Tab>

  <Tab title="Reports">
    Tools that persist analysis back to W\&B.

    | Tool                       | Use when                                                                                                                                        | Example prompt                                               |
    | -------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------ |
    | `create_wandb_report_tool` | An explicit request to create a report or save findings. Accepts Markdown plus a `panels` array for line plots, bar plots, and run comparisons. | "Create a W\&B report comparing runs `abc123` and `def456`." |
    | `log_analysis_to_wandb`    | Values computed in the MCP session (latency distributions, error breakdowns) need to be persisted as a run before being referenced in a report. | "Log these latency percentiles to W\&B as an analysis run."  |
  </Tab>

  <Tab title="Artifacts and registry">
    Tools that enable you to inspect and diff models, datasets, and other versioned artifacts.

    | Tool                             | Use when                                                                                              | Example prompt                                                     |
    | -------------------------------- | ----------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------ |
    | `list_registries_tool`           | The question is about model registries, registered models, or registered datasets in an organization. | "What registries exist in `your-org`?"                             |
    | `list_registry_collections_tool` | To see what models or datasets live inside a specific registry.                                       | "What collections are in the `model` registry of `your-org`?"      |
    | `list_artifact_versions_tool`    | To list available versions of a model, dataset, or other artifact collection.                         | "List versions of `production-model` in `your-team/your-project`." |
    | `get_artifact_details_tool`      | To inspect one specific artifact version, including lineage and files.                                | "What is in `production-model:v3`?"                                |
    | `compare_artifact_versions_tool` | The question is what changed between two artifact versions.                                           | "Diff `production-model:v2` and `production-model:v3`."            |
  </Tab>

  <Tab title="Docs">
    Tools that answer product questions from the official W\&B documentation.

    | Tool                     | Use when                                                                                                   | Example prompt                            |
    | ------------------------ | ---------------------------------------------------------------------------------------------------------- | ----------------------------------------- |
    | `search_wandb_docs_tool` | The question is how to use a W\&B or Weave feature or API. Proxies [docs.wandb.ai](https://docs.wandb.ai). | "How do I create a leaderboard in Weave?" |
  </Tab>
</Tabs>

### Schema-first trace queries

For Weave trace queries, call `infer_trace_schema_tool` first to discover available fields, then call `query_weave_traces_tool` with a precise column list and `detail_level`:

| `detail_level` | Returns                                    | Use when                                         |
| -------------- | ------------------------------------------ | ------------------------------------------------ |
| `schema`       | Structural fields only. Fastest.           | Browsing or counting.                            |
| `summary`      | Truncated inputs and outputs. The default. | Most analysis tasks.                             |
| `full`         | Everything untruncated.                    | Drilling into a small number of specific traces. |

This pattern keeps token usage low for broad questions and lets the agent escalate to `full` only for the traces that matter.

## Usage tips

The following sections describe practices and workflows that help you get better results from the W\&B MCP Server. Start with the general practices, then read the section that matches your workload for more specific advice and multi-step tool chains.

### General best practices

Follow these practices regardless of your use case:

* **Specify the entity and project.** MCP tools need an explicit entity (your team or personal account) and project name. Include both in every question, for example "in `your-team/your-project`".
* **Ask focused questions.** Prefer "Which eval had the highest F1 score?" to "What is my best evaluation?". Specific metrics and time ranges produce better tool calls.
* **Verify full retrieval.** For broad questions such as "What are my best performing runs?", ask the agent to confirm it retrieved all available runs rather than only the most recent ones.
* **Combine with W\&B Skills.** [W\&B Skills](/platform/wb-skills) teach coding agents how to structure W\&B workflows. Skills provide patterns and MCP provides data access, and the two work well together.

### For trace-heavy workflows

Follow these practices when working with Weave traces:

* **Start with the schema.** Call `infer_trace_schema_tool` before `query_weave_traces_tool` to provide the agent with the valid fields and filter values.
* **Pick the right `detail_level`.** Use `schema` to browse, `summary` (the default) for analysis, and `full` only when drilling into a small number of specific traces.
* **Chain `resolve_trace_roots_tool`.** After a child-trace query, pass the resulting `trace_id` list to `resolve_trace_roots_tool` to map each trace to its root session in one batched call.
* **Prefer `summarize_evaluation_tool` for evals.** It aggregates the `Evaluation.evaluate` and `predict_and_score` hierarchy automatically. Only fall back to `query_weave_traces_tool` for raw trace data.

For an end-to-end workflow, see [Triage failing LLM calls](#triage-failing-llm-calls).

### For run-heavy workflows

Follow these practices when working with W\&B Models runs:

* **Probe before you query.** Call `probe_project_tool` on an unfamiliar run-based project to discover metric keys, config keys, and tags before constructing GraphQL.
* **Use `get_run_history_tool` for time series.** GraphQL doesn't sample, so for loss curves and other time-series data `get_run_history_tool` is both faster and cheaper.
* **Let `compare_runs_tool` do the diff.** It returns config and metric deltas with aligned history in a single call, avoiding manual comparison.
* **Run a health check first.** When a training run looks wrong, call `diagnose_run_tool` before digging into history manually.

For end-to-end workflows, see [Diagnose a bad training run](#diagnose-a-bad-training-run) and [Summarize evals and compare model versions](#summarize-evals-and-compare-model-versions).

### For Dedicated Cloud and Self-Managed

Follow these practices for non-multi-tenant deployments:

* Prefer the hosted server on your instance at `https://[YOUR-INSTANCE]/mcp`. It exposes the same tools as the Multi-tenant server with no client-side `WANDB_BASE_URL` needed. Only fall back to a local install if the hosted server isn't yet enabled.
* When you do run locally against your instance, set `WANDB_BASE_URL` to your instance URL in the client's `env` block. Without it, the server targets `api.wandb.ai` and the server returns no data.
* Rate limits on Dedicated Cloud are separate from Multi-tenant. See [Dedicated Cloud rate limits](/platform/hosting/hosting-options/dedicated-cloud/rate-limits) for defaults and how to request changes.

### For local installs

Follow these practices when running the server on your own machine:

* Prefer STDIO transport for desktop clients (Cursor, VS Code, Claude Code, Claude Desktop). Only switch to HTTP transport when a client explicitly requires it (for example, the OpenAI Responses API).
* When tool calls fail silently, set `MCP_SERVER_LOG_LEVEL=DEBUG` in the client's `env` block and recheck the client's MCP logs.
* If you install from GitHub (`uvx --from git+https://github.com/wandb/wandb-mcp-server wandb_mcp_server`), `uvx` pins to the default branch. Pin an explicit tag by appending `@v0.3.2` to the Git URL when you need a stable version.

## Recommended workflows

Most real questions need more than one tool. The following workflows show common multi-step tool chains you can ask your agent to perform.

### Explore an unfamiliar project

To explore what has been logged to a project, chain these tools:

1. `list_entities_tool` to find an entity or team.
2. `query_wandb_entity_projects` to find the project.
3. `probe_project_tool` for run-based projects, or `infer_trace_schema_tool` for Weave trace projects.
4. A targeted `query_wandb_tool` or `query_weave_traces_tool` call using the discovered keys.

### Triage failing LLM calls

To find bad traces and the sessions that produced them, chain these tools:

1. `query_weave_traces_tool` with a filter on error or exception fields, and `detail_level="summary"`.
2. `resolve_trace_roots_tool` on the resulting `trace_id` list to map each failure to its root session.
3. `query_weave_traces_tool` with `detail_level="full"` on a small number of specific roots to drill in.
4. `create_wandb_report_tool` to document the findings.

### Diagnose a bad training run

To run a health check on a suspicious training run, chain these tools:

1. `get_run_history_tool` to pull the loss and validation curves.
2. `diagnose_run_tool` for automated convergence, overfitting, and NaN checks.
3. `compare_runs_tool` against a known-good baseline run.
4. `create_wandb_report_tool` with line-plot panels to share the diagnosis.

### Summarize evals and compare model versions

To find which model version performed best on an evaluation, chain these tools:

1. `summarize_evaluation_tool` for per-scorer pass rates and error counts.
2. `list_artifact_versions_tool` on the relevant model collection.
3. `compare_artifact_versions_tool` between the candidate and current production version.
4. `log_analysis_to_wandb` and `create_wandb_report_tool` to publish the comparison.

## Troubleshooting

Use the following table to help you diagnose and resolve issues using the W\&B MCP Server:

| Symptom                                                                  | Cause and fix                                                                                                                                                                                                                                                           |
| ------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `401 Unauthorized` or `Invalid API key`                                  | Your W\&B API key is missing, malformed, or not authorized for the target entity or team. Regenerate a key at [wandb.ai/authorize](https://wandb.ai/authorize) and confirm it is passed as a bearer token or set in `WANDB_API_KEY`.                                    |
| Empty results for queries you expect to succeed                          | The team, entity, or project name is incorrect, or the API key doesn't have access. Confirm both with the agent and retry.                                                                                                                                              |
| `404 Not Found` or `connection refused` on `https://[YOUR-INSTANCE]/mcp` | The hosted MCP server isn't yet enabled on your Dedicated Cloud or Self-Managed instance, or the client is pointed at the wrong URL. Contact [W\&B support](mailto:support@wandb.com) to request enablement, then confirm the URL in [Connection URL](#connection-url). |
| `429 Too Many Requests` on Dedicated Cloud                               | You have hit your instance's rate limits. See [Dedicated Cloud rate limits](/platform/hosting/hosting-options/dedicated-cloud/rate-limits) for how to request higher limits.                                                                                            |
| Local server can't find `uvx` in Claude Desktop                          | Use the full path to `uvx` in the `command` field of `claude_desktop_config.json`.                                                                                                                                                                                      |


# Use W&B Skills
Source: https://docs.wandb.ai/platform/wb-skills

Install W&B Skills to teach your coding agent how to train models, build agents, and analyze experiments using W&B's AI development platform.

W\&B Skills are reusable instruction sets that teach coding agents how to use W\&B effectively. Instead of manually guiding your agent through W\&B APIs and best practices, install Skills so that the agent can work with experiment tracking, tracing, evaluations, and monitoring on its own.

Skills work with several major coding agents, including:

* Claude Code
* Codex
* Cursor
* GitHub Copilot
* Gemini CLI

For a full list of supported agents, see the [W\&B Skills CLI documentation](https://github.com/vercel-labs/skills#supported-agents).

## W\&B Skills capabilities

Skills covers both the [W\&B Models SDK](/models/ref) (training runs, metrics, artifacts, sweeps) and the [Weave SDK](/weave/reference/python-sdk) (traces, evaluations, scorers). It includes helper libraries, reference docs, and data analysis patterns so your agent can handle the following workflows.

| Workflow           | Capabilities                                                                                                                                                                                                                  |
| ------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Model training** | <ul><li>Log metrics and rich media during training and fine-tuning.</li><li>Track and compare experiments.</li><li>Analyze runs and results, such as loss curves and accuracy scores.</li><li>Tune hyperparameters.</li></ul> |
| **Agent building** | <ul><li>Trace agentic AI applications.</li><li>Analyze traces and classify failure modes.</li><li>Evaluate models and agents with labeled datasets.</li><li>Run online evaluations for production monitoring.</li></ul>       |

## Prerequisites

Skills requires the following:

* [Node.js](https://nodejs.org/) (for the `npx` command).
* A W\&B API key. Create one at [wandb.ai/authorize](https://wandb.ai/authorize) and then set it as an environment variable. Replace `[YOUR-API-KEY]` with your API key:
  ```bash theme={null}
  export WANDB_API_KEY=[YOUR-API-KEY]
  ```
* Optional: Set your W\&B project name as a `WANDB_PROJECT` environment variable. This lets your agent target the correct W\&B project without you specifying it each time.

## Install W\&B Skills

Choose a global installation to make Skills available to all your projects, or a project-specific installation to scope Skills to a single project.

To install W\&B Skills globally for all your projects, use the `--global` flag:

```bash theme={null}
npx skills add wandb/skills --skill '*' --yes --global
```

To install Skills only for the current project, run the install command from your project directory without the `--global` flag:

```bash theme={null}
npx skills add wandb/skills --skill '*' --yes
```

You can also install Skills for specific agents using the `--agent` flag:

```bash theme={null}
npx skills add wandb/skills --agent claude-code --skill '*' --yes --global
```

For a list of `--agent` and `--skill` options, see the [skills CLI documentation](https://github.com/vercel-labs/skills#supported-agents).

After installation completes, your agent has access to W\&B Skills and is ready to handle W\&B-related tasks.

## Use W\&B Skills

Once installed, you can ask the agent to perform W\&B-related tasks for your project. The following example prompts demonstrate some of the tasks your agent can do with W\&B Skills:

* "Log training metrics for my PyTorch model to W\&B."
* "Analyze the loss curves for my last 10 runs and identify the best performing configuration."
* "Trace my LangChain agent and log the results to Weave."
* "Run an evaluation on my agent using the test dataset and summarize the results."
* "Find the failure modes in my last evaluation and classify them."
* "Compare the configs of run A and run B and show me the differences."

## Usage tips

Skills performs better with specific queries than with broad, open-ended questions. The following table compares recommended prompts with prompts that are too vague.

| Recommended                                             | Not recommended                 |
| ------------------------------------------------------- | ------------------------------- |
| "What is the final validation loss for my last 5 runs?" | "How is my model doing?"        |
| "Summarize the token usage across my last 10 traces."   | "Show me all my traces."        |
| "Compare the configs of run A and run B."               | "What are my best runs?"        |
| "What eval had the highest F1 score?"                   | "How are my evaluations going?" |


# Reference
Source: https://docs.wandb.ai/reference

API, SDK, and CLI references for all W&B products.

## W\&B Models

<CardGroup>
  <Card title="Python SDK" href="/models/ref/python">
    Core Python library for experiment tracking, data types, global functions, automations, and custom charts.
  </Card>

  <Card title="Public API" href="/models/ref/python/public-api">
    Programmatic access to runs, projects, artifacts, reports, teams, and related resources.
  </Card>

  <Card title="CLI" href="/models/ref/cli">
    Command line interface for logging in, running jobs, executing sweeps, and managing artifacts.
  </Card>

  <Card title="Query Expression Language" href="/models/ref/query-panel">
    Query language for selecting and aggregating data in query panels.
  </Card>

  <Card title="Reports and Workspaces API" href="/models/ref/wandb_workspaces/reports">
    Programmatic authoring of W\&B Reports and workspaces.
  </Card>
</CardGroup>

## W\&B Weave

<CardGroup>
  <Card title="Python SDK" href="/weave/reference/python-sdk">
    Python client for Weave tracing, evaluations, and trace server integration.
  </Card>

  <Card title="TypeScript SDK" href="/weave/reference/typescript-sdk">
    TypeScript client for Weave, covering classes, functions, interfaces, and type aliases.
  </Card>

  <Card title="Service API" href="/weave/reference/service-api">
    REST API for integrating directly with Weave services.
  </Card>
</CardGroup>

## Serverless Inference

<CardGroup>
  <Card title="API Reference" href="/inference/api-reference">
    REST API reference for the Serverless Inference chat-completions and list-models endpoints.
  </Card>
</CardGroup>

## Serverless Training

<CardGroup>
  <Card title="API Reference" href="/serverless-training/api-reference">
    REST API reference for Serverless Training (RL and SFT) endpoints.
  </Card>
</CardGroup>


# Serverless Sandboxes
Source: https://docs.wandb.ai/sandboxes

On-demand, isolated compute environments that you can create, use, and discard from Python with W&B Serverless Sandboxes.

<Warning>
  Serverless Sandboxes is in public preview.
</Warning>

Serverless Sandboxes gives you on-demand, isolated compute environments that you can create, use, and discard using Python.

Serverless Sandboxes is built using the CoreWeave Sandbox library. For the underlying API reference and library docs, see the [CoreWeave Sandbox documentation](https://docs.coreweave.com/products/coreweave-sandbox).

## How it works

A *sandbox* is a single isolated compute environment. You [create](/sandboxes/create-sandbox) it, [run commands inside it](/sandboxes/run-commands), and stop it when it is done. Each sandbox runs in its own container with its own filesystem, network, and process space.

W\&B authenticates your identity when you create and manage sandboxes. To use W\&B or Weave inside a sandbox, pass your API key using the [W\&B Secrets Manager](/platform/secrets#manage-access-to-secrets) or environment variables. See [Secrets](/sandboxes/secrets) for more information about using secrets in sandboxes.

A sandbox goes through [several states in its lifecycle](/sandboxes/lifecycle). When a container is running, you can [execute commands](/sandboxes/run-commands) inside it.

[Read, write, and mount read-only](/sandboxes/file-access) files to and from the sandbox. Common examples include reading in a Python script to execute, writing out logs or results, or mounting a directory of data read-only for the sandbox to access.

Use a *session* to manage multiple sandboxes that share configuration. When a session closes, all of its sandboxes are stopped automatically. See [Manage multiple sandboxes](/sandboxes/create-sandbox#create-multiple-sandboxes-with-a-session) for more details.

## Basic usage

Follow these steps to create a sandbox and run a command inside it:

1. Install the W\&B Python SDK (`wandb`) and the serverless sandbox dependencies with the following command:
   ```bash theme={null}
   pip install wandb[sandbox]
   ```

2. Log in to W\&B with the [`wandb login`](/models/ref/cli/wandb-login) CLI command. When prompted, provide your API key to authenticate your identity and access your W\&B account:

   ```bash theme={null}
   wandb login
   ```

3. Copy and paste the following code snippet into a Python file and run it. The code snippet accomplishes the following:

   1. Create a sandbox with [`Sandbox.run()`](https://docs.coreweave.com/products/coreweave-sandbox/client/ref/core/sandbox#run).
   2. Run the command `echo "Hello from Serverless Sandboxes!"` inside the sandbox using the [`Sandbox.exec()`](https://docs.coreweave.com/products/coreweave-sandbox/client/ref/core/sandbox#exec) method.
   3. Print the output to the console using the `Process` object returned by `Sandbox.exec()`.

   ```python show lines title="hello_sandbox.py" theme={null}
   from wandb.sandbox import Sandbox

   with Sandbox.run() as sandbox:
       process = sandbox.exec(["echo", "Hello from Serverless Sandboxes!"]).result()
       print(process.stdout)
   ```

   You should see the output `Hello from Serverless Sandboxes!` printed to the console.

   The sandbox automatically stops when the context manager (the `with` block) exits. For more information on sandbox lifecycle and states, see [Lifecycle of sandboxes](/sandboxes/lifecycle).

## Serverless Sandboxes tutorials

For more in-depth examples, see:

* [Invoke an agent in a Serverless Sandbox tutorial](/sandboxes/invoke-agent-sandbox-tutorial)
* [Train a PyTorch model in a Serverless Sandbox tutorial](/sandboxes/mltrain-in-sandbox-tutorial)


# Create sandboxes
Source: https://docs.wandb.ai/sandboxes/create-sandbox

Create W&B Serverless Sandboxes that run code in isolated containers with their own filesystem, network, and process space.

<Warning>
  Serverless Sandboxes is in public preview.
</Warning>

Create W\&B Serverless Sandboxes to run code in isolated environments. Start a single sandbox, run commands in it, and manage multiple sandboxes with a session. Each sandbox runs in its own container, with its own [file system](/sandboxes/file-access), network, and process space.

<Info>
  By default, sandboxes use `python:3.11` as the base image. To use a different image, pass `container_image` to [`Sandbox.run()`](https://docs.coreweave.com/products/coreweave-sandbox/client/ref/core/sandbox#run) or [`SandboxDefaults`](https://docs.coreweave.com/products/coreweave-sandbox/client/ref/core/sandbox-defaults). W\&B supports public container images only.

  The following code snippet creates a sandbox with the `python:3.15` image and runs `python --version` inside it.

  ```python theme={null}
  from wandb.sandbox import Sandbox

  with Sandbox.run(container_image="python:3.15") as sandbox:
      sandbox.exec(["python", "--version"]).result()
  ```
</Info>

## Create a sandbox

Use [`Sandbox.run()`](https://docs.coreweave.com/products/coreweave-sandbox/client/ref/core/sandbox#run) to create and start a sandbox. This method returns a `Sandbox` object that you can use to interact with the environment.

<Tip>
  W\&B recommends using a context manager (`with` statement) to stop the sandbox automatically when the block exits, even if an error occurs.
</Tip>

The following example creates a sandbox with the default container image (`python:3.11`):

```python theme={null}
from wandb.sandbox import Sandbox

with Sandbox.run() as sandbox:
    print(sandbox)
```

To learn how to run commands inside a sandbox, see [Run commands](/sandboxes/run-commands). To learn about sandbox lifecycle and states, see [Sandbox lifecycle](/sandboxes/lifecycle).

### Start a sandbox without a main command

Call `Sandbox.run()` without a command when you want to create a sandbox first and run work inside it later.

```python theme={null}
from wandb.sandbox import Sandbox

with Sandbox.run() as sandbox:
    print(sandbox)
```

This pattern is useful for interactive and multi-step workflows. To learn how to run commands inside a sandbox, see [Run commands](/sandboxes/run-commands).

### Start a sandbox with a main command

You can also pass a command to `Sandbox.run()`. Use this pattern when the sandbox runs a single job from start to finish. When the main process exits, the sandbox enters a terminal state such as [COMPLETED or FAILED](/sandboxes/lifecycle).

The command you provide to `Sandbox.run()` starts as the sandbox's main process.

```python theme={null}
from wandb.sandbox import Sandbox

sandbox = Sandbox.run("python", "train.py")
```

`Sandbox.run()` returns a `Sandbox` object that you can use to monitor the command and wait for it to finish. For example, to wait for the command to complete and retrieve its result:

```python theme={null}
from wandb.sandbox import Sandbox

sandbox = Sandbox.run("python", "train.py")

# Optionally wait for the command to complete
sandbox.wait_until_complete().result()
```

## Create multiple sandboxes with a session

When you need more than one sandbox at a time, or want to share configuration across several sandboxes, use a session.

Use [`Session`](https://docs.coreweave.com/products/coreweave-sandbox/client/ref/core/session) to create and manage multiple sandboxes. When the session closes (for example, when you exit a `with` block), it automatically stops all sandboxes it created.

You can optionally pass a [`SandboxDefaults`](https://docs.coreweave.com/products/coreweave-sandbox/client/ref/core/sandbox-defaults) object to a session to define reusable default configuration for all sandboxes created by that session. For example, you can specify a default container image, network configuration, or maximum lifetime for all sandboxes in the session.

`session.sandbox()` returns an unstarted sandbox. It auto-starts when you call `Sandbox.exec()`, `Sandbox.read_file()`, or any other operation.

The following code snippet creates a session with two sandboxes that share a default configuration (`SandboxDefaults`):

```python theme={null}
from wandb.sandbox import Session, SandboxDefaults

defaults = SandboxDefaults(
    container_image="python:3.11",
    max_lifetime_seconds=300,
    tags=("batch-job",),
)

with Session(defaults) as session:
    sandbox1 = session.sandbox()
    sandbox2 = session.sandbox()

    print(sandbox1)
    print(sandbox2)
```


# File operations
Source: https://docs.wandb.ai/sandboxes/file-access

Learn how to read, write, and mount files in Serverless Sandboxes.

<Warning>
  Serverless Sandboxes is in public preview.
</Warning>

Use file operations to share data between your local environment and a sandbox. You can read files from a sandbox, write files to it, or mount local files and directories into it.

For example, you can write a Python script to a sandbox, run it, and read the output file back to your local environment. You can also mount a directory of training data into a sandbox for a machine learning job.

<Tip>
  **Choose the right file access method**

  Mount files or directories when you want the sandbox to access local data without copying it.

  Read and write files when you want to transfer smaller files between your local environment and the sandbox, or when you want to save sandbox output locally.
</Tip>

## Write a file to the sandbox

Write a file to the sandbox when you need to upload inputs your sandbox code uses, such as scripts, configs, or data files. Transfer a file from your local environment to the sandbox using the [`Sandbox.write_file()`](https://docs.coreweave.com/products/coreweave-sandbox/client/ref/core/sandbox#write_file) method.

```python theme={null}
from pathlib import Path
from wandb.sandbox import Sandbox

# Path of the local file you want to write to the sandbox
text_file = Path("hello.txt")

with Sandbox.run() as sandbox:
    # Write a file to the sandbox
    sandbox.write_file("hello.txt", text_file.read_bytes()).result()
```

See the `Sandbox` class reference documentation for a full list of parameters and options for [`Sandbox.write_file()`](https://docs.coreweave.com/products/coreweave-sandbox/client/ref/core/sandbox#write_file).

## Read a file from the sandbox

Save a file from the sandbox to your local environment using the [`Sandbox.read_file()`](https://docs.coreweave.com/products/coreweave-sandbox/client/ref/core/sandbox#read_file) method.

```python theme={null}
from pathlib import Path
from wandb.sandbox import Sandbox

with Sandbox.run() as sandbox:
    # Demo creates a file in the sandbox to read back
    sandbox.exec(["sh", "-c", "echo 'Hello, world.' > hello.txt"]).result()

    # Save the file on local machine (not sandbox)
    content = sandbox.read_file("hello.txt").result()  # b"Hello, world.\n"
    Path("hello.txt").write_bytes(content)
```

See the `Sandbox` class reference documentation for a full list of parameters and options for [`Sandbox.read_file()`](https://docs.coreweave.com/products/coreweave-sandbox/client/ref/core/sandbox#read_file).

## Mount a file or directory

Use mounted files to provide local files to the sandbox at creation time. Unlike `Sandbox.write_file()`, which transfers files to a running sandbox, mounted files are available as soon as the sandbox starts. Mounted files appear in the sandbox at the path you specify.

<Info>
  Mounted files are read-only in the sandbox. If you need to modify files in the sandbox, use `Sandbox.write_file()` instead.
</Info>

The following code snippet mounts the local files `train.py` and `requirements.txt` to the sandbox root directory. The sandbox installs dependencies from `requirements.txt` and then runs `train.py`.

```python theme={null}
from pathlib import Path
from wandb.sandbox import Sandbox, NetworkOptions

mounted_files = [
    {"mount_path": "requirements.txt", "file_content": Path("requirements.txt").read_bytes()},
    {"mount_path": "train.py", "file_content": Path("train.py").read_bytes()},
        ] 

print("Starting sandbox...")
with Sandbox.run(mounted_files=mounted_files) as sandbox:

    # The mounted files are available in the sandbox at the specified mount paths
    print("Installing dependencies...")
    result = sandbox.exec(["pip", "install", "-r", "requirements.txt"], check=True).result()
    print(result.stdout)

    print("Running script...")
    result = sandbox.exec(["python", "train.py"]).result()
    print(result.stdout)
```


# Tutorial: Invoke an agent in a sandbox
Source: https://docs.wandb.ai/sandboxes/invoke-agent-sandbox-tutorial

Learn how to invoke an OpenAI agent within a Serverless Sandbox environment

<Warning>
  Serverless Sandboxes is in public preview.
</Warning>

In this tutorial, you invoke an agent in a W\&B Serverless Sandbox. You start a sandbox with the required environment variables, install dependencies, and run a Python script that creates and invokes a simple OpenAI agent. The agent uses tool calls to get weather for a location and return a punny forecast.

<Note>
  This tutorial uses OpenAI as the language model for the agent, which requires an OpenAI API key.
</Note>

## Prerequisites

Before you begin, install the W\&B Python SDK, authenticate with W\&B, and store your OpenAI API key as a secret so the sandbox can use it.

### Install W\&B Python SDK

Install the W\&B Python SDK using `pip`:

```bash theme={null}
pip install wandb
```

### Log in and authenticate with W\&B

Run the [`wandb login`](/models/ref/cli/wandb-login) CLI command and follow the prompts to log in to your W\&B account:

```bash theme={null}
wandb login
```

### Store API keys in Secret Manager

Store your OpenAI API key in the [W\&B Secret Manager](/platform/secrets) as `OPENAI_API_KEY`. For more information about using secrets in sandboxes, see [Secrets](/sandboxes/secrets).

This lets you securely access the API key in the sandbox without hardcoding it in your code or sandbox configuration.

## Copy agent code

With your prerequisites in place, save the agent code locally so you can run it inside a sandbox. The script defines an OpenAI agent with two tools and a structured response format.

<Accordion title="OpenAI agent code">
  Copy the following code into a file named `demo.py` in the same directory as this tutorial. Then, run the previous code snippet to invoke an OpenAI agent in a sandbox.

  ```python title="demo.py" theme={null}
  import json
  import os
  import urllib.request
  import urllib.parse

  from openai import OpenAI

  api_key = os.getenv("OPENAI_API_KEY")
  if not api_key:
      raise ValueError("OPENAI_API_KEY environment variable not set")

  client = OpenAI(api_key=api_key)

  SYSTEM_PROMPT = """You are an expert weather forecaster, who speaks in puns.

  You have access to two tools:

  - get_weather_for_location: use this to get the weather for a specific location
  - get_user_location: use this to get the user's location

  If a user asks you for the weather, make sure you know the location. If you can tell from the question that they mean wherever they are, use the get_user_location tool to find their location.
  """

  TOOLS = [
      {
          "type": "function",
          "function": {
              "name": "get_weather_for_location",
              "description": "Get weather for a given location name.",
              "parameters": {
                  "type": "object",
                  "properties": {
                      "location": {
                          "type": "string",
                          "description": "The location to get weather for.",
                      }
                  },
                  "required": ["location"],
              },
          },
      },
      {
          "type": "function",
          "function": {
              "name": "get_user_location",
              "description": "Retrieve the user's current location from application context.",
              "parameters": {
                  "type": "object",
                  "properties": {},
                  "additionalProperties": False,
              },
          },
      },
  ]

  RESPONSE_FORMAT = {
      "type": "json_schema",
      "json_schema": {
          "name": "weather_response",
          "strict": True,
          "schema": {
              "type": "object",
              "properties": {
                  "punny_response": {"type": "string"},
                  "weather_conditions": {"type": ["string", "null"]},
              },
              "required": ["punny_response", "weather_conditions"],
              "additionalProperties": False,
          },
      },
  }

  _WMO_CODES = {
      0: "clear sky", 1: "mainly clear", 2: "partly cloudy", 3: "overcast",
      45: "foggy", 48: "depositing rime fog",
      51: "light drizzle", 53: "moderate drizzle", 55: "dense drizzle",
      61: "slight rain", 63: "moderate rain", 65: "heavy rain",
      71: "slight snowfall", 73: "moderate snowfall", 75: "heavy snowfall",
      80: "slight rain showers", 81: "moderate rain showers", 82: "violent rain showers",
      95: "thunderstorm", 96: "thunderstorm with slight hail", 99: "thunderstorm with heavy hail",
  }


  def get_weather_for_location(location: str) -> str:
      geo_url = "https://geocoding-api.open-meteo.com/v1/search?" + urllib.parse.urlencode(
          {"name": location, "count": 1}
      )
      with urllib.request.urlopen(geo_url) as resp:
          geo = json.loads(resp.read())

      if not geo.get("results"):
          return f"Could not find a location named '{location}'."

      loc = geo["results"][0]
      lat, lon = loc["latitude"], loc["longitude"]
      name = loc.get("name", location)

      weather_url = "https://api.open-meteo.com/v1/forecast?" + urllib.parse.urlencode({
          "latitude": lat,
          "longitude": lon,
          "current": "temperature_2m,weather_code,wind_speed_10m",
          "temperature_unit": "fahrenheit",
          "wind_speed_unit": "mph",
      })
      with urllib.request.urlopen(weather_url) as resp:
          weather = json.loads(resp.read())

      cur = weather["current"]
      condition = _WMO_CODES.get(cur["weather_code"], "unknown conditions")
      temp = cur["temperature_2m"]
      wind = cur["wind_speed_10m"]

      return f"{name}: {condition}, {temp}°F, wind {wind} mph"


  def get_user_location(user_id: str) -> str:
      return "Miami" if user_id == "1" else "San Francisco"


  TOOL_DISPATCH = {
      "get_weather_for_location": lambda args, _ctx: get_weather_for_location(args["location"]),
      "get_user_location": lambda _args, ctx: get_user_location(ctx["user_id"]),
  }


  def run_agent(messages: list[dict], context: dict) -> dict:
      while True:
          response = client.chat.completions.create(
              model="gpt-4o",
              temperature=0,
              messages=messages,
              tools=TOOLS,
              response_format=RESPONSE_FORMAT,
          )

          message = response.choices[0].message

          if message.tool_calls:
              messages.append({
                  "role": "assistant",
                  "content": message.content,
                  "tool_calls": [
                      {
                          "id": tc.id,
                          "type": "function",
                          "function": {
                              "name": tc.function.name,
                              "arguments": tc.function.arguments,
                          },
                      }
                      for tc in message.tool_calls
                  ],
              })

              for tool_call in message.tool_calls:
                  fn_name = tool_call.function.name
                  fn_args = json.loads(tool_call.function.arguments)

                  tool_fn = TOOL_DISPATCH.get(fn_name)
                  if tool_fn is None:
                      raise ValueError(f"Unknown tool: {fn_name}")

                  result = tool_fn(fn_args, context)
                  messages.append({
                      "role": "tool",
                      "tool_call_id": tool_call.id,
                      "content": result,
                  })
              continue

          messages.append({
              "role": "assistant",
              "content": message.content,
          })
          return json.loads(message.content)


  # --- Run the agent ---

  conversation: list[dict] = [{"role": "system", "content": SYSTEM_PROMPT}]
  context = {"user_id": "1"}

  # First turn
  conversation.append({"role": "user", "content": "what is the weather outside?"})
  result = run_agent(conversation, context)
  print(result)
  # {'punny_response': "Florida is still having a 'sun-derful' day! ...", 'weather_conditions': "It's always sunny in Florida!"}

  # Second turn (conversation history is preserved)
  conversation.append({"role": "user", "content": "thank you!"})
  result = run_agent(conversation, context)
  print(json.dumps(result, indent=2))
  ```
</Accordion>


# Sandbox lifecycle
Source: https://docs.wandb.ai/sandboxes/lifecycle

Learn about the lifecycle of a Serverless Sandbox, including its states, how to wait for state changes, and how to stop a sandbox.

<Warning>
  Serverless Sandboxes is in public preview.
</Warning>

This page describes the lifecycle of a Serverless Sandbox so that you can choose the right method for waiting on state changes, stopping a sandbox, and structuring code around its lifetime. Understanding the lifecycle helps you avoid race conditions, distinguish startup failures from runtime errors, and clean up resources predictably.

A W\&B Serverless Sandbox goes through several states during its lifecycle. The sandbox state determines which operations are available.

Usually, a sandbox starts in `PENDING`, moves to `CREATING` while the container is provisioned, and then enters `RUNNING` when it's ready for use.

If you start a sandbox with a main command, that command becomes the sandbox's main process. When the main process exits, the sandbox enters a terminal state such as `COMPLETED` (exit code `0`) or `FAILED` (an error during startup or execution). A sandbox can also enter `TERMINATED` if it's stopped externally or exceeds its maximum lifetime.

```text theme={null}
PENDING -> CREATING -> RUNNING -> COMPLETED
                               -> FAILED
                               -> TERMINATED
```

The following sections describe sandbox states, how to wait for readiness or completion, and how to stop a sandbox. For more information about creating sandboxes and running commands, see [Create sandboxes](/sandboxes/create-sandbox) and [Run commands](/sandboxes/run-commands).

## Sandbox states

The following table summarizes the states of a sandbox:

| State        | Description                                                                                    |
| ------------ | ---------------------------------------------------------------------------------------------- |
| `PENDING`    | The sandbox request was accepted and is waiting to be scheduled.                               |
| `CREATING`   | The container is provisioning.                                                                 |
| `RUNNING`    | The sandbox is ready for operations.                                                           |
| `COMPLETED`  | The main process exited successfully with code `0`.                                            |
| `FAILED`     | The sandbox or its main process encountered an error during startup or execution.              |
| `TERMINATED` | An external process stopped the sandbox, or it ended because it exceeded its maximum lifetime. |

Most operations require the sandbox to be in the `RUNNING` state. However, many operations wait until the sandbox is ready before proceeding.

After a sandbox enters a terminal state, it's no longer available for additional work.

## Wait for readiness or completion

The following sections describe the two methods for waiting on sandbox state changes and when to use each. Use different wait methods depending on what you need:

* Use `Sandbox.wait()` to [wait until the sandbox is ready for use](/sandboxes/lifecycle#wait-for-a-sandbox-to-start).
* Use `Sandbox.wait_until_complete()` to [wait until the sandbox's main process finishes](/sandboxes/lifecycle#wait-for-a-sandbox-to-complete).

### Wait for a sandbox to start

Use `Sandbox.wait()` to explicitly wait for the sandbox to reach the `RUNNING` state. This method is useful for debugging startup problems or when you want to distinguish startup failures from errors in later commands.

For example, the following code creates a sandbox, waits for it to become ready, and then runs a command:

```python theme={null}
import wandb
from wandb.sandbox import Sandbox

with Sandbox.run() as sandbox:
    sandbox.wait()  # block until the sandbox is running
    result = sandbox.exec(["python", "-c", "print('hello from sandbox')"]).result()
    print(result.stdout)
```

### Wait for a sandbox to complete

Use [`Sandbox.wait_until_complete()`](https://docs.coreweave.com/products/coreweave-sandbox/client/ref/core/sandbox#wait_until_complete) when the sandbox's main process represents the full workload and you want to wait for that job to finish.

```python theme={null}
from wandb.sandbox import Sandbox

sandbox = Sandbox.run("python", "train.py")
sandbox.wait_until_complete(timeout=3600.0).result()  # block until the main process exits or timeout is reached
print(f"Exit code: {sandbox.returncode}")
```

This pattern is useful when you start a sandbox with a main command such as `Sandbox.run("python", "train.py")` and want the sandbox lifecycle to match that command's execution.

When the main process exits, the sandbox enters a terminal state. A successful run typically ends in `COMPLETED`. If the process fails, the sandbox may enter `FAILED`.

If you need to run commands interactively instead, use a context manager with `Sandbox.exec()`:

```python theme={null}
from wandb.sandbox import Sandbox

with Sandbox.run() as sandbox:
    result = sandbox.exec(["python", "train.py"]).result()
    print(result.stdout)
# sandbox automatically stopped on exit
```

## Explicit lifecycle control

Use methods such as `Sandbox.wait()` and `Sandbox.stop()` when you need explicit control over the sandbox lifecycle.

Usually, you don't need to call these methods directly. Operations such as `Sandbox.exec()`, `Sandbox.read_file()`, and `Sandbox.write_file()` wait for readiness, and the context manager (`with Sandbox.run() as sandbox:`) stops the sandbox when the block exits.

### Stop a sandbox

Use [`Sandbox.stop()`](https://docs.coreweave.com/products/coreweave-sandbox/client/ref/core/sandbox#stop) when you no longer need the sandbox, when you want to end a long-running process, or when you need to clean up resources before the sandbox reaches its maximum lifetime.

```python theme={null}
from wandb.sandbox import Sandbox

sandbox = Sandbox.run("sleep", "infinity")

# ... use the sandbox ...

sandbox.stop().result()
```

You can also control shutdown behavior with additional options:

```python theme={null}
from wandb.sandbox import Sandbox

with Sandbox.run("sleep", "infinity") as sandbox:
    sandbox.stop(
        graceful_shutdown_seconds=30.0,  # Wait before force-killing the sandbox.
        missing_ok=True,                 # Don't raise an error if it's already stopped.
    ).result()
```


# Tutorial: Train a PyTorch model
Source: https://docs.wandb.ai/sandboxes/mltrain-in-sandbox-tutorial

Learn how to train a PyTorch model in a Serverless Sandbox environment with this step-by-step tutorial.

<Warning>
  Serverless Sandboxes is in public preview.
</Warning>

In this tutorial, you train a PyTorch model in a Serverless Sandbox environment. To do this, you start a sandbox with the appropriate environment variables, install the necessary dependencies, and run a Python script. The script trains a neural network on the UCI Zoo dataset.

By the end of this tutorial, you have a trained PyTorch model file saved locally. This demonstrates how to use a Serverless Sandbox to run isolated ML training workloads without configuring local infrastructure. This tutorial is intended for ML practitioners and developers who want to evaluate Serverless Sandboxes for reproducible training jobs.

## Prerequisites

Before you get started, complete the following setup steps.

### Install the W\&B Python SDK

The W\&B Python SDK provides the `Sandbox` interface you use later to create and interact with the Serverless Sandbox. Install it using `pip`:

```bash theme={null}
pip install wandb
```

### Log in and authenticate with W\&B

W\&B Serverless Sandboxes run under your W\&B account, so you must authenticate before you create one. Use the `wandb login` CLI command and follow the prompts to log in:

```bash theme={null}
wandb login
```

See the [`wandb login`](/models/ref/cli/wandb-login) reference documentation for more information about how W\&B searches for credentials.

## Copy the training script and dependencies

Prepare the three files required for this tutorial: a requirements file, a hyperparameters file, and a training script. Expand the following dropdown, then copy each code sample into a separate file in the same directory as this tutorial.

In the next section, you run a script that reads these files and trains a PyTorch model in a W\&B Serverless Sandbox.

<Accordion title="PyTorch training model script">
  Copy and paste the following code into a file named `requirements.txt`. This file contains the dependencies for the training script.

  ```txt title="requirements.txt" theme={null}
  torch
  pandas
  ucimlrepo
  scikit-learn
  pyyaml
  ```

  Copy and paste the following code into a YAML file named `hyperparameters.yaml`. This file contains the hyperparameters for the training script.

  ```yaml title="hyperparameters.yaml" theme={null}
  learning_rate: 0.1
  epochs: 1000
  model_type: Multivariate_neural_network_classifier
  ```

  Copy and paste the following code into a file named `train.py`. This script trains a PyTorch model on the UCI Zoo dataset and saves the trained model to a file named `zoo_wandb.pth`.

  ```python title="train.py" theme={null}
  import argparse
  import torch 
  from torch import nn
  import yaml
  import pandas as pd
  from ucimlrepo import fetch_ucirepo

  from sklearn.model_selection import train_test_split

  class NeuralNetwork(nn.Module):
      def __init__(self):
          super().__init__()
          self.linear_stack = nn.Sequential(
              nn.Linear(in_features=16 , out_features=16),
              nn.Sigmoid(),
              nn.Linear(in_features=16, out_features=7)
          )

      def forward(self, x):
          logits = self.linear_stack(x)
          return logits

  def main(args):
      # Load hyperparameters from the provided config file
      with open(args.config, 'r') as f:
          hyperparameter_config = yaml.safe_load(f)

      # fetch dataset 
      zoo = fetch_ucirepo(id=111) 
      
      # data (as pandas dataframes) 
      X = zoo.data.features 
      y = zoo.data.targets

      print("features: ", X.shape, "type: ", type(X))
      print("labels: ", y.shape, "type: ", type(y))

      ## Process data
      # Data type of the data must match the data type of the model, the default dtype for nn.Linear is torch.float32
      dataset = torch.tensor(X.values).type(torch.float32) 

      # Convert to tensor and format labels from 0 - 6 for indexing
      labels = torch.tensor(y.values)  - 1

      print("dataset: ", dataset.shape, "dtype: ",dataset.dtype)
      print("labels: ", labels.shape, "dtype: ",labels.dtype)

      torch.save(dataset, "zoo_dataset.pt")
      torch.save(labels, "zoo_labels.pt")

      # Describe how we split the training dataset for future reference, reproducibility.
      config = {
          "random_state" : 42,
          "test_size" : 0.25,
          "shuffle" : True
      }

      # Split dataset into training and test set
      X_train, X_test, y_train, y_test = train_test_split(
          dataset,labels, 
          random_state=config["random_state"],
          test_size=config["test_size"], 
          shuffle=config["shuffle"]
      )

      # Save the files locally
      torch.save(X_train, "zoo_dataset_X_train.pt")
      torch.save(y_train, "zoo_labels_y_train.pt")

      torch.save(X_test, "zoo_dataset_X_test.pt")
      torch.save(y_test, "zoo_labels_y_test.pt")


      ## Define model
      model = NeuralNetwork()
      loss_fn = nn.CrossEntropyLoss()
      optimizer = torch.optim.SGD(model.parameters(), lr=hyperparameter_config["learning_rate"])
      print(model)

      # Set initial dummy loss value to compare to in training loop
      prev_best_loss = 1e10 

      # Training loop
      for e in range(hyperparameter_config["epochs"] + 1):
          pred = model(X_train)
          loss = loss_fn(pred, y_train.squeeze(1))
          
          loss.backward()
          optimizer.step()
          optimizer.zero_grad()

          # Checkpoint/save model if loss improves
          if (e % 100 == 0) and (loss <= prev_best_loss):
              print("epoch: ", e, "loss:", loss.item())
          
              # Store new best loss
              prev_best_loss = loss

      print("Saving model...")
      PATH = 'zoo_wandb.pth' 
      torch.save(model.state_dict(), PATH)

  if __name__ == "__main__":
      parser = argparse.ArgumentParser(description="Train a simple neural network on the zoo dataset.")
      parser.add_argument("--config", type=str, required=True, help="Path to the hyperparameter configuration file.")
      args = parser.parse_args()
      main(args)
  ```
</Accordion>

## Create the sandbox and run the training script

With your training files in place, create and manage a W\&B Serverless Sandbox from a single Python script. The following code snippet creates a sandbox, copies the training script and dependencies into it, runs the training script, and downloads the generated model file.

The next section explains the code line by line.

Copy and paste the following code into a Python file and run it. Save it in the same directory as the `train.py`, `requirements.txt`, and `hyperparameters.yaml` files you created in the previous step.

```python Show lines title="train_in_sandbox.py" theme={null}
from pathlib import Path
from wandb.sandbox import Sandbox, NetworkOptions

# Files to mount to the sandbox. Specify the path inside the
# sandbox and the content of each file as bytes as a dictionary
mounted_files = [
    {"mount_path": "train.py", "file_content": Path("train.py").read_bytes()},
    {"mount_path": "requirements.txt", "file_content": Path("requirements.txt").read_bytes()},
        ] 

print("Starting sandbox...")
with Sandbox.run(
    mounted_files=mounted_files,
    container_image="python:3.13",
    network=NetworkOptions(egress_mode="internet"),
    max_lifetime_seconds=3600
) as sandbox:
    sandbox.write_file("hyperparameters.yaml", Path("hyperparameters.yaml").read_bytes()).result()

    # Install dependencies
    print("Installing dependencies...")
    sandbox.exec(["pip", "install", "-r", "requirements.txt"], check=True).result()

    # Run the script
    print("Running script...")
    result = sandbox.exec(["python", "train.py", "--config", "hyperparameters.yaml"]).result()
    print(result.stdout)
    print(result.stderr)
    print(f"Exit code: {result.returncode}")

    # Save the generated model file locally
    print("Downloading zoo_wandb.pth...")
    model_data = sandbox.read_file("zoo_wandb.pth").result()
    Path("zoo_wandb.pth").write_bytes(model_data)
    print("Saved zoo_wandb.pth")
```

The previous code snippet does the following:

1. (Lines 6-9) List the files to mount to the sandbox: `train.py` and `requirements.txt`.
2. (Line 12) Start the sandbox. The sandbox is configured to use the `python:3.13` container image, have internet access, and a maximum lifetime of 3600 seconds (1 hour).
3. (Line 18) Write the `hyperparameters.yaml` file to the sandbox. This lets the training script (`train.py`) access the hyperparameters when it runs.
4. (Line 22) Install dependencies. The command `pip install -r requirements.txt` runs inside the sandbox to install the necessary dependencies for the training script.
5. (Line 26) Run the training script. The command `python train.py --config hyperparameters.yaml` runs inside the sandbox to start the training process. The script trains a PyTorch model on the UCI Zoo dataset and saves the trained model to a file named `zoo_wandb.pth`.
6. (Lines 27-29) Print the output and exit code. After the training script finishes, the standard output, standard error, and exit code are printed to the console for debugging and verification.
7. (Lines 33-34) Download the generated model file. The `read_file()` method reads `zoo_wandb.pth` from the sandbox, and the script saves it locally.

After the script completes, you have a trained PyTorch model saved as `zoo_wandb.pth` in your working directory. The sandbox that produced it is created, used, and torn down on demand.


# Configuration
Source: https://docs.wandb.ai/sandboxes/ref-link-configuration





# Core
Source: https://docs.wandb.ai/sandboxes/ref-link-core





# Discovery
Source: https://docs.wandb.ai/sandboxes/ref-link-discovery





# Exceptions
Source: https://docs.wandb.ai/sandboxes/ref-link-exceptions





# Types
Source: https://docs.wandb.ai/sandboxes/ref-link-types





# Utilities
Source: https://docs.wandb.ai/sandboxes/ref-link-util





# Run commands in a sandbox
Source: https://docs.wandb.ai/sandboxes/run-commands

Learn how to run commands in a sandbox environment and read output and exit codes.

<Warning>
  Serverless Sandboxes is in public preview.
</Warning>

<br />

[Run commands in a sandbox](#run-commands-in-a-sandbox), [read command output and exit codes](#read-output-and-exit-codes), and [check for errors](#check-for-sandbox-execution-errors). Use these features to run scripts, training jobs, and other processes in a W\&B Serverless Sandbox and programmatically check the results.

## Run commands in a sandbox

The sandbox client supports two patterns for running commands, depending on whether you need to run a single command or a sequence of related commands. Based on your use case, use one of these patterns to run commands in a sandbox:

* [Single command](#run-a-single-command): Pass the command to `Sandbox.run()` and call `Sandbox.wait_until_complete()` to block until it finishes. Best for batch jobs, tests, or training runs.
* [Multiple commands](#run-multiple-commands-sequentially): Call `Sandbox.run()` with no arguments, then use `Sandbox.exec()` for each step. Best for workflows that combine setup, execution, and result retrieval.

### Run a single command

Pass the command to `Sandbox.run()`, then call `Sandbox.wait_until_complete()` to wait for the sandbox to reach a terminal state.

The following example starts a sandbox that runs `python train.py`, then waits up to 3600 seconds for the sandbox to complete.

```python theme={null}
from wandb.sandbox import Sandbox

sandbox = Sandbox.run("python", "train.py")
sandbox.wait_until_complete(timeout=3600.0).result()
print(f"Exit code: {sandbox.returncode}")
```

### Run multiple commands sequentially

Run multiple commands in the same sandbox for workflows with multiple steps. For example, install dependencies, write files, run scripts, and read results.

To run multiple commands in a sandbox, start the sandbox without a main command. Use a series of `Sandbox.exec()` to run each command and wait for it to complete.

The following example starts a sandbox without a main command, then runs two commands sequentially in the same sandbox.

```python theme={null}
from wandb.sandbox import Sandbox

with Sandbox.run() as sandbox:
    result = sandbox.exec(["pip", "install", "torch"]).result()
    print(result.stdout)

    result = sandbox.exec(["python", "train.py"]).result()
    print(result.stdout)
```

`Sandbox.exec()` runs a command in the sandbox and returns a `Process` object. Use the returned object to wait for the command to finish, read output, and inspect the exit code.

For more information about `Sandbox.exec()` and its parameters, see [`Sandbox.exec()`](https://docs.coreweave.com/products/coreweave-sandbox/client/ref/core/sandbox#exec) in the CoreWeave Sandboxes reference documentation.

#### Keep the sandbox running for additional commands

Sometimes, you might want to start the sandbox with a long-running main process to keep it available while running other commands with `Sandbox.exec()`.

The following example starts a sandbox with `sleep infinity` as the main process, then runs a training script inside the sandbox with `Sandbox.exec()`.

```python theme={null}
from wandb.sandbox import Sandbox

with Sandbox.run("sleep", "infinity") as sandbox:
    result = sandbox.exec(["python", "train.py"]).result()
    print(result.stdout)
```

## Read output and exit codes

After you run a command, you can read its output and check whether it succeeded. `Sandbox.exec()` returns a [`Process`](https://docs.coreweave.com/products/coreweave-sandbox/client/ref/types/process) object. To wait for the command to finish, call `Process.result()`.

The result includes standard output (`stdout`), standard error (`stderr`), and the exit code (`returncode`).

Use the exit code to determine whether the command succeeded. By convention, an exit code of 0 indicates success. The possible exit codes and their meanings depend on the command.

For example, the following code snippet runs a Python command that prints the standard output (`stdout`) and the exit code (`returncode`). After the command finishes, the snippet prints the standard output, standard error, and exit code to the console.

```python theme={null}
from wandb.sandbox import Sandbox

with Sandbox.run() as sandbox:
    result = sandbox.exec(["python", "-c", "print('hello')"]).result()
    print(result.stdout)      # "hello\\n"
    print(result.returncode)  # 0
```

## Check for sandbox execution errors

Use `check=True` to raise an exception when a command fails instead of inspecting exit codes manually. If the command exits with a non-zero exit code, `Sandbox.exec(check=True)` raises a [SandboxExecutionError](https://docs.coreweave.com/products/coreweave-sandbox/client/ref/exceptions/exceptions#sandboxexecutionerror).

```python theme={null}
from wandb.sandbox import Sandbox, SandboxExecutionError

with Sandbox.run() as sandbox:
    try:
        sandbox.exec(["python", "might_fail.py"], check=True).result()
    except SandboxExecutionError as e:
        print(f"Command failed with exit code {e.exec_result.returncode}")
        print(e.exec_result.stderr)
```


# Secrets
Source: https://docs.wandb.ai/sandboxes/secrets

Learn how to manage secrets in Serverless Sandboxes.

<Warning>
  Serverless Sandboxes is in public preview.
</Warning>

Use the [W\&B Secrets Manager](/platform/secrets) with the W\&B Python SDK to access sensitive information such as API keys and tokens in a sandbox.

To use a secret in a sandbox, first add it to your team's Secrets Manager. Then reference that secret by name when you create the sandbox.

<Caution>
  Don't include the secret's value directly in your code or in the sandbox configuration. Reference secrets by name only.
</Caution>

W\&B injects each requested secret into the sandbox as an environment variable. By default, the environment variable name matches the secret name. You can customize the environment variable name with the `env_var` parameter.

<Note>
  See [Manage secrets](/platform/secrets) for information about creating and managing secrets in your W\&B account.
</Note>

## Access secrets in a sandbox

Use the `Secret` class to specify which secrets to include in a sandbox. Pass one or more `Secret` objects to the `secrets` parameter of `Sandbox.run()`. Each `Secret` object identifies a secret by name. W\&B injects the secret value into the sandbox as an environment variable.

<Tip>
  Pass the secret name, not the secret value. Within the sandbox, read the secret from the corresponding environment variable.
</Tip>

The following example makes two existing secrets available in the sandbox: `HF_TOKEN` and `OPENAI_API_KEY`.

```python theme={null}
from wandb.sandbox import Sandbox, Secret

with Sandbox.run(
    secrets=[
        Secret(name="HF_TOKEN"),
        Secret(name="OPENAI_API_KEY"),
    ],
) as sandbox:
    result = sandbox.exec([
        "python",
        "-c",
        """
import os
if "HF_TOKEN" in os.environ:
    print(f"Hugging Face token found.")
if "OPENAI_API_KEY" in os.environ:
    print(f"OpenAI key token found.")
""",
    ]).result()
    print(result.stdout)
```

Inside the sandbox, access each secret through its environment variable. In this example, `HF_TOKEN` is available as `os.environ["HF_TOKEN"]`, and `OPENAI_API_KEY` is available as `os.environ["OPENAI_API_KEY"]`.

To customize the environment variable name for a secret, see [Use a custom environment variable name](#use-a-custom-environment-variable-name).

## Use a custom environment variable name

By default, a secret's environment variable name matches the secret name. To customize a sandbox's environment variable for a given secret, set the `env_var` parameter on `Secret` (`Secret(env_var="CUSTOM_NAME")`).

The following example checks that the `HF_TOKEN` secret is available in the sandbox:

```python theme={null}
from wandb.sandbox import Sandbox, Secret

with Sandbox.run(
    secrets=[
        Secret(name="HF_TOKEN", env_var="HUGGINGFACE_TOKEN")
    ],
) as sandbox:
    result = sandbox.exec([
        "python",
        "-c",
        """
import os
if "HUGGINGFACE_TOKEN" in os.environ:
    print(f"Hugging Face token found!")
""",
    ]).result()
    print(result.stdout)
```


# Serverless Training
Source: https://docs.wandb.ai/serverless-training

Post-train and fine-tune LLMs on managed serverless infrastructure with Serverless RL and Serverless SFT.

Use Serverless Training to post-train and fine-tune LLMs on managed, serverless infrastructure. W\&B provisions the training infrastructure ([on CoreWeave](https://docs.coreweave.com/docs/platform)) for you while allowing full flexibility in your environment's setup. You get instant access to a managed training cluster that elastically auto-scales to dozens of GPUs. Serverless Training is now in public preview.

Serverless Training offers two complementary methods:

* **[Serverless RL](/serverless-training/usage)**: Post-train models with reinforcement learning so they learn new behaviors and improve reliability, speed, and costs when performing multi-turn agentic tasks. Serverless RL splits RL workflows into inference and training phases and multiplexes them across jobs to increase GPU utilization and reduce your training time and costs.
* **[Serverless SFT](/serverless-training/sft)**: Fine-tune models with supervised learning on curated datasets. Use SFT for distillation, teaching output style and format, or warming up a model before applying RL.

Serverless Training is ideal for tasks like:

* Voice agents
* Deep research assistants
* On-prem models
* Content marketing analysis agents

Serverless Training trains low-rank adapters (LoRAs) to specialize a model for your specific task. This extends the original model's capabilities with on-the-job experience. W\&B automatically stores the LoRAs you train as artifacts in your account. You can also save them locally or to a third party for backup. Serverless Inference also automatically hosts models that you train through Serverless Training.

See the ART [quickstart](https://art.openpipe.ai/getting-started/quick-start) or [Google Colab notebook](https://colab.research.google.com/github/openpipe/art-notebooks/blob/main/examples/2048/2048.ipynb) to get started.

## Why Serverless Training?

Serverless Training can provide the following advantages in your post-training:

* **Lower training costs**: By multiplexing shared infrastructure across many users, skipping the setup process for each job, and scaling your GPU costs down to 0 when you're not actively training, Serverless Training reduces training costs significantly.
* **Faster training time**: By splitting inference requests across many GPUs and immediately provisioning training infrastructure when you need it, Serverless Training speeds up your training jobs and lets you iterate faster.
* **Automatic deployment**: Serverless Training automatically deploys every checkpoint you train, so you do not need to manually set up hosting infrastructure. You can access and test trained models immediately in local, staging, or production environments.

## How Serverless Training uses W\&B services

Serverless Training uses a combination of the following W\&B components to operate:

* [Inference](/inference): To run your models
* [Models](/models): To track performance metrics during the LoRA adapter's training
* [Artifacts](/models/artifacts): To store and version the LoRA adapters
* [Weave (optional)](/weave): To gain observability into how the model responds at each step of the training loop

Serverless Training is in public preview. During the preview, W\&B charges you only for inference usage and artifact storage. W\&B does not charge for adapter training during the preview period.


# Available models
Source: https://docs.wandb.ai/serverless-training/available-models

Browse the foundation models available for training with Serverless RL, including supported model variants and capabilities.

Serverless RL currently supports the following foundation models for training.

To express interest in a particular model, contact [support](mailto:support@wandb.ai).

## Model catalog

| Model                       | Model ID (for API usage)           | Type | Context Window | Parameters                | Description                                                                                                                         |
| --------------------------- | ---------------------------------- | ---- | -------------- | ------------------------- | ----------------------------------------------------------------------------------------------------------------------------------- |
| OpenPipe Qwen3 14B Instruct | `OpenPipe/Qwen3-14B-Instruct`      | Text | 32.8k          | 14.8B (Total)             | An efficient multilingual, dense, instruction-tuned model, optimized by OpenPipe for building agents with finetuning.               |
| Qwen3 30B A3B               | `Qwen/Qwen3-30B-A3B-Instruct-2507` | Text | 262k           | 3.3B-30.5B (Active-Total) | Qwen3-30B-A3B-Instruct-2507 is a 30.5B MoE instruction-tuned model with enhanced reasoning, coding, and long-context understanding. |


# Prerequisites
Source: https://docs.wandb.ai/serverless-training/prerequisites

Set up your environment for Serverless Training by creating an account, generating an API key, and configuring a project.

Complete these steps before using Serverless Training features through the OpenPipe ART framework or API.

<Tip>
  Before starting, review the [usage information and limits](/serverless-training/usage-limits) to understand costs and restrictions.
</Tip>

## Sign up and create an API key

If you don't already have a W\&B account, sign up before continuing. To authenticate your machine with W\&B, you must first generate an API key.

To create an API key, select the **Personal API key** or **Service Account API key** tab for details.

<Tabs>
  <Tab title="Personal API key">
    To create a personal API key owned by your user ID:

    1. Log in to W\&B, click your user profile icon, then click **User Settings**.
    2. Click **Create new API key**.
    3. Provide a descriptive name for your API key.
    4. Click **Create**.
    5. Copy the displayed API key immediately and store it securely.
  </Tab>

  <Tab title="Service account API key">
    To create an API key owned by a service account:

    1. Navigate to the **Service Accounts** tab in your team or organization settings.
    2. Find the service account in the list.
    3. Click the **action (<Icon icon="ellipsis" />)** menu, then click **Create API key**.
    4. Provide a name for the API key, then click **Create**.
    5. Copy the displayed API key immediately and store it securely.
    6. Click **Done**.

    You can create multiple API keys for a single service account to support different environments or workflows.
  </Tab>
</Tabs>

<Warning>
  The full API key is only shown once at creation time. After you close the dialog, you cannot view the full API key again. Only the key ID (first part of the key) is visible in your settings. If you lose the full API key, you must create a new API key.
</Warning>

For secure storage options, see [Store API keys securely](/platform/app/settings-page/user-settings/#store-and-handle-api-keys-securely).

## Create a project in W\&B

Create a project in your W\&B account to track usage, record training metrics, and save trained models. For more information, see the [Projects guide](https://docs.wandb.ai/models/track/project-page).

## Next steps

After you complete the prerequisites, try the following resources:

* Check the [API reference](/serverless-training/api-reference) to learn about available endpoints.
* Try the [ART quickstart](https://art.openpipe.ai/getting-started/quick-start).


# API Reference
Source: https://docs.wandb.ai/serverless-training/ref-link-api





# Serverless SFT
Source: https://docs.wandb.ai/serverless-training/sft

Learn how to fine-tune models using supervised fine-tuning (SFT) on W&B

Use Serverless SFT to fine-tune LLMs with supervised learning on curated datasets. Serverless SFT is in public preview. W\&B provisions the training infrastructure ([on CoreWeave](https://docs.coreweave.com/docs/platform)) for you and gives you full flexibility to set up your environment. You get instant access to a managed training cluster that auto-scales to handle your training workloads.

Serverless SFT is ideal for tasks such as:

* **Distillation**: Transferring knowledge from a larger, more capable model into a smaller, faster one.
* **Teaching output style and format**: Training a model to follow specific response formats, tone, or structure.
* **Warmup before RL**: Pre-training a model with supervised examples before applying reinforcement learning for further refinement.

Serverless SFT trains low-rank adapters (LoRAs) to specialize a model for your specific task. W\&B automatically stores the LoRAs you train as artifacts in your account. You can also save them locally or to a third party for backup. Serverless Inference also automatically hosts models that you train through Serverless SFT.

To begin training a model with Serverless SFT, see the ART [Serverless SFT docs](https://art.openpipe.ai/fundamentals/sft-training).

## Why Serverless SFT

Supervised fine-tuning (SFT) is a training technique where a model learns from curated input-output examples. Serverless SFT on W\&B provides the following advantages:

* **Lower training costs**: Serverless SFT multiplexes shared infrastructure across many users, skips the setup process for each job, and scales your GPU costs down to zero when you aren't actively training. This reduces training costs.
* **Faster training time**: Serverless SFT immediately provisions training infrastructure when you need it. This speeds up your training jobs and lets you iterate faster.
* **Automatic deployment**: Serverless SFT automatically deploys every checkpoint you train, so you don't need to manually set up hosting infrastructure. You can access and test trained models immediately in local, staging, or production environments.

## How Serverless SFT uses W\&B services

Serverless SFT uses a combination of the following W\&B components to operate:

* [Inference](/inference): To run your models.
* [Models](/models): To track performance metrics during the LoRA adapter's training.
* [Artifacts](/models/artifacts): To store and version the LoRA adapters.
* [Weave](/weave) (Optional): To gain observability into how the model responds at each step of the training loop.

<Note>
  Serverless SFT is in public preview. During the preview, W\&B charges you only for inference usage and artifact storage. W\&B doesn't charge for adapter training during the preview period.
</Note>


# How to use Serverless SFT
Source: https://docs.wandb.ai/serverless-training/sft/usage

Fine-tune models with Serverless SFT using OpenPipe's ART framework and the Serverless Training API for supervised learning.

Use Serverless supervised fine-tuning (SFT) through [OpenPipe's ART framework](https://art.openpipe.ai/getting-started/about) and the Serverless Training API.

To start using Serverless SFT, satisfy the [prerequisites](/serverless-training/prerequisites) for W\&B tools, and then see the [ART Serverless SFT documentation](https://art.openpipe.ai/fundamentals/sft-training).

To learn about Serverless SFT's API endpoints, see the [Serverless Training API reference](/serverless-training/api-reference).


# How to use Serverless RL
Source: https://docs.wandb.ai/serverless-training/usage

Post-train models with Serverless RL using OpenPipe's ART framework and the Serverless Training API for reinforcement learning.

Use Serverless RL through [OpenPipe's ART framework](https://art.openpipe.ai/getting-started/about) and the [Serverless Training API](/serverless-training/api-reference).

To start using Serverless RL, you must satisfy the [prerequisites](/serverless-training/prerequisites) to use W\&B tools, and then go through the ART [quickstart](https://art.openpipe.ai/getting-started/quick-start).

For code examples and workflows, see the [Google Colab notebook](https://colab.research.google.com/github/openpipe/art-notebooks/blob/main/examples/2048/2048.ipynb).


# Usage information and limits
Source: https://docs.wandb.ai/serverless-training/usage-limits

Understand pricing, usage limits, and account restrictions for W&B Serverless Training

This page describes the pricing model, concurrency limits, and geographic restrictions that apply to W\&B Serverless Training. Review this information to estimate costs and to understand the constraints that affect how you run training and inference workloads.

## Pricing

Pricing has three components: inference, training, and storage. For specific billing rates, visit our [pricing page](https://site.wandb.ai/pricing/training). The following sections describe each component.

### Inference

Pricing for Serverless Training inference requests matches Serverless Inference pricing. See [model-specific costs](https://site.wandb.ai/pricing/reinforcement-learning). Learn more about purchasing credits, account tiers, and usage caps in the [Serverless Inference docs](/inference/usage-limits#purchase-more-credits).

### Training

At each training step, Serverless Training collects batches of trajectories that include your agent's outputs and associated rewards (calculated by your reward function). Serverless Training uses the batched trajectories to update the weights of a LoRA adapter that specializes a base model for your task. The training jobs to update these LoRAs run on dedicated GPU clusters that Serverless Training manages.

Training is free during the public preview period.

### Model storage

Serverless Training stores checkpoints of your trained LoRAs so you can evaluate, serve, or continue training them at any time. W\&B bills storage monthly based on total checkpoint size and your [pricing plan](https://wandb.ai/site/pricing). Every plan includes at least 5 GB of free storage, which is enough for roughly 30 LoRAs. To save space, delete low-performing LoRAs. See the [ART SDK](https://art.openpipe.ai/features/checkpoint-deletion) for instructions.

## Limits

The following limits apply to Serverless Training usage. Review them when sizing workloads or when planning to use the service from a new region.

* **Inference concurrency limits**: By default, Serverless Training supports up to 2,000 concurrent requests per user and 6,000 per project. If you exceed your rate limit, the Inference API returns a `429 Concurrency limit reached for requests` response. To avoid this error, reduce the number of concurrent requests your training job or production workload makes at once. If you need a higher rate limit, request one at [support@wandb.com](mailto:support@wandb.com).

* **Geographic restrictions**: Serverless Training is only available in supported geographic locations. For more information, see the [Terms of Service](https://site.wandb.ai/terms/).


# Use your trained models
Source: https://docs.wandb.ai/serverless-training/use-trained-models

Make inference requests to the models you've trained

After you train a model with Serverless RL, it's automatically available for inference. This page shows you how to construct the endpoint for a trained model and send inference requests to it. Use this endpoint to integrate your model into your application or evaluation workflows.

To send requests to your trained model, you need the following:

* Your [W\&B API key](https://wandb.ai/settings)
* The [Serverless Training API's](/serverless-training/api-reference) base URL, `https://api.training.wandb.ai/v1/`
* Your model's endpoint

The model's endpoint uses the following schema:

```text theme={null}
wandb-artifact:///[ENTITY]/[PROJECT]/[MODEL-NAME]:[STEP]
```

The schema consists of:

* Your W\&B entity's (team) name
* The name of the project associated with your model
* The trained model's name
* The training step of the model you want to deploy. This is usually the step where the model performed best in your evaluations.

For example, if your W\&B team is named `email-specialists`, your project is called `mail-search`, your trained model is named `agent-001`, and you want to deploy it on step 25, the endpoint looks like this:

```text theme={null}
wandb-artifact:///email-specialists/mail-search/agent-001:step25
```

After you have your endpoint, you can integrate it into your normal inference workflows. The following examples show how to make inference requests to your trained model using a cURL request or the [Python OpenAI SDK](https://github.com/openai/openai-python). Choose the example that matches your environment.

## cURL

```bash theme={null}
curl https://api.training.wandb.ai/v1/chat/completions \
    -H "Authorization: Bearer $WANDB_API_KEY" \
    -H "Content-Type: application/json" \
    -d '{
            "model": "wandb-artifact://[ENTITY]/[PROJECT]/[MODEL-NAME]:[STEP]",
            "messages": [
            {"role": "system", "content": "You are a helpful assistant."},
            {"role": "user", "content": "Summarize our training run."}
            ],
            "temperature": 0.7,
            "top_p": 0.95
        }'
```

## OpenAI SDK

```python theme={null}
from openai import OpenAI

WANDB_API_KEY = "your-wandb-api-key"
ENTITY = "my-entity"
PROJECT = "my-project"

client = OpenAI(
    base_url="https://api.training.wandb.ai/v1",
    api_key=WANDB_API_KEY
)

response = client.chat.completions.create(
    model=f"wandb-artifact:///{ENTITY}/{PROJECT}/my-model:step100",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Summarize our training run."},
    ],
    temperature=0.7,
    top_p=0.95,
)

print(response.choices[0].message.content)
```


# Overview
Source: https://docs.wandb.ai/weave/cookbooks

Practical guides and examples for building with Weights & Biases Weave

Welcome to the W\&B Weave cookbook collection. These hands-on guides provide practical examples and best practices for building applications with Weave. Use this page to browse cookbooks by topic and jump into the example that matches what you're trying to build.

<CardGroup>
  <Card title="Getting started" icon="rocket" href="/weave/cookbooks/Intro_to_Weave_Hello_Trace">
    Start with Weave through practical, hands-on examples covering tracing and evaluation basics.
  </Card>

  <Card title="Evaluations and datasets" icon="chart-line" href="/weave/cookbooks/leaderboard_quickstart">
    Learn how to create evaluations, work with datasets, and build leaderboards for model comparison.
  </Card>

  <Card title="Models and prompts" icon="brain" href="/weave/cookbooks/Models_and_Weave_Integration_Demo">
    Explore model management, prompt engineering, and optimization techniques with real-world examples.
  </Card>

  <Card title="Advanced topics" icon="code-branch" href="/weave/cookbooks/multi-agent-structured-output">
    Explore complex use cases including multi-agent systems, code generation, and specialized pipelines.
  </Card>

  <Card title="Production and monitoring" icon="shield-check" href="/weave/cookbooks/online_monitoring">
    Implement production-ready patterns for monitoring, feedback collection, and safety guardrails.
  </Card>

  <Card title="API and integration" icon="plug" href="/weave/cookbooks/weave_via_service_api">
    Learn how to integrate Weave using the service API and connect with external systems.
  </Card>
</CardGroup>

## Featured cookbooks

The following sections list featured cookbooks grouped by topic, from getting started through production and API integration.

### Getting started

* [Hello Trace](/weave/cookbooks/Intro_to_Weave_Hello_Trace) - Take your first steps with Weave tracing.
* [Hello Eval](/weave/cookbooks/Intro_to_Weave_Hello_Eval) - Get introduced to evaluation concepts.

### Evaluations and datasets

* [Leaderboard Quickstart](/weave/cookbooks/leaderboard_quickstart) - Build model comparison leaderboards.
* [Hugging Face Datasets](/weave/cookbooks/hf_dataset_evals) - Evaluate models using Hugging Face datasets.
* [Import from CSV](/weave/cookbooks/import_from_csv) - Load and work with CSV data.

### Models and prompts

* [Models and Weave Integration](/weave/cookbooks/Models_and_Weave_Integration_Demo) - Integrate models with Weave.
* [Chain of Density](/weave/cookbooks/chain_of_density) - Apply advanced prompting techniques.
* [DSPy Prompt Optimization](/weave/cookbooks/dspy_prompt_optimization) - Optimize prompts programmatically.
* [NotDiamond Custom Routing](/weave/cookbooks/notdiamond_custom_routing) - Implement intelligent model routing.

### Advanced topics

* [Multi-Agent Structured Output](/weave/cookbooks/multi-agent-structured-output) - Build complex multi-agent systems.
* [Code Generation](/weave/cookbooks/codegen) - Build code generation pipelines.
* [OCR Pipeline](/weave/cookbooks/ocr-pipeline) - Process documents with OCR.
* [Audio Processing](/weave/cookbooks/audio_with_weave) - Work with audio data in Weave.

### Production and monitoring

* [Online Monitoring](/weave/cookbooks/online_monitoring) - Monitor systems in real time.
* [Production Feedback](/weave/cookbooks/feedback_prod) - Collect and analyze user feedback.
* [Scorers as Guardrails](/weave/cookbooks/scorers_as_guardrails) - Implement safety measures.
* [Custom Model Costs](/weave/cookbooks/custom_model_cost) - Track and optimize costs.
* [PII Protection](/weave/cookbooks/pii) - Handle sensitive data safely.

### API and integration

* [Service API Usage](/weave/cookbooks/weave_via_service_api) - Integrate directly with the service API.


# Introduction to Evaluations
Source: https://docs.wandb.ai/weave/cookbooks/Intro_to_Weave_Hello_Eval

Learn how to use introduction to evaluations with W&B Weave

<Note>
  This is an interactive notebook. You can run it locally or use the following links:

  * [Open in Google Colab](https://colab.research.google.com/github/wandb/docs/blob/main/weave/cookbooks/source/Intro_to_Weave_Hello_Eval.ipynb)
  * [View source on GitHub](https://github.com/wandb/docs/blob/main/weave/cookbooks/source/Intro_to_Weave_Hello_Eval.ipynb)
</Note>

This notebook introduces W\&B Weave evaluations by walking you through a minimal end-to-end example. You define a Weave `Model`, run it against a small dataset, score the outputs with a custom scoring function, and review the results in Weave. It's intended for developers who are new to Weave and want a quick, hands-on starting point before exploring more advanced evaluation workflows.

## Prerequisites

Before you can run a Weave evaluation, complete the following prerequisites.

1. Install the W\&B Weave SDK and log in with your [API key](https://wandb.ai/settings#apikeys).
2. Install the OpenAI SDK and log in with your [API key](https://platform.openai.com/api-keys).
3. Initialize your W\&B project.

```python lines theme={null}
# Install dependencies and imports
!pip install wandb weave openai -q

import os
from getpass import getpass

from openai import OpenAI
from pydantic import BaseModel

import weave

# 🔑 Setup your API keys
# Running this cell will prompt you for your API key with `getpass` and will not echo to the terminal.
#####
print("---")
print(
    "Create a W&B API key at: https://wandb.ai/settings#apikeys"
)
os.environ["WANDB_API_KEY"] = getpass("Enter your W&B API key: ")
print("---")
print("You can generate your OpenAI API key here: https://platform.openai.com/api-keys")
os.environ["OPENAI_API_KEY"] = getpass("Enter your OpenAI API key: ")
print("---")
#####

# 🏠 Enter your W&B project name
weave_client = weave.init("MY_PROJECT_NAME")  # 🐝 Your W&B project name
```

## Run your first evaluation

With your environment set up, you're ready to define and run an evaluation.

The following code sample shows how to evaluate an LLM using Weave's `Model` and `Evaluation` APIs. First, define a Weave model by subclassing `weave.Model`, specifying the model name and prompt format, and tracking a `predict` method with `@weave.op`. The `predict` method sends a prompt to OpenAI and parses the response into a structured output using a Pydantic schema (`FruitExtract`). Then, create a small evaluation dataset that consists of input sentences and expected targets. Next, define a custom scoring function (also tracked using `@weave.op`) that compares the model's output to the target label. Finally, wrap everything in a `weave.Evaluation`, specifying your dataset and scorers, and call `evaluate()` to run the evaluation pipeline asynchronously.

```python lines theme={null}
# 1. Construct a Weave model
class FruitExtract(BaseModel):
    fruit: str
    color: str
    flavor: str

class ExtractFruitsModel(weave.Model):
    model_name: str
    prompt_template: str

    @weave.op()
    def predict(self, sentence: str) -> dict:
        client = OpenAI()

        response = client.beta.chat.completions.parse(
            model=self.model_name,
            messages=[
                {
                    "role": "user",
                    "content": self.prompt_template.format(sentence=sentence),
                }
            ],
            response_format=FruitExtract,
        )
        result = response.choices[0].message.parsed
        return result

model = ExtractFruitsModel(
    name="gpt4o",
    model_name="gpt-4o",
    prompt_template='Extract fields ("fruit": <str>, "color": <str>, "flavor": <str>) as json, from the following text : {sentence}',
)

# 2. Collect some samples
sentences = [
    "There are many fruits that were found on the recently discovered planet Goocrux. There are neoskizzles that grow there, which are purple and taste like candy.",
    "Pounits are a bright green color and are more savory than sweet.",
    "Finally, there are fruits called glowls, which have a very sour and bitter taste which is acidic and caustic, and a pale orange tinge to them.",
]
labels = [
    {"fruit": "neoskizzles", "color": "purple", "flavor": "candy"},
    {"fruit": "pounits", "color": "green", "flavor": "savory"},
    {"fruit": "glowls", "color": "orange", "flavor": "sour, bitter"},
]
examples = [
    {"id": "0", "sentence": sentences[0], "target": labels[0]},
    {"id": "1", "sentence": sentences[1], "target": labels[1]},
    {"id": "2", "sentence": sentences[2], "target": labels[2]},
]

# 3. Define a scoring function for your evaluation
@weave.op()
def fruit_name_score(target: dict, output: FruitExtract) -> dict:
    target_flavors = [f.strip().lower() for f in target["flavor"].split(",")]
    output_flavors = [f.strip().lower() for f in output.flavor.split(",")]
    # Check if any target flavor is present in the output flavors
    matches = any(tf in of for tf in target_flavors for of in output_flavors)
    return {"correct": matches}

# 4. Run your evaluation
evaluation = weave.Evaluation(
    name="fruit_eval",
    dataset=examples,
    scorers=[fruit_name_score],
)
await evaluation.evaluate(model)
```

After the evaluation finishes, Weave logs the model, dataset, and per-example scores to your project so you can inspect the results in the Weave UI.

## Looking for more examples

Now that you've run a basic evaluation, you can explore more advanced workflows in the following tutorials:

* Learn how to [build an evaluation pipeline end-to-end](/weave/tutorial-eval).
* Learn how to [evaluate a RAG application](/weave/tutorial-rag).


# Introduction to Traces
Source: https://docs.wandb.ai/weave/cookbooks/Intro_to_Weave_Hello_Trace

Learn how to use introduction to traces with W&B Weave

<Note>
  This is an interactive notebook. You can run it locally or use the following links:

  * [Open in Google Colab](https://colab.research.google.com/github/wandb/docs/blob/main/weave/cookbooks/source/Intro_to_Weave_Hello_Trace.ipynb)
  * [View source on GitHub](https://github.com/wandb/docs/blob/main/weave/cookbooks/source/Intro_to_Weave_Hello_Trace.ipynb)
</Note>

This notebook shows you how to capture your first trace in W\&B Weave by decorating an OpenAI-powered function with `@weave.op`. Tracing helps you inspect the inputs, outputs, and intermediate steps of LLM calls so you can debug and improve your applications. This tutorial targets developers who are new to Weave and want a minimal end-to-end example.

## 🔑 Prerequisites

Before you can begin tracing in Weave, complete the following prerequisites. These steps install the required SDKs, authenticate you with W\&B and OpenAI, and initialize a project to send traces to.

1. Install the Weave SDK and log in with your [API key](https://wandb.ai/settings#apikeys).
2. Install the OpenAI SDK and log in with your [API key](https://platform.openai.com/api-keys).
3. Initialize your W\&B project.

```python lines theme={null}
# Install dependencies and imports
!pip install wandb weave openai -q

import json
import os
from getpass import getpass

from openai import OpenAI

import weave

# 🔑 Setup your API keys
# Running this cell will prompt you for your API key with `getpass` and will not echo to the terminal.
#####
print("---")
print(
    "Create a W&B API key at: https://wandb.ai/settings#apikeys"
)
os.environ["WANDB_API_KEY"] = getpass("Enter your W&B API key: ")
print("---")
print("You can generate your OpenAI API key here: https://platform.openai.com/api-keys")
os.environ["OPENAI_API_KEY"] = getpass("Enter your OpenAI API key: ")
print("---")
#####

# 🏠 Enter your W&B project name
weave_client = weave.init("MY_PROJECT_NAME")  # 🐝 Your W&B project name
```

## 🐝 Run your first trace

The following code sample shows how to capture and visualize a trace in Weave using the `@weave.op` decorator. It defines a function called `extract_fruit` that sends a prompt to OpenAI's GPT-4o to extract structured data (fruit, color, and flavor) from a sentence. By decorating the function with `@weave.op`, Weave automatically tracks the function execution, including inputs, outputs, and intermediate steps. When you call the function with a sample sentence, Weave saves the full trace and makes it viewable in the Weave UI.

```python lines theme={null}
@weave.op()  # 🐝 Decorator to track requests
def extract_fruit(sentence: str) -> dict:
    client = OpenAI()
    system_prompt = (
        "Parse sentences into a JSON dict with keys: fruit, color and flavor."
    )
    response = client.chat.completions.create(
        model="gpt-4o",
        messages=[
            {"role": "system", "content": system_prompt},
            {"role": "user", "content": sentence},
        ],
        temperature=0.7,
        response_format={"type": "json_object"},
    )
    extracted = response.choices[0].message.content
    return json.loads(extracted)

sentence = "There are many fruits that were found on the recently discovered planet Goocrux. There are neoskizzles that grow there, which are purple and taste like candy."
extract_fruit(sentence)
```

You now have a working Weave project that captures traces from a decorated function, and you can view the trace in the Weave UI.

## 🚀 More examples

Use the following resources to go deeper with Weave tracing:

* [Quickstart guide](/weave/quickstart)
* [Advanced tracing topics](/weave/tutorial-tracing_2)
* [Tracing in Weave](/weave/guides/tracking/tracing)


# Use Weave with W&B Models
Source: https://docs.wandb.ai/weave/cookbooks/Models_and_Weave_Integration_Demo

Interactive notebook that walks through tracking experiments in W&B Models alongside LLM traces and evaluations in Weave.

This notebook walks through an end-to-end workflow that combines W\&B Models and Weave to build, evaluate, and publish a Retrieval-Augmented Generation (RAG) application. You retrieve a fine-tuned chat model from the W\&B Models Registry, swap it into an existing `RagModel` tracked in Weave, evaluate the updated application with `weave.Evaluation`, and publish the new RAG model back to the Registry. This workflow is intended for teams who train and fine-tune models with W\&B Models and want to integrate them into LLM applications tracked and evaluated with Weave.

<Note>
  This is an interactive notebook. You can run it locally or use the following links:

  * [Open in Google Colab](https://colab.research.google.com/github/wandb/docs/blob/main/weave/cookbooks/source/Models_and_Weave_Integration_Demo.ipynb)
  * [View source on GitHub](https://github.com/wandb/docs/blob/main/weave/cookbooks/source/Models_and_Weave_Integration_Demo.ipynb)
</Note>

## Prerequisites

First, install the required libraries, set up API keys, log in to W\&B, and create a new W\&B project.

1. Install `weave`, `pandas`, `unsloth`, `wandb`, `litellm`, `pydantic`, `torch`, and `faiss-gpu` using `pip`.

```python lines theme={null}
%%capture
!pip install weave wandb pandas pydantic litellm faiss-gpu
python
%%capture
!pip install unsloth
# Also get the latest nightly Unsloth!
!pip uninstall unsloth -y && pip install --upgrade --no-cache-dir "unsloth[colab-new] @ git+https://github.com/unslothai/unsloth.git"
```

2. Add the required API keys from your environment.

```python lines theme={null}
import os

from google.colab import userdata

os.environ["WANDB_API_KEY"] = userdata.get("WANDB_API_KEY")  # W&B Models and Weave
os.environ["OPENAI_API_KEY"] = userdata.get(
    "OPENAI_API_KEY"
)  # OpenAI - for retrieval embeddings
os.environ["GEMINI_API_KEY"] = userdata.get(
    "GEMINI_API_KEY"
)  # Gemini - for the base chat model
```

3. Log in to W\&B, and create a new project.

```python lines theme={null}
import pandas as pd
import wandb

import weave

wandb.login()

PROJECT = "weave-cookboook-demo"
ENTITY = "wandb-smle"

weave.init(ENTITY + "/" + PROJECT)
```

## Download `ChatModel` from Models Registry and implement `UnslothLoRAChatModel`

In this scenario, the Model Team has already fine-tuned the Llama-3.2 model with the `unsloth` library for performance optimization, and the model is available in the W\&B Models Registry. In this step, you retrieve the fine-tuned [`ChatModel`](https://wandb.ai/wandb-smle/weave-cookboook-demo/weave/object-versions?filter=%7B%22objectName%22%3A%22RagModel%22%7D\&peekPath=%2Fwandb-smle%2Fweave-rag-experiments%2Fobjects%2FChatModelRag%2Fversions%2F2mhdPb667uoFlXStXtZ0MuYoxPaiAXj3KyLS1kYRi84%3F%26) from the Registry and convert it into a `weave.Model` to make it compatible with the [`RagModel`](https://wandb.ai/wandb-smle/weave-cookboook-demo/weave/object-versions?filter=%7B%22objectName%22%3A%22RagModel%22%7D\&peekPath=%2Fwandb-smle%2Fweave-cookboook-demo%2Fobjects%2FRagModel%2Fversions%2FcqRaGKcxutBWXyM0fCGTR1Yk2mISLsNari4wlGTwERo%3F%26).

<Note>
  The `RagModel` referenced in the following code is a top-level `weave.Model` that can be considered a complete RAG Application. It contains a `ChatModel`, vector database, and a prompt. The `ChatModel` is also a `weave.Model`, which contains code to download an artifact from the W\&B Registry. `ChatModel` can be changed modularly to support any kind of other LLM chat model as part of the `RagModel`. For more information, [view the model in Weave](https://wandb.ai/wandb-smle/weave-cookboook-demo/weave/evaluations?peekPath=%2Fwandb-smle%2Fweave-cookboook-demo%2Fobjects%2FRagModel%2Fversions%2Fx7MzcgHDrGXYHHDQ9BA8N89qDwcGkdSdpxH30ubm8ZM%3F%26).
</Note>

To load the `ChatModel`, use `unsloth.FastLanguageModel` or `peft.AutoPeftModelForCausalLM` with adapters for efficient integration into the app. After you download the model from the Registry, set up the initialization and prediction logic with the `model_post_init` method. The required code for this step is available in the **Use** tab of the Registry, and you can copy it directly into your implementation.

The following code defines the `UnslothLoRAChatModel` class to manage, initialize, and use the fine-tuned Llama-3.2 model retrieved from the W\&B Models Registry. `UnslothLoRAChatModel` uses `unsloth.FastLanguageModel` for optimized inference. The `model_post_init` method downloads and sets up the model, while the `predict` method processes user queries and generates responses. To adapt the code for your use case, update the `MODEL_REG_URL` with the correct Registry path for your fine-tuned model, and adjust parameters like `max_seq_length` or `dtype` based on your hardware or requirements.

```python lines theme={null}
from typing import Any

from pydantic import PrivateAttr
from unsloth import FastLanguageModel

import weave

class UnslothLoRAChatModel(weave.Model):
    """
    We define an extra ChatModel class to be able store and version more parameters than just the model name.
    Especially, relevant if we consider fine-tuning (locally or aaS) because of specific parameters.
    """

    chat_model: str
    cm_temperature: float
    cm_max_new_tokens: int
    cm_quantize: bool
    inference_batch_size: int
    dtype: Any
    device: str
    _model: Any = PrivateAttr()
    _tokenizer: Any = PrivateAttr()

    def model_post_init(self, __context):
        # we can simply paste this from the "Use" tab from the registry
        run = wandb.init(project=PROJECT, job_type="model_download")
        artifact = run.use_artifact(f"{self.chat_model}")
        model_path = artifact.download()

        # unsloth version (enable native 2x faster inference)
        self._model, self._tokenizer = FastLanguageModel.from_pretrained(
            model_name=model_path,
            max_seq_length=self.cm_max_new_tokens,
            dtype=self.dtype,
            load_in_4bit=self.cm_quantize,
        )
        FastLanguageModel.for_inference(self._model)

    @weave.op()
    async def predict(self, query: list[str]) -> dict:
        # add_generation_prompt = true - Must add for generation
        input_ids = self._tokenizer.apply_chat_template(
            query,
            tokenize=True,
            add_generation_prompt=True,
            return_tensors="pt",
        ).to("cuda")

        output_ids = self._model.generate(
            input_ids=input_ids,
            max_new_tokens=64,
            use_cache=True,
            temperature=1.5,
            min_p=0.1,
        )

        decoded_outputs = self._tokenizer.batch_decode(
            output_ids[0][input_ids.shape[1] :], skip_special_tokens=True
        )

        return "".join(decoded_outputs).strip()
python
MODEL_REG_URL = "wandb32/wandb-registry-RAG Chat Models/Finetuned Llama-3.2:v3"

max_seq_length = 2048  # Choose any! We auto support RoPE Scaling internally!
dtype = (
    None  # None for auto detection. Float16 for Tesla T4, V100, Bfloat16 for Ampere+
)
load_in_4bit = True  # Use 4bit quantization to reduce memory usage. Can be False.

new_chat_model = UnslothLoRAChatModel(
    name="UnslothLoRAChatModelRag",
    chat_model=MODEL_REG_URL,
    cm_temperature=1.0,
    cm_max_new_tokens=max_seq_length,
    cm_quantize=load_in_4bit,
    inference_batch_size=max_seq_length,
    dtype=dtype,
    device="auto",
)
python
await new_chat_model.predict(
    [{"role": "user", "content": "What is the capital of Germany?"}]
)
```

## Integrate the new `ChatModel` version into `RagModel`

Building a RAG application from a fine-tuned chat model lets you reuse tailored components without rebuilding the entire pipeline. In this step, you retrieve the existing `RagModel` from your Weave project and update its `ChatModel` to use the fine-tuned model. Swapping in the new chat model leaves the other components, like the vector database and prompts, untouched, preserving the application's overall structure while improving performance.

The following code retrieves the `RagModel` object using a reference from the Weave project. The code then updates the `chat_model` attribute of the `RagModel` to use the new `UnslothLoRAChatModel` instance created in the previous step. After this, the code publishes the updated `RagModel` to create a new version. Finally, the code runs a sample prediction query with the updated `RagModel` to verify that it uses the new chat model.

```python lines theme={null}
RagModel = weave.ref(
    "weave://wandb-smle/weave-cookboook-demo/object/RagModel:cqRaGKcxutBWXyM0fCGTR1Yk2mISLsNari4wlGTwERo"
).get()
python
RagModel.chat_model.chat_model
python
await RagModel.predict("When was the first conference on climate change?")
python
# MAGIC: exchange chat_model and publish new version (no need to worry about other RAG components)
RagModel.chat_model = new_chat_model
python
RagModel.chat_model.chat_model
python
# first publish new version so that in prediction we reference new version
PUB_REFERENCE = weave.publish(RagModel, "RagModel")
python
await RagModel.predict("When was the first conference on climate change?")
```

## Run a `weave.Evaluation`

With the updated `RagModel` published, the next step is to confirm that the new fine-tuned chat model performs as expected within the application. In this step, you evaluate the performance of the updated `RagModel` with an existing `weave.Evaluation`. This process confirms that the new fine-tuned chat model performs as expected within the RAG application. To streamline integration and enable collaboration between the Models and Apps teams, you log evaluation results to both the model's W\&B run and the Weave workspace.

In Models:

* The evaluation summary is logged to the W\&B run used to download the fine-tuned chat model. This includes summary metrics and graphs displayed in a [workspace view](https://wandb.ai/wandb-smle/weave-cookboook-demo/workspace?nw=eglm8z7o9) for analysis.
* The evaluation trace ID is added to the run's configuration, linking directly to the Weave page for improved traceability by the Model Team.

In Weave:

* The artifact or registry link for the `ChatModel` is stored as an input to the `RagModel`.
* The W\&B run ID is saved as an extra column in the evaluation traces for better context.

The following code demonstrates how to retrieve an evaluation object, run the evaluation with the updated `RagModel`, and log the results to both W\&B and Weave. Ensure that the evaluation reference (`WEAVE_EVAL`) matches your project setup.

```python lines theme={null}
# MAGIC: we can simply get an evaluation with a eval dataset and scorers and use them
WEAVE_EVAL = "weave://wandb-smle/weave-cookboook-demo/object/climate_rag_eval:ntRX6qn3Tx6w3UEVZXdhIh1BWGh7uXcQpOQnIuvnSgo"
climate_rag_eval = weave.ref(WEAVE_EVAL).get()
python
with weave.attributes({"wandb-run-id": wandb.run.id}):
    # use .call attribute to retrieve both the result and the call in order to save eval trace to Models
    summary, call = await climate_rag_eval.evaluate.call(climate_rag_eval, RagModel)
python
# log to models
wandb.run.log(pd.json_normalize(summary, sep="/").to_dict(orient="records")[0])
wandb.run.config.update(
    {"weave_url": f"https://wandb.ai/wandb-smle/weave-cookboook-demo/r/call/{call.id}"}
)
wandb.run.finish()
```

## Save the new RAG model to the Registry

Now that you've evaluated the updated `RagModel`, the final step is to publish it back to the W\&B Models Registry so other teams can discover and reuse it. To make the updated `RagModel` available for future use by both the Models and Apps teams, push it to the W\&B Models Registry as a reference artifact.

The following code retrieves the `weave` object version and name for the updated `RagModel` and uses them to create reference links. The code then creates a new artifact in W\&B with metadata containing the model's Weave URL. The code logs this artifact to the W\&B Registry and links it to a designated registry path.

Before running the code, ensure the `ENTITY` and `PROJECT` variables match your W\&B setup, and specify the correct target registry path. This process finalizes the workflow by publishing the new `RagModel` to the W\&B ecosystem for collaboration and reuse.

After running the code in this section, your updated `RagModel` is available in the W\&B Models Registry as a referenced artifact, completing the round-trip between W\&B Models and Weave.

```python lines theme={null}
MODELS_OBJECT_VERSION = PUB_REFERENCE.digest  # weave object version
MODELS_OBJECT_NAME = PUB_REFERENCE.name  # weave object name
python
models_url = f"https://wandb.ai/{ENTITY}/{PROJECT}/weave/objects/{MODELS_OBJECT_NAME}/versions/{MODELS_OBJECT_VERSION}"
models_link = (
    f"weave://{ENTITY}/{PROJECT}/object/{MODELS_OBJECT_NAME}:{MODELS_OBJECT_VERSION}"
)

with wandb.init(project=PROJECT, entity=ENTITY) as run:
    # create new Artifact
    artifact_model = wandb.Artifact(
        name="RagModel",
        type="model",
        description="Models Link from RagModel in Weave",
        metadata={"url": models_url},
    )
    artifact_model.add_reference(models_link, name="model", checksum=False)

    # log new artifact
    run.log_artifact(artifact_model, aliases=[MODELS_OBJECT_VERSION])

    # link to registry
    run.link_artifact(
        artifact_model, target_path="wandb32/wandb-registry-RAG Models/RAG Model"
    )
```


# Audio with Weave
Source: https://docs.wandb.ai/weave/cookbooks/audio_with_weave

Track and evaluate audio data in your LLM applications using W&B Weave's audio logging and tracing features.

<Note>
  This is an interactive notebook. You can run it locally or use the following links:

  * [Open in Google Colab](https://colab.research.google.com/github/wandb/docs/blob/main/weave/cookbooks/source/audio_with_weave.ipynb)
  * [View source on GitHub](https://github.com/wandb/docs/blob/main/weave/cookbooks/source/audio_with_weave.ipynb)
</Note>

This notebook shows you how to use Weave to log and trace audio generated by OpenAI's audio APIs, so you can inspect prompts, audio outputs, and transcripts alongside the rest of your LLM application data. The notebook targets developers who already build with OpenAI's audio models and want observability into their audio traces.

The notebook first uses the OpenAI chat completions API with GPT 4o Audio Preview to generate audio responses to text prompts and track these in Weave.

<img alt="OpenAI chat completions API interface with GPT 4o Audio Preview integration and audio response generation workflow" />

For the advanced use case, the notebook uses the OpenAI Realtime API to stream audio in real time, so you can see how Weave captures both sides of a live conversation. Click the following thumbnail to view the [video demonstration](https://www.youtube.com/watch?v=lnnd73xDElw).

[![Video thumbnail for the Weave Realtime audio demonstration](https://img.youtube.com/vi/lnnd73xDElw/0.jpg)](https://www.youtube.com/watch?v=lnnd73xDElw "Weave Realtime audio demonstration")

## Setup

This section installs the Python packages, loads API credentials, and imports the libraries needed for the chat completions example.

Start by installing the OpenAI (`openai`) and Weave (`weave`) dependencies, as well as the API key management dependency `set-env`.

```python lines theme={null}
%%capture
!pip install openai
!pip install weave
!pip install set-env-colab-kaggle-dotenv -q # for env var
python
%%capture
# Temporary workaround to fix bug in openai:
# TypeError: Client.__init__() got an unexpected keyword argument 'proxies'
# See https://community.openai.com/t/error-with-openai-1-56-0-client-init-got-an-unexpected-keyword-argument-proxies/1040332/15
!pip install "httpx<0.28"
```

Next, load the required API keys for OpenAI and Weave. This example uses `set_env`, which is compatible with Google Colab's secret keys manager and serves as an alternative to Colab's specific `google.colab.userdata`. See the [`set-env-colab-kaggle-dotenv` usage instructions](https://pypi.org/project/set-env-colab-kaggle-dotenv/).

```python lines theme={null}
# Set environment variables.
from set_env import set_env

_ = set_env("OPENAI_API_KEY")
_ = set_env("WANDB_API_KEY")
```

Finally, import the required libraries.

```python lines theme={null}
import base64
import os
import time
import wave

import numpy as np
from IPython.display import display
from openai import OpenAI

import weave
```

## Audio streaming and storage example

With the dependencies installed and credentials loaded, you can now set up a call to OpenAI's completions endpoint with audio modality enabled. First, create the OpenAI client and initiate a Weave project so that Weave logs subsequent calls to your workspace.

```python lines theme={null}
client = OpenAI(api_key=os.environ.get("OPENAI_API_KEY"))
weave.init("openai-audio-chat")
```

Now, define the OpenAI completions request and add the Weave decorator (op). The `@weave.op()` decorator instructs Weave to capture the function's inputs, outputs, and audio file in a trace.

The following code defines the function `prompt_endpoint_and_log_trace`. This function has three primary steps:

1. Make a completion object using the `gpt-4o-audio-preview` model, which supports text and audio inputs and outputs.

   * Prompt the model to count to 13 slowly with varying accents.
   * Set the completion to `stream`.

2. Open a new output file to receive the streamed data chunk by chunk.

3. Return an open file handler to the audio file so Weave logs the audio data in the trace.

```python lines theme={null}
SAMPLE_RATE = 22050

@weave.op()
def prompt_endpoint_and_log_trace(system_prompt=None, user_prompt=None):
    if not system_prompt:
        system_prompt = "You're the fastest counter in the world"
    if not user_prompt:
        user_prompt = "Count to 13 super super slow, enunciate each number with a dramatic flair, changing up accents as you go along. British, French, German, Spanish, etc."
    # Request from the OpenAI API with audio modality
    completion = client.chat.completions.create(
        model="gpt-4o-audio-preview",
        modalities=["text", "audio"],
        audio={"voice": "fable", "format": "pcm16"},
        stream=True,
        messages=[
            {"role": "system", "content": system_prompt},
            {"role": "user", "content": user_prompt},
        ],
    )

    # Open a wave file for writing
    with wave.open("./output.wav", "wb") as wav_file:
        wav_file.setnchannels(1)  # Mono
        wav_file.setsampwidth(2)  # 16-bit
        wav_file.setframerate(SAMPLE_RATE)  # Sample rate (adjust if needed)

        # Write chunks as they are streamed in from the API
        for chunk in completion:
            if (
                hasattr(chunk, "choices")
                and chunk.choices is not None
                and len(chunk.choices) > 0
                and hasattr(chunk.choices[0].delta, "audio")
                and chunk.choices[0].delta.audio.get("data") is not None
            ):
                # Decode the base64 audio data
                audio_data = base64.b64decode(chunk.choices[0].delta.audio.get("data"))

                # Write the current chunk to the wave file
                wav_file.writeframes(audio_data)

    # Return the file to Weave op
    return wave.open("output.wav", "rb")
```

## Testing

After defining the function, run the following cell to invoke it end-to-end and confirm that audio is generated and logged. Weave stores the system and user prompts in a trace along with the output audio.
After running the cell, click the trace link printed in the cell output to view your trace. At this point, you have a complete, traced chat-completions audio call in Weave.

```python lines theme={null}
from IPython.display import Audio

# Call the function to write the audio stream
prompt_endpoint_and_log_trace(
    system_prompt="You're the fastest counter in the world",
    user_prompt="Count to 13 super super slow, enunciate each number with a dramatic flair, changing up accents as you go along. British, French, German, Spanish, etc.",
)

# Display the updated audio stream
display(Audio("output.wav", rate=SAMPLE_RATE, autoplay=True))
```

## Advanced usage: Realtime API with Weave

The remainder of this cookbook walks through a more advanced example that uses OpenAI's Realtime API together with Weave to trace a live, bidirectional audio conversation.

<img alt="Realtime Audio API integration with Weave and streaming audio conversation interface" />

OpenAI's Realtime API is a conversational API for building real-time audio and text assistants.

Before you run the Realtime example, review the following requirements:

* Review the cells in [Microphone configuration](#microphone-configuration).
* Due to limitations of the Google Colab execution environment, you must run this on your host machine as a Jupyter Notebook. You can't run it in the browser.
  * On macOS, you must install `portaudio` through [Brew](https://formulae.brew.sh/formula/portaudio) for PyAudio to function.
* The `enable_audio_playback` toggle plays back the audio that the assistant outputs. Headphones are required if you enable this, because echo detection requires a complex implementation.

## Requirements setup

The Realtime example requires additional packages for audio I/O and websocket communication. Install them and reload your environment variables.

```python lines theme={null}
%%capture
!pip install numpy==2.0
!pip install weave
!pip install pyaudio # On mac, you may need to install portaudio first with `brew install portaudio`
!pip install websocket-client
!pip install set-env-colab-kaggle-dotenv -q # for env var
!pip install resampy
python
import io
import json
import os
import threading
from typing import Optional

import pyaudio
import resampy
import websocket
from set_env import set_env

import weave
python
# Set environment variables.
# See: https://pypi.org/project/set-env-colab-kaggle-dotenv/ for usage instructions.
_ = set_env("OPENAI_API_KEY")
_ = set_env("WANDB_API_KEY")
```

## Microphone configuration

Because the Realtime example records from your microphone and plays audio back through your speakers, you must tell PyAudio which devices to use.

Run the following cell to find all available audio devices. Then, populate the `INPUT_DEVICE_INDEX` and the `OUTPUT_DEVICE_INDEX` based on the devices listed. Your input device has at least one input channel, and your output device has at least one output channel.

```python lines theme={null}
# Get device list from pyaudio so we can configure the next cell
p = pyaudio.PyAudio()
devices_data = {i: p.get_device_info_by_index(i) for i in range(p.get_device_count())}
for i, device in devices_data.items():
    print(
        f"Found device @{i}: {device['name']} with sample rate: {device['defaultSampleRate']} and input channels: {device['maxInputChannels']} and output channels: {device['maxOutputChannels']}"
    )
python
INPUT_DEVICE_INDEX = 3  # @param                                                 # Choose based on device list above. Make sure device has > 0 input channels.
OUTPUT_DEVICE_INDEX = 12  # @param                                                # Chose based on device list above. Make sure device has > 0 output channels.
enable_audio_playback = True  # @param {type:"boolean"}                           # Toggle on assistant audio playback. Requires headphones.

# Audio recording and streaming parameters
INPUT_DEVICE_CHANNELS = devices_data[INPUT_DEVICE_INDEX][
    "maxInputChannels"
]  # From device list above
SAMPLE_RATE = int(
    devices_data[INPUT_DEVICE_INDEX]["defaultSampleRate"]
)  # From device list above
CHUNK = int(SAMPLE_RATE / 10)  # Samples per frame
SAMPLE_WIDTH = p.get_sample_size(pyaudio.paInt16)  # Samples per frame for the format
CHUNK_DURATION = 0.3  # Seconds of audio per chunk sent to OAI API
OAI_SAMPLE_RATE = (
    24000  # OAI Sample Rate is 24kHz, we need this to play or save assistant audio
)
OUTPUT_DEVICE_CHANNELS = 1  # Set to 1 for mono output
```

## OpenAI Realtime API schema implementation

The next sections build up the Realtime client piece by piece: the message schema, an audio writer, the Weave-instrumented model, and the recorder.

The OpenAI Python SDK doesn't yet provide Realtime API support. This example implements the complete OpenAI Realtime API schema in Pydantic for greater legibility, and may deprecate it once official support ships.

### Pydantic schema for the OpenAI Realtime API

```python lines theme={null}
from enum import Enum
from typing import Any, Literal, Union

from pydantic import BaseModel, Field, ValidationError

class BaseEvent(BaseModel):
    type: Union["ClientEventTypes", "ServerEventTypes"]
    event_id: Optional[str] = None  # Add event_id as an optional field for all events

    # def model_dump_json(self, *args, **kwargs):
    #     # Only include non-None fields
    #     return super().model_dump_json(*args, exclude_none=True, **kwargs)

class ChatMessage(BaseModel):
    role: Literal["user", "assistant"]
    content: str
    timestamp: float

""" CLIENT EVENTS """

class ClientEventTypes(str, Enum):
    SESSION_UPDATE = "session.update"
    CONVERSATION_ITEM_CREATE = "conversation.item.create"
    CONVERSATION_ITEM_TRUNCATE = "conversation.item.truncate"
    CONVERSATION_ITEM_DELETE = "conversation.item.delete"
    RESPONSE_CREATE = "response.create"
    RESPONSE_CANCEL = "response.cancel"
    INPUT_AUDIO_BUFFER_APPEND = "input_audio_buffer.append"
    INPUT_AUDIO_BUFFER_COMMIT = "input_audio_buffer.commit"
    INPUT_AUDIO_BUFFER_CLEAR = "input_audio_buffer.clear"
    ERROR = "error"

#### Session Update
class TurnDetection(BaseModel):
    type: Literal["server_vad"]
    threshold: float = Field(..., ge=0.0, le=1.0)
    prefix_padding_ms: int
    silence_duration_ms: int

class InputAudioTranscription(BaseModel):
    model: Optional[str] = None

class ToolParameterProperty(BaseModel):
    type: str

class ToolParameter(BaseModel):
    type: str
    properties: dict[str, ToolParameterProperty]
    required: list[str]

class Tool(BaseModel):
    type: Literal["function", "code_interpreter", "file_search"]
    name: Optional[str] = None
    description: Optional[str] = None
    parameters: Optional[ToolParameter] = None

class Session(BaseModel):
    modalities: Optional[list[str]] = None
    instructions: Optional[str] = None
    voice: Optional[str] = None
    input_audio_format: Optional[str] = None
    output_audio_format: Optional[str] = None
    input_audio_transcription: Optional[InputAudioTranscription] = None
    turn_detection: Optional[TurnDetection] = None
    tools: Optional[list[Tool]] = None
    tool_choice: Optional[str] = None
    temperature: Optional[float] = None
    max_output_tokens: Optional[int] = None

class SessionUpdate(BaseEvent):
    type: Literal[ClientEventTypes.SESSION_UPDATE] = ClientEventTypes.SESSION_UPDATE
    session: Session

#### Audio Buffers
class InputAudioBufferAppend(BaseEvent):
    type: Literal[ClientEventTypes.INPUT_AUDIO_BUFFER_APPEND] = (
        ClientEventTypes.INPUT_AUDIO_BUFFER_APPEND
    )
    audio: str

class InputAudioBufferCommit(BaseEvent):
    type: Literal[ClientEventTypes.INPUT_AUDIO_BUFFER_COMMIT] = (
        ClientEventTypes.INPUT_AUDIO_BUFFER_COMMIT
    )

class InputAudioBufferClear(BaseEvent):
    type: Literal[ClientEventTypes.INPUT_AUDIO_BUFFER_CLEAR] = (
        ClientEventTypes.INPUT_AUDIO_BUFFER_CLEAR
    )

#### Messages
class MessageContent(BaseModel):
    type: Literal["input_audio"]
    audio: str

class ConversationItemContent(BaseModel):
    type: Literal["input_text", "input_audio", "text", "audio"]
    text: Optional[str] = None
    audio: Optional[str] = None
    transcript: Optional[str] = None

class FunctionCallContent(BaseModel):
    call_id: str
    name: str
    arguments: str

class FunctionCallOutputContent(BaseModel):
    output: str

class ConversationItem(BaseModel):
    id: Optional[str] = None
    type: Literal["message", "function_call", "function_call_output"]
    status: Optional[Literal["completed", "in_progress", "incomplete"]] = None
    role: Literal["user", "assistant", "system"]
    content: list[
        Union[ConversationItemContent, FunctionCallContent, FunctionCallOutputContent]
    ]
    call_id: Optional[str] = None
    name: Optional[str] = None
    arguments: Optional[str] = None
    output: Optional[str] = None

class ConversationItemCreate(BaseEvent):
    type: Literal[ClientEventTypes.CONVERSATION_ITEM_CREATE] = (
        ClientEventTypes.CONVERSATION_ITEM_CREATE
    )
    item: ConversationItem

class ConversationItemTruncate(BaseEvent):
    type: Literal[ClientEventTypes.CONVERSATION_ITEM_TRUNCATE] = (
        ClientEventTypes.CONVERSATION_ITEM_TRUNCATE
    )
    item_id: str
    content_index: int
    audio_end_ms: int

class ConversationItemDelete(BaseEvent):
    type: Literal[ClientEventTypes.CONVERSATION_ITEM_DELETE] = (
        ClientEventTypes.CONVERSATION_ITEM_DELETE
    )
    item_id: str

#### Responses
class ResponseCreate(BaseEvent):
    type: Literal[ClientEventTypes.RESPONSE_CREATE] = ClientEventTypes.RESPONSE_CREATE

class ResponseCancel(BaseEvent):
    type: Literal[ClientEventTypes.RESPONSE_CANCEL] = ClientEventTypes.RESPONSE_CANCEL

# Update the Event union to include all event types
ClientEvent = Union[
    SessionUpdate,
    InputAudioBufferAppend,
    InputAudioBufferCommit,
    InputAudioBufferClear,
    ConversationItemCreate,
    ConversationItemTruncate,
    ConversationItemDelete,
    ResponseCreate,
    ResponseCancel,
]

""" SERVER EVENTS """

class ServerEventTypes(str, Enum):
    ERROR = "error"
    RESPONSE_AUDIO_TRANSCRIPT_DONE = "response.audio_transcript.done"
    RESPONSE_AUDIO_TRANSCRIPT_DELTA = "response.audio_transcript.delta"
    RESPONSE_AUDIO_DELTA = "response.audio.delta"
    SESSION_CREATED = "session.created"
    SESSION_UPDATED = "session.updated"
    CONVERSATION_CREATED = "conversation.created"
    INPUT_AUDIO_BUFFER_COMMITTED = "input_audio_buffer.committed"
    INPUT_AUDIO_BUFFER_CLEARED = "input_audio_buffer.cleared"
    INPUT_AUDIO_BUFFER_SPEECH_STARTED = "input_audio_buffer.speech_started"
    INPUT_AUDIO_BUFFER_SPEECH_STOPPED = "input_audio_buffer.speech_stopped"
    CONVERSATION_ITEM_CREATED = "conversation.item.created"
    CONVERSATION_ITEM_INPUT_AUDIO_TRANSCRIPTION_COMPLETED = (
        "conversation.item.input_audio_transcription.completed"
    )
    CONVERSATION_ITEM_INPUT_AUDIO_TRANSCRIPTION_FAILED = (
        "conversation.item.input_audio_transcription.failed"
    )
    CONVERSATION_ITEM_TRUNCATED = "conversation.item.truncated"
    CONVERSATION_ITEM_DELETED = "conversation.item.deleted"
    RESPONSE_CREATED = "response.created"
    RESPONSE_DONE = "response.done"
    RESPONSE_OUTPUT_ITEM_ADDED = "response.output_item.added"
    RESPONSE_OUTPUT_ITEM_DONE = "response.output_item.done"
    RESPONSE_CONTENT_PART_ADDED = "response.content_part.added"
    RESPONSE_CONTENT_PART_DONE = "response.content_part.done"
    RESPONSE_TEXT_DELTA = "response.text.delta"
    RESPONSE_TEXT_DONE = "response.text.done"
    RESPONSE_AUDIO_DONE = "response.audio.done"
    RESPONSE_FUNCTION_CALL_ARGUMENTS_DELTA = "response.function_call_arguments.delta"
    RESPONSE_FUNCTION_CALL_ARGUMENTS_DONE = "response.function_call_arguments.done"
    RATE_LIMITS_UPDATED = "rate_limits.updated"

#### Errors
class ErrorDetails(BaseModel):
    type: Optional[str] = None
    code: Optional[str] = None
    message: Optional[str] = None
    param: Optional[str] = None

class ErrorEvent(BaseEvent):
    type: Literal[ServerEventTypes.ERROR] = ServerEventTypes.ERROR
    error: ErrorDetails

#### Session
class SessionCreated(BaseEvent):
    type: Literal[ServerEventTypes.SESSION_CREATED] = ServerEventTypes.SESSION_CREATED
    session: Session

class SessionUpdated(BaseEvent):
    type: Literal[ServerEventTypes.SESSION_UPDATED] = ServerEventTypes.SESSION_UPDATED
    session: Session

#### Conversation
class Conversation(BaseModel):
    id: str
    object: Literal["realtime.conversation"]

class ConversationCreated(BaseEvent):
    type: Literal[ServerEventTypes.CONVERSATION_CREATED] = (
        ServerEventTypes.CONVERSATION_CREATED
    )
    conversation: Conversation

class ConversationItemCreated(BaseEvent):
    type: Literal[ServerEventTypes.CONVERSATION_ITEM_CREATED] = (
        ServerEventTypes.CONVERSATION_ITEM_CREATED
    )
    previous_item_id: Optional[str] = None
    item: ConversationItem

class ConversationItemInputAudioTranscriptionCompleted(BaseEvent):
    type: Literal[
        ServerEventTypes.CONVERSATION_ITEM_INPUT_AUDIO_TRANSCRIPTION_COMPLETED
    ] = ServerEventTypes.CONVERSATION_ITEM_INPUT_AUDIO_TRANSCRIPTION_COMPLETED
    item_id: str
    content_index: int
    transcript: str

class ConversationItemInputAudioTranscriptionFailed(BaseEvent):
    type: Literal[
        ServerEventTypes.CONVERSATION_ITEM_INPUT_AUDIO_TRANSCRIPTION_FAILED
    ] = ServerEventTypes.CONVERSATION_ITEM_INPUT_AUDIO_TRANSCRIPTION_FAILED
    item_id: str
    content_index: int
    error: dict[str, Any]

class ConversationItemTruncated(BaseEvent):
    type: Literal[ServerEventTypes.CONVERSATION_ITEM_TRUNCATED] = (
        ServerEventTypes.CONVERSATION_ITEM_TRUNCATED
    )
    item_id: str
    content_index: int
    audio_end_ms: int

class ConversationItemDeleted(BaseEvent):
    type: Literal[ServerEventTypes.CONVERSATION_ITEM_DELETED] = (
        ServerEventTypes.CONVERSATION_ITEM_DELETED
    )
    item_id: str

#### Response
class ResponseUsage(BaseModel):
    total_tokens: int
    input_tokens: int
    output_tokens: int
    input_token_details: Optional[dict[str, int]] = None
    output_token_details: Optional[dict[str, int]] = None

class ResponseOutput(BaseModel):
    id: str
    object: Literal["realtime.item"]
    type: str
    status: str
    role: str
    content: list[dict[str, Any]]

class ResponseContentPart(BaseModel):
    type: str
    text: Optional[str] = None

class ResponseOutputItemContent(BaseModel):
    type: str
    text: Optional[str] = None

class ResponseStatusDetails(BaseModel):
    type: str
    reason: str

class ResponseOutputItem(BaseModel):
    id: str
    object: Literal["realtime.item"]
    type: str
    status: str
    role: str
    content: list[ResponseOutputItemContent]

class Response(BaseModel):
    id: str
    object: Literal["realtime.response"]
    status: str
    status_details: Optional[ResponseStatusDetails] = None
    output: list[ResponseOutput]
    usage: Optional[ResponseUsage]

class ResponseCreated(BaseEvent):
    type: Literal[ServerEventTypes.RESPONSE_CREATED] = ServerEventTypes.RESPONSE_CREATED
    response: Response

class ResponseDone(BaseEvent):
    type: Literal[ServerEventTypes.RESPONSE_DONE] = ServerEventTypes.RESPONSE_DONE
    response: Response

class ResponseOutputItemAdded(BaseEvent):
    type: Literal[ServerEventTypes.RESPONSE_OUTPUT_ITEM_ADDED] = (
        ServerEventTypes.RESPONSE_OUTPUT_ITEM_ADDED
    )
    response_id: str
    output_index: int
    item: ResponseOutputItem

class ResponseOutputItemDone(BaseEvent):
    type: Literal[ServerEventTypes.RESPONSE_OUTPUT_ITEM_DONE] = (
        ServerEventTypes.RESPONSE_OUTPUT_ITEM_DONE
    )
    response_id: str
    output_index: int
    item: ResponseOutputItem

class ResponseContentPartAdded(BaseEvent):
    type: Literal[ServerEventTypes.RESPONSE_CONTENT_PART_ADDED] = (
        ServerEventTypes.RESPONSE_CONTENT_PART_ADDED
    )
    response_id: str
    item_id: str
    output_index: int
    content_index: int
    part: ResponseContentPart

class ResponseContentPartDone(BaseEvent):
    type: Literal[ServerEventTypes.RESPONSE_CONTENT_PART_DONE] = (
        ServerEventTypes.RESPONSE_CONTENT_PART_DONE
    )
    response_id: str
    item_id: str
    output_index: int
    content_index: int
    part: ResponseContentPart

#### Response Text
class ResponseTextDelta(BaseEvent):
    type: Literal[ServerEventTypes.RESPONSE_TEXT_DELTA] = (
        ServerEventTypes.RESPONSE_TEXT_DELTA
    )
    response_id: str
    item_id: str
    output_index: int
    content_index: int
    delta: str

class ResponseTextDone(BaseEvent):
    type: Literal[ServerEventTypes.RESPONSE_TEXT_DONE] = (
        ServerEventTypes.RESPONSE_TEXT_DONE
    )
    response_id: str
    item_id: str
    output_index: int
    content_index: int
    text: str

#### Response Audio
class ResponseAudioTranscriptDone(BaseEvent):
    type: Literal[ServerEventTypes.RESPONSE_AUDIO_TRANSCRIPT_DONE] = (
        ServerEventTypes.RESPONSE_AUDIO_TRANSCRIPT_DONE
    )
    transcript: str

class ResponseAudioTranscriptDelta(BaseEvent):
    type: Literal[ServerEventTypes.RESPONSE_AUDIO_TRANSCRIPT_DELTA] = (
        ServerEventTypes.RESPONSE_AUDIO_TRANSCRIPT_DELTA
    )
    delta: str

class ResponseAudioDelta(BaseEvent):
    type: Literal[ServerEventTypes.RESPONSE_AUDIO_DELTA] = (
        ServerEventTypes.RESPONSE_AUDIO_DELTA
    )
    response_id: str
    item_id: str
    delta: str

class ResponseAudioDone(BaseEvent):
    type: Literal[ServerEventTypes.RESPONSE_AUDIO_DONE] = (
        ServerEventTypes.RESPONSE_AUDIO_DONE
    )
    response_id: str
    item_id: str
    output_index: int
    content_index: int

class InputAudioBufferCommitted(BaseEvent):
    type: Literal[ServerEventTypes.INPUT_AUDIO_BUFFER_COMMITTED] = (
        ServerEventTypes.INPUT_AUDIO_BUFFER_COMMITTED
    )
    previous_item_id: Optional[str] = None
    item_id: Optional[str] = None
    event_id: Optional[str] = None

class InputAudioBufferCleared(BaseEvent):
    type: Literal[ServerEventTypes.INPUT_AUDIO_BUFFER_CLEARED] = (
        ServerEventTypes.INPUT_AUDIO_BUFFER_CLEARED
    )

class InputAudioBufferSpeechStarted(BaseEvent):
    type: Literal[ServerEventTypes.INPUT_AUDIO_BUFFER_SPEECH_STARTED] = (
        ServerEventTypes.INPUT_AUDIO_BUFFER_SPEECH_STARTED
    )
    audio_start_ms: int
    item_id: str

class InputAudioBufferSpeechStopped(BaseEvent):
    type: Literal[ServerEventTypes.INPUT_AUDIO_BUFFER_SPEECH_STOPPED] = (
        ServerEventTypes.INPUT_AUDIO_BUFFER_SPEECH_STOPPED
    )
    audio_end_ms: int
    item_id: str

#### Function Calls
class ResponseFunctionCallArgumentsDelta(BaseEvent):
    type: Literal[ServerEventTypes.RESPONSE_FUNCTION_CALL_ARGUMENTS_DELTA] = (
        ServerEventTypes.RESPONSE_FUNCTION_CALL_ARGUMENTS_DELTA
    )
    response_id: str
    item_id: str
    output_index: int
    call_id: str
    delta: str

class ResponseFunctionCallArgumentsDone(BaseEvent):
    type: Literal[ServerEventTypes.RESPONSE_FUNCTION_CALL_ARGUMENTS_DONE] = (
        ServerEventTypes.RESPONSE_FUNCTION_CALL_ARGUMENTS_DONE
    )
    response_id: str
    item_id: str
    output_index: int
    call_id: str
    arguments: str

#### Rate Limits
class RateLimit(BaseModel):
    name: str
    limit: int
    remaining: int
    reset_seconds: float

class RateLimitsUpdated(BaseEvent):
    type: Literal[ServerEventTypes.RATE_LIMITS_UPDATED] = (
        ServerEventTypes.RATE_LIMITS_UPDATED
    )
    rate_limits: list[RateLimit]

ServerEvent = Union[
    ErrorEvent,
    ConversationCreated,
    ResponseAudioTranscriptDone,
    ResponseAudioTranscriptDelta,
    ResponseAudioDelta,
    ResponseCreated,
    ResponseDone,
    ResponseOutputItemAdded,
    ResponseOutputItemDone,
    ResponseContentPartAdded,
    ResponseContentPartDone,
    ResponseTextDelta,
    ResponseTextDone,
    ResponseAudioDone,
    ConversationItemInputAudioTranscriptionCompleted,
    SessionCreated,
    SessionUpdated,
    InputAudioBufferCleared,
    InputAudioBufferSpeechStarted,
    InputAudioBufferSpeechStopped,
    ConversationItemCreated,
    ConversationItemInputAudioTranscriptionFailed,
    ConversationItemTruncated,
    ConversationItemDeleted,
    RateLimitsUpdated,
]

EVENT_TYPE_TO_MODEL = {
    ServerEventTypes.ERROR: ErrorEvent,
    ServerEventTypes.RESPONSE_AUDIO_TRANSCRIPT_DONE: ResponseAudioTranscriptDone,
    ServerEventTypes.RESPONSE_AUDIO_TRANSCRIPT_DELTA: ResponseAudioTranscriptDelta,
    ServerEventTypes.RESPONSE_AUDIO_DELTA: ResponseAudioDelta,
    ServerEventTypes.CONVERSATION_ITEM_INPUT_AUDIO_TRANSCRIPTION_COMPLETED: ConversationItemInputAudioTranscriptionCompleted,
    ServerEventTypes.SESSION_CREATED: SessionCreated,
    ServerEventTypes.SESSION_UPDATED: SessionUpdated,
    ServerEventTypes.CONVERSATION_CREATED: ConversationCreated,
    ServerEventTypes.INPUT_AUDIO_BUFFER_COMMITTED: InputAudioBufferCommitted,
    ServerEventTypes.INPUT_AUDIO_BUFFER_CLEARED: InputAudioBufferCleared,
    ServerEventTypes.INPUT_AUDIO_BUFFER_SPEECH_STARTED: InputAudioBufferSpeechStarted,
    ServerEventTypes.INPUT_AUDIO_BUFFER_SPEECH_STOPPED: InputAudioBufferSpeechStopped,
    ServerEventTypes.CONVERSATION_ITEM_CREATED: ConversationItemCreated,
    ServerEventTypes.CONVERSATION_ITEM_INPUT_AUDIO_TRANSCRIPTION_FAILED: ConversationItemInputAudioTranscriptionFailed,
    ServerEventTypes.CONVERSATION_ITEM_TRUNCATED: ConversationItemTruncated,
    ServerEventTypes.CONVERSATION_ITEM_DELETED: ConversationItemDeleted,
    ServerEventTypes.RESPONSE_CREATED: ResponseCreated,
    ServerEventTypes.RESPONSE_DONE: ResponseDone,
    ServerEventTypes.RESPONSE_OUTPUT_ITEM_ADDED: ResponseOutputItemAdded,
    ServerEventTypes.RESPONSE_OUTPUT_ITEM_DONE: ResponseOutputItemDone,
    ServerEventTypes.RESPONSE_CONTENT_PART_ADDED: ResponseContentPartAdded,
    ServerEventTypes.RESPONSE_CONTENT_PART_DONE: ResponseContentPartDone,
    ServerEventTypes.RESPONSE_TEXT_DELTA: ResponseTextDelta,
    ServerEventTypes.RESPONSE_TEXT_DONE: ResponseTextDone,
    ServerEventTypes.RESPONSE_AUDIO_DONE: ResponseAudioDone,
    ServerEventTypes.RATE_LIMITS_UPDATED: RateLimitsUpdated,
}

def parse_server_event(event_data: dict) -> ServerEvent:
    event_type = event_data.get("type")
    if not event_type:
        raise ValueError("Event data is missing 'type' field")

    model_class = EVENT_TYPE_TO_MODEL.get(event_type)
    if not model_class:
        raise ValueError(f"Unknown event type: {event_type}")

    try:
        return model_class(**event_data)
    except ValidationError as e:
        raise ValueError(f"Failed to parse event of type {event_type}: {str(e)}") from e
```

## Audio stream writer (to disk and in memory)

The following helper class buffers streamed audio chunks into a WAV file (or an in-memory buffer) so the audio can later be passed to Weave for logging.

```python lines theme={null}
class StreamingWavWriter:
    """Writes audio integer or byte array chunks to a WAV file."""

    wav_file = None
    buffer = None
    in_memory = False

    def __init__(
        self,
        filename=None,
        channels=INPUT_DEVICE_CHANNELS,
        sample_width=SAMPLE_WIDTH,
        framerate=SAMPLE_RATE,
    ):
        self.in_memory = filename is None
        if self.in_memory:
            self.buffer = io.BytesIO()
            self.wav_file = wave.open(self.buffer, "wb")
        else:
            self.wav_file = wave.open(filename, "wb")

        self.wav_file.setnchannels(channels)
        self.wav_file.setsampwidth(sample_width)
        self.wav_file.setframerate(framerate)

    def append_int16_chunk(self, int16_data):
        if int16_data is not None:
            self.wav_file.writeframes(
                int16_data.tobytes()
                if isinstance(int16_data, np.ndarray)
                else int16_data
            )

    def close(self):
        self.wav_file.close()

    def get_wav_buffer(self):
        assert self.in_memory, "Buffer only available if stream is in memory."
        return self.buffer
```

## Realtime audio model

The realtime (RT) audio model uses a websocket to send events to OpenAI's Realtime API. The model works as follows:

1. **init**: Initialize local buffers (input audio) and streams (assistant playback stream, user audio disk writer stream) and open a connection to the Realtime API.
2. **receive\_messages\_thread**: A thread handles receiving messages from the API. The code handles four primary event types:
   * `RESPONSE_AUDIO_TRANSCRIPT_DONE`: The server signals that the assistant's response is complete and provides the transcript.
   * `CONVERSATION_ITEM_INPUT_AUDIO_TRANSCRIPTION_COMPLETED`: The server signals that the user's audio is transcribed, and sends the transcript of the user's audio. The code logs the transcript to Weave and prints it for the user.
   * `RESPONSE_AUDIO_DELTA`: The server sends a new chunk of assistant response audio. The code appends this to the ongoing response data by the response ID, and adds this to the output stream for playback.
   * `RESPONSE_DONE`: The server signals completion of an assistant response. The code gets all audio chunks associated with the response, as well as the transcript, and logs them in Weave.
3. **send\_audio**: A handler appends user audio chunks to a buffer, and sends chunks of audio when the audio buffer reaches a certain size.

```python lines theme={null}
class RTAudioModel(weave.Model):
    """Model class for realtime e2e audio OpenAI model interaction with Whisper user transcription for logging."""

    realtime_model_name: str = "gpt-4o-realtime-preview-2024-10-01"  # realtime e2e audio only model interaction

    stop_event: Optional[threading.Event] = threading.Event()  # Event to stop the model
    ws: Optional[websocket.WebSocket] = None  # Websocket for OpenAI communications

    user_wav_writer: Optional[StreamingWavWriter] = (
        None  # Stream for writing user output to file
    )
    input_audio_buffer: Optional[np.ndarray] = None  # Buffer for user audio chunks
    assistant_outputs: dict[str, StreamingWavWriter] = (
        None  # Assistant outputs aggregated to send to weave
    )
    playback_stream: Optional[pyaudio.Stream] = (
        None  # Playback stream for playing assistant responses
    )

    def __init__(self):
        super().__init__()
        self.stop_event.clear()
        self.user_wav_writer = StreamingWavWriter(
            filename="user_audio.wav", framerate=SAMPLE_RATE
        )
        self.input_audio_buffer = np.array([], dtype=np.int16)
        self.ws = websocket.WebSocket()
        self.assistant_outputs = {}

        # Open the assistant audio playback stream if enabled
        if enable_audio_playback:
            self.playback_stream = pyaudio.PyAudio().open(
                format=pyaudio.paInt16,
                channels=OUTPUT_DEVICE_CHANNELS,
                rate=OAI_SAMPLE_RATE,
                output=True,
                output_device_index=OUTPUT_DEVICE_INDEX,
            )

        # Connect Websocket
        try:
            self.ws.connect(
                f"wss://api.openai.com/v1/realtime?model={self.realtime_model_name}",
                header={
                    "Authorization": f"Bearer {os.environ.get('OPENAI_API_KEY')}",
                    "OpenAI-Beta": "realtime=v1",
                },
            )

            # Send config msg
            config_event = SessionUpdate(
                session=Session(
                    modalities=["text", "audio"],  # modalities to use
                    input_audio_transcription=InputAudioTranscription(
                        model="whisper-1"
                    ),  # whisper-1 for transcription
                    turn_detection=TurnDetection(
                        type="server_vad",
                        threshold=0.3,
                        prefix_padding_ms=300,
                        silence_duration_ms=600,
                    ),  # server VAD to detect silence
                )
            )
            self.ws.send(config_event.model_dump_json(exclude_none=True))
            self.log_ws_message(config_event.model_dump_json(exclude_none=True), "Sent")

            # Start listener
            websocket_thread = threading.Thread(target=self.receive_messages_thread)
            websocket_thread.daemon = True
            websocket_thread.start()

        except Exception as e:
            print(f"Error connecting to WebSocket: {e}")

    ##### Weave Integration and Message Handlers #####
    def handle_assistant_response_audio_delta(self, data: ResponseAudioDelta):
        if data.response_id not in self.assistant_outputs:
            self.assistant_outputs[data.response_id] = StreamingWavWriter(
                framerate=OAI_SAMPLE_RATE
            )

        data_bytes = base64.b64decode(data.delta)
        self.assistant_outputs[data.response_id].append_int16_chunk(data_bytes)

        if enable_audio_playback:
            self.playback_stream.write(data_bytes)

        return {"assistant_audio": data_bytes}

    @weave.op()
    def handle_assistant_response_done(self, data: ResponseDone):
        wave_file_stream = self.assistant_outputs[data.response.id]
        wave_file_stream.close()
        wave_file_stream.buffer.seek(0)
        weave_payload = {
            "assistant_audio": wave.open(wave_file_stream.get_wav_buffer(), "rb"),
            "assistant_transcript": data.response.output[0]
            .content[0]
            .get("transcript", "Transcript Unavailable."),
        }
        return weave_payload

    @weave.op()
    def handle_user_transcription_done(
        self, data: ConversationItemInputAudioTranscriptionCompleted
    ):
        return {"user_transcript": data.transcript}

    ##### Message Receiver and Sender #####
    def receive_messages_thread(self):
        while not self.stop_event.is_set():
            try:
                data = json.loads(self.ws.recv())
                self.log_ws_message(json.dumps(data, indent=2))

                parsed_event = parse_server_event(data)

                if parsed_event.type == ServerEventTypes.RESPONSE_AUDIO_TRANSCRIPT_DONE:
                    print("Assistant: ", parsed_event.transcript)
                elif (
                    parsed_event.type
                    == ServerEventTypes.CONVERSATION_ITEM_INPUT_AUDIO_TRANSCRIPTION_COMPLETED
                ):
                    print("User: ", parsed_event.transcript)
                    self.handle_user_transcription_done(parsed_event)
                elif parsed_event.type == ServerEventTypes.RESPONSE_AUDIO_DELTA:
                    self.handle_assistant_response_audio_delta(parsed_event)
                elif parsed_event.type == ServerEventTypes.RESPONSE_DONE:
                    self.handle_assistant_response_done(parsed_event)
                elif parsed_event.type == ServerEventTypes.ERROR:
                    print(
                        f"\nError from server: {parsed_event.error.model_dump_json(exclude_none=True)}"
                    )
            except websocket.WebSocketConnectionClosedException:
                print("\nWebSocket connection closed")
                break
            except json.JSONDecodeError:
                continue
            except Exception as e:
                print(f"\nError in receive_messages: {e}")
                break

    def send_audio(self, audio_chunk):
        if self.ws and self.ws.connected:
            self.input_audio_buffer = np.append(
                self.input_audio_buffer, np.frombuffer(audio_chunk, dtype=np.int16)
            )
            if len(self.input_audio_buffer) >= SAMPLE_RATE * CHUNK_DURATION:
                try:
                    # Resample audio to OAI sample rate
                    resampled_audio = (
                        resampy.resample(
                            self.input_audio_buffer, SAMPLE_RATE, OAI_SAMPLE_RATE
                        )
                        if SAMPLE_RATE != OAI_SAMPLE_RATE
                        else self.input_audio_buffer
                    )

                    # Send audio chunk to OAI API
                    audio_event = InputAudioBufferAppend(
                        audio=base64.b64encode(
                            resampled_audio.astype(np.int16).tobytes()
                        ).decode("utf-8")  # Convert audio array to b64 bytes
                    )
                    self.ws.send(audio_event.model_dump_json(exclude_none=True))
                    self.log_ws_message(
                        audio_event.model_dump_json(exclude_none=True), "Sent"
                    )
                finally:
                    self.user_wav_writer.append_int16_chunk(self.input_audio_buffer)

                    # Clear the audio buffer
                    self.input_audio_buffer = np.array([], dtype=np.int16)
        else:
            print("Error sending audio: websocket not initialized.")

    ##### General Utility Functions #####
    def log_ws_message(self, message, direction="Received"):
        with open("websocket_log.txt", "a") as log_file:
            log_file.write(
                f"{time.strftime('%Y-%m-%d %H:%M:%S')} - {direction}: {message}\n"
            )

    def stop(self):
        self.stop_event.set()

        if self.ws:
            self.ws.close()

        self.user_wav_writer.close()
```

## Audio recorder

With the model defined, you need a way to feed it microphone input. This example uses a PyAudio input stream with a handler linked to the `send_audio` method of the RTAudio model. The code returns the stream to the main thread so it can exit safely upon program completion.

```python lines theme={null}
# Audio capture stream
def record_audio(realtime_model: RTAudioModel) -> pyaudio.Stream:
    """Setup a Pyaudio input stream and use the RTAudioModel as a callback for streaming data."""

    def audio_callback(in_data, frame_count, time_info, status):
        realtime_model.send_audio(in_data)
        return (None, pyaudio.paContinue)

    p = pyaudio.PyAudio()
    stream = p.open(
        format=pyaudio.paInt16,
        channels=INPUT_DEVICE_CHANNELS,
        rate=SAMPLE_RATE,
        input=True,
        input_device_index=INPUT_DEVICE_INDEX,
        frames_per_buffer=CHUNK,
        stream_callback=audio_callback,
    )
    stream.start_stream()

    print("Recording started. Please begin speaking to your personal assistant...")
    return stream
```

## Main thread

This final cell ties the previous components together and runs the realtime assistant. The main thread initiates a Realtime Audio Model with Weave integrated. Next, the code opens a recording and waits for a keyboard interrupt from the user. When you stop the cell, you have a complete Weave trace of the conversation, including user and assistant transcripts and audio.

```python lines theme={null}
weave.init(project_name="realtime-oai-audio-testing")

realtime_model = RTAudioModel()

if realtime_model.ws and realtime_model.ws.connected:
    recording_stream: pyaudio.Stream = record_audio(realtime_model)

    try:
        while not realtime_model.stop_event.is_set():
            time.sleep(1)
    except KeyboardInterrupt:
        pass
    except Exception as e:
        print(f"Error in main loop: {e}")
        import traceback

        traceback.print_exc()
    finally:
        print("Exiting...")
        realtime_model.stop()
        if recording_stream and recording_stream.is_active():
            recording_stream.stop_stream()
            recording_stream.close()
else:
    print(
        "WebSocket connection failed. Please check your API key and internet connection."
    )
```


# Chain Of Density
Source: https://docs.wandb.ai/weave/cookbooks/chain_of_density

Implement chain-of-density summarization techniques with W&B Weave for iterative text compression and evaluation.

<Note>
  This is an interactive notebook. You can run it locally or use the links below:

  * [Open in Google Colab](https://colab.research.google.com/github/wandb/docs/blob/main/weave/cookbooks/source/chain_of_density.ipynb)
  * [View source on GitHub](https://github.com/wandb/docs/blob/main/weave/cookbooks/source/chain_of_density.ipynb)
</Note>

Summarizing complex technical documents while preserving crucial details is a challenging task. The Chain of Density (CoD) summarization technique offers a solution by iteratively refining summaries to be more concise and information-dense. This guide demonstrates how to implement CoD using Weave for tracking and evaluating the application.

<Frame>
  <img alt="Weave evaluation dashboard with Chain of Density summarization results, metrics, and performance comparisons" />
</Frame>

## What is chain of density summarization

[![arXiv](https://img.shields.io/badge/arXiv-2309.04269-b31b1b.svg)](https://arxiv.org/abs/2309.04269)

Chain of Density (CoD) is an iterative summarization technique that produces increasingly concise and information-dense summaries. It works by:

1. Starting with an initial summary
2. Iteratively refining the summary, making it more concise while preserving key information
3. Increasing the density of entities and technical details with each iteration

This approach is particularly useful for summarizing scientific papers or technical documents where preserving detailed information is crucial.

## Why use Weave

In this tutorial, you'll use Weave to implement and evaluate a Chain of Density summarization pipeline for ArXiv papers. You'll learn how to:

* **Track your LLM pipeline**: Use Weave to automatically log inputs, outputs, and intermediate steps of your summarization process.
* **Evaluate LLM outputs**: Create consistent evaluations of your summaries using Weave's built-in tools.
* **Build composable operations**: Combine and reuse Weave operations across different parts of your summarization pipeline.
* **Integrate with existing code**: Add Weave to your existing Python code with minimal overhead.

By the end of this tutorial, you'll have created a CoD summarization pipeline that uses Weave's capabilities for model serving, evaluation, and result tracking.

## Set up the environment

First, set up the environment and import the necessary libraries. This step installs the dependencies needed for the pipeline, including Weave for tracking, Anthropic for the LLM, and PyPDF2 for reading ArXiv PDFs.

```python lines theme={null}
!pip install -qU anthropic weave pydantic requests PyPDF2 set-env-colab-kaggle-dotenv
```

Because the pipeline calls Anthropic's Claude model, you'll need an Anthropic API key before running the following code.

> To get an Anthropic API key:
>
> 1. Sign up for an account at [https://www.anthropic.com](https://www.anthropic.com).
> 2. Navigate to the API section in your account settings.
> 3. Generate a new API key.
> 4. Store the API key securely in your `.env` file.

```python lines theme={null}
import io
import os
from datetime import datetime, timezone

import anthropic
import requests
from pydantic import BaseModel
from PyPDF2 import PdfReader
from set_env import set_env

import weave

set_env("WANDB_API_KEY")
set_env("ANTHROPIC_API_KEY")

weave.init("summarization-chain-of-density-cookbook")
anthropic_client = anthropic.Anthropic(api_key=os.getenv("ANTHROPIC_API_KEY"))
```

This code uses Weave to track the experiment and Anthropic's Claude model for text generation. The `weave.init([PROJECT_NAME])` call sets up a new Weave project for the summarization task.

## Define the ArxivPaper model

With the environment ready, the next step is to define the data structure the pipeline operates on. Create an `ArxivPaper` class to represent the data:

```python lines theme={null}
# Define ArxivPaper model
class ArxivPaper(BaseModel):
    entry_id: str
    updated: datetime
    published: datetime
    title: str
    authors: list[str]
    summary: str
    pdf_url: str

# Create sample ArxivPaper
arxiv_paper = ArxivPaper(
    entry_id="http://arxiv.org/abs/2406.04744v1",
    updated=datetime(2024, 6, 7, 8, 43, 7, tzinfo=timezone.utc),
    published=datetime(2024, 6, 7, 8, 43, 7, tzinfo=timezone.utc),
    title="CRAG -- Comprehensive RAG Benchmark",
    authors=["Xiao Yang", "Kai Sun", "Hao Xin"],  # Truncated for brevity
    summary="Retrieval-Augmented Generation (RAG) has recently emerged as a promising solution...",  # Truncated
    pdf_url="https://arxiv.org/pdf/2406.04744",
)
```

This class encapsulates the metadata and content of an ArXiv paper, which is the input to the summarization pipeline.

## Load PDF content

The `ArxivPaper` model holds metadata and a PDF URL, but the summarization pipeline needs the full text of the paper. To work with the full paper content, add a function to load and extract text from PDFs:

```python lines theme={null}
@weave.op()
def load_pdf(pdf_url: str) -> str:
    # Download the PDF
    response = requests.get(pdf_url)
    pdf_file = io.BytesIO(response.content)

    # Read the PDF
    pdf_reader = PdfReader(pdf_file)

    # Extract text from all pages
    text = ""
    for page in pdf_reader.pages:
        text += page.extract_text()

    return text
```

## Implement Chain of Density summarization

Now, implement the core CoD summarization logic using Weave operations:

<Frame>
  <img alt="Weave trace visualization with Chain of Density summarization pipeline execution" />
</Frame>

```python lines theme={null}
# Chain of Density Summarization
@weave.op()
def summarize_current_summary(
    document: str,
    instruction: str,
    current_summary: str = "",
    iteration: int = 1,
    model: str = "claude-3-sonnet-20240229",
):
    prompt = f"""
    Document: {document}
    Current summary: {current_summary}
    Instruction to focus on: {instruction}
    Iteration: {iteration}

    Generate an increasingly concise, entity-dense, and highly technical summary from the provided document that specifically addresses the given instruction.
    """
    response = anthropic_client.messages.create(
        model=model, max_tokens=4096, messages=[{"role": "user", "content": prompt}]
    )
    return response.content[0].text

@weave.op()
def iterative_density_summarization(
    document: str,
    instruction: str,
    current_summary: str,
    density_iterations: int,
    model: str = "claude-3-sonnet-20240229",
):
    iteration_summaries = []
    for iteration in range(1, density_iterations + 1):
        current_summary = summarize_current_summary(
            document, instruction, current_summary, iteration, model
        )
        iteration_summaries.append(current_summary)
    return current_summary, iteration_summaries

@weave.op()
def final_summary(
    instruction: str, current_summary: str, model: str = "claude-3-sonnet-20240229"
):
    prompt = f"""
    Given this summary: {current_summary}
    And this instruction to focus on: {instruction}
    Create an extremely dense, final summary that captures all key technical information in the most concise form possible, while specifically addressing the given instruction.
    """
    return (
        anthropic_client.messages.create(
            model=model, max_tokens=4096, messages=[{"role": "user", "content": prompt}]
        )
        .content[0]
        .text
    )

@weave.op()
def chain_of_density_summarization(
    document: str,
    instruction: str,
    current_summary: str = "",
    model: str = "claude-3-sonnet-20240229",
    density_iterations: int = 2,
):
    current_summary, iteration_summaries = iterative_density_summarization(
        document, instruction, current_summary, density_iterations, model
    )
    final_summary_text = final_summary(instruction, current_summary, model)
    return {
        "final_summary": final_summary_text,
        "accumulated_summary": current_summary,
        "iteration_summaries": iteration_summaries,
    }
```

Here's what each function does:

* `summarize_current_summary`: Generates a single summary iteration based on the current state.
* `iterative_density_summarization`: Applies the CoD technique by calling `summarize_current_summary` multiple times.
* `chain_of_density_summarization`: Orchestrates the entire summarization process and returns the results.

The `@weave.op()` decorators ensure that Weave tracks the inputs, outputs, and execution of these functions.

## Create a Weave Model

With the summarization functions in place, the next step is to package them as a Weave Model so that runs, parameters, and versions are tracked together. Now, wrap the summarization pipeline in a Weave Model:

<Frame>
  <img alt="Weave Model configuration interface for Chain of Density summarization with model settings and parameters" />
</Frame>

```python lines theme={null}
# Weave Model
class ArxivChainOfDensityPipeline(weave.Model):
    model: str = "claude-3-sonnet-20240229"
    density_iterations: int = 3

    @weave.op()
    def predict(self, paper: ArxivPaper, instruction: str) -> dict:
        text = load_pdf(paper.pdf_url)
        result = chain_of_density_summarization(
            text,
            instruction,
            model=self.model,
            density_iterations=self.density_iterations,
        )
        return result
```

This `ArxivChainOfDensityPipeline` class encapsulates the summarization logic as a Weave Model, providing several key benefits:

* Automatic experiment tracking: Weave captures inputs, outputs, and parameters for each run of the model.
* Versioning: Changes to the model's attributes or code are automatically versioned, creating a clear history of how your summarization pipeline evolves over time.
* Reproducibility: The versioning and tracking let you reproduce any previous result or configuration of your summarization pipeline.
* Hyperparameter management: Model attributes (like `model` and `density_iterations`) are clearly defined and tracked across different runs, facilitating experimentation.
* Integration with Weave ecosystem: Using `weave.Model` works with other Weave tools, such as evaluations and serving capabilities.

## Implement evaluation metrics

With the pipeline producing summaries, you need a way to measure their quality systematically. To assess the quality of the summaries, implement simple evaluation metrics:

```python lines theme={null}
import json

@weave.op()
def evaluate_summary(
    summary: str, instruction: str, model: str = "claude-3-sonnet-20240229"
) -> dict:
    prompt = f"""
    Summary: {summary}
    Instruction: {instruction}

    Evaluate the summary based on the following criteria:
    1. Relevance (1-5): How well does the summary address the given instruction?
    2. Conciseness (1-5): How concise is the summary while retaining key information?
    3. Technical Accuracy (1-5): How accurately does the summary convey technical details?

    Your response MUST be in the following JSON format:
    {{
        "relevance": {{
            "score": <int>,
            "explanation": "<string>"
        }},
        "conciseness": {{
            "score": <int>,
            "explanation": "<string>"
        }},
        "technical_accuracy": {{
            "score": <int>,
            "explanation": "<string>"
        }}
    }}

    Ensure that the scores are integers between 1 and 5, and that the explanations are concise.
    """
    response = anthropic_client.messages.create(
        model=model, max_tokens=1000, messages=[{"role": "user", "content": prompt}]
    )
    print(response.content[0].text)

    eval_dict = json.loads(response.content[0].text)

    return {
        "relevance": eval_dict["relevance"]["score"],
        "conciseness": eval_dict["conciseness"]["score"],
        "technical_accuracy": eval_dict["technical_accuracy"]["score"],
        "average_score": sum(eval_dict[k]["score"] for k in eval_dict) / 3,
        "evaluation_text": response.content[0].text,
    }
```

These evaluation functions use the Claude model to assess the quality of the generated summaries based on relevance, conciseness, and technical accuracy.

## Create a Weave Dataset and run evaluation

Now that the scoring function is defined, the final step is to apply it to sample inputs and execute the evaluation. To evaluate the pipeline, create a Weave Dataset and run an evaluation:

<Frame>
  <img alt="Weave Dataset configuration interface for evaluation with dataset selection and configuration options" />
</Frame>

```python lines theme={null}
# Create a Weave Dataset
dataset = weave.Dataset(
    name="arxiv_papers",
    rows=[
        {
            "paper": arxiv_paper,
            "instruction": "What was the approach to experimenting with different data mixtures?",
        },
    ],
)

weave.publish(dataset)
```

For the evaluation, use an LLM-as-a-judge approach. This technique involves using a language model to assess the quality of outputs generated by another model or system. It uses the LLM's understanding and reasoning capabilities to provide nuanced evaluations, especially for tasks where traditional metrics may be insufficient.

[![arXiv](https://img.shields.io/badge/arXiv-2306.05685-b31b1b.svg)](https://arxiv.org/abs/2306.05685)

<Frame>
  <img alt="Weave evaluation dashboard with Chain of Density summarization results, metrics, and performance comparisons" />
</Frame>

```python lines theme={null}
# Define the scorer function
@weave.op()
def quality_scorer(instruction: str, output: dict) -> dict:
    result = evaluate_summary(output["final_summary"], instruction)
    return result
```

```python lines theme={null}
# Run evaluation
evaluation = weave.Evaluation(dataset=dataset, scorers=[quality_scorer])
arxiv_chain_of_density_pipeline = ArxivChainOfDensityPipeline()
results = await evaluation.evaluate(arxiv_chain_of_density_pipeline)
```

This code creates a dataset with the sample ArXiv paper, defines a quality scorer, and runs an evaluation of the summarization pipeline.

## Conclusion

This example demonstrated how to implement a Chain of Density summarization pipeline for ArXiv papers using Weave. You learned how to:

* Create Weave operations for each step of the summarization process
* Wrap the pipeline in a Weave Model for tracking and evaluation
* Implement custom evaluation metrics using Weave operations
* Create a dataset and run an evaluation of the pipeline

Weave tracks inputs, outputs, and intermediate steps throughout the summarization process, making it easier to debug, optimize, and evaluate your LLM application.

You can extend this example to handle larger datasets, implement more sophisticated evaluation metrics, or integrate with other LLM workflows.

<a href="https://wandb.ai/wandb_fc/arxiv-reader/reports/Building-a-bot-to-summarize-arXiv-papers-as-PDFs-using-Anthrophic-and-W-B-Weave--Vmlldzo4Nzg0ODI4">
  View Full Report on W\&B
</a>


# Codegen
Source: https://docs.wandb.ai/weave/cookbooks/codegen

Build and evaluate code generation pipelines with W&B Weave to trace prompts, outputs, and quality metrics.

<Note>
  This is an interactive notebook. You can run it locally or use the following links:

  * [Open in Google Colab](https://colab.research.google.com/github/wandb/docs/blob/main/weave/cookbooks/source/codegen.ipynb)
  * [View source on GitHub](https://github.com/wandb/docs/blob/main/weave/cookbooks/source/codegen.ipynb)
</Note>

Generating high-quality code with proper structure, documentation, and tests is a challenging task. This guide is for developers who want to build an LLM-powered code generation workflow and systematically measure its quality. This notebook demonstrates how to create a code-generation pipeline that produces Python functions evaluated against the HumanEval test suite. The pipeline uses Weave for evaluation comparison and tracking and OpenAI's GPT models for code generation with structured outputs.

<Frame>
  <img alt="Weave evaluation dashboard comparing code generation runs" />
</Frame>

## Why use Weave

This tutorial uses Weave to implement and evaluate a code generation pipeline. You learn how to:

* **Track your LLM pipeline**: Log inputs, outputs, and intermediate steps of your code generation process.
* **Evaluate LLM outputs**: Create and compare evaluations of your generated code with debugging tools and visualizations.

## Set up the environment

Set up your environment and import the necessary libraries. These dependencies provide the formatting tools, dataset loader, and OpenAI and Weave clients used throughout the pipeline.

```python lines theme={null}
!pip install -qU autopep8 autoflake weave isort openai set-env-colab-kaggle-dotenv datasets
python
%%capture
# Temporary workaround to fix bug in openai:
# TypeError: Client.__init__() got an unexpected keyword argument 'proxies'
# See https://community.openai.com/t/error-with-openai-1-56-0-client-init-got-an-unexpected-keyword-argument-proxies/1040332/15
!pip install "httpx<0.28"
python
import ast
import os
import re
import subprocess
import tempfile
import traceback

import autopep8
import isort
from autoflake import fix_code
from datasets import load_dataset
from openai import OpenAI
from pydantic import BaseModel
from set_env import set_env

import weave
from weave import Dataset, Evaluation

set_env("WANDB_API_KEY")
set_env("OPENAI_API_KEY")
python
WEAVE_PROJECT = "codegen-cookbook-example"
weave.init(WEAVE_PROJECT)
python
client = OpenAI()
python
human_eval = load_dataset("openai_humaneval")
selected_examples = human_eval["test"][:3]
```

<Note>
  Weave automatically tracks OpenAI API calls, including inputs, outputs, and metadata. You don't need to add any additional logging code for your OpenAI interactions. Weave handles it in the background.
</Note>

## Structured outputs and Pydantic models

This code generation pipeline uses OpenAI's [structured outputs mode](https://platform.openai.com/docs/guides/structured-outputs) and Pydantic models to produce consistent and well-formatted responses from the language model. This approach offers several advantages:

* **Type safety**: Defining Pydantic models for the expected outputs enforces a strict structure for the generated code, program runners, and unit tests.
* **Easier parsing**: The structured output mode parses the model's response directly into the predefined Pydantic models, reducing the need for complex post-processing.
* **Improved reliability**: Specifying the exact format you expect reduces the likelihood of unexpected or malformed outputs from the language model.

The following example defines Pydantic models and uses them with OpenAI's structured outputs:

```python lines theme={null}
class GeneratedCode(BaseModel):
    function_signature: str
    function_args_with_docstring_within_triple_quotes: str
    code_logic: str

class FormattedGeneratedCode(BaseModel):
    full_code: str
```

## Implement a code formatter

To produce consistent and clean code output, implement a `CodeFormatter` class using Weave operations. This formatter applies linting and styling rules to the generated code, program runner, and unit tests.

```python lines theme={null}
class CodeFormatter(BaseModel):
    @weave.op()
    def lint_code(self, code: str) -> str:
        # Replace escaped newlines with actual newlines
        code = code.replace("\\n", "\n")

        # Remove unused imports and variables
        code = fix_code(
            code, remove_all_unused_imports=True, remove_unused_variables=True
        )

        # Sort imports
        code = isort.code(code)

        # Apply PEP 8 formatting
        code = autopep8.fix_code(code, options={"aggressive": 2})

        return code

    @weave.op()
    def add_imports(self, code: str) -> str:
        tree = ast.parse(code)
        from_imports = {}
        global_names = set()

        for node in ast.walk(tree):
            if isinstance(node, ast.Name) and node.id not in dir(__builtins__):
                global_names.add(node.id)

        # Only add typing imports that are actually used
        typing_imports = global_names.intersection(
            {"List", "Dict", "Tuple", "Set", "Optional", "Union"}
        )
        if typing_imports:
            from_imports["typing"] = typing_imports

        # Remove names that are defined within the function
        function_def = next(
            node for node in tree.body if isinstance(node, ast.FunctionDef)
        )
        local_names = {arg.arg for arg in function_def.args.args}
        local_names.update(
            node.id
            for node in ast.walk(function_def)
            if isinstance(node, ast.Name) and isinstance(node.ctx, ast.Store)
        )

        global_names -= local_names
        global_names -= {"sorted"}  # Remove built-in functions

        # Construct the import statements
        import_statements = []
        for module, names in from_imports.items():
            names_str = ", ".join(sorted(names))
            import_statements.append(f"from {module} import {names_str}")

        return (
            "\n".join(import_statements) + ("\n\n" if import_statements else "") + code
        )

    @weave.op()
    def format_generated_code(
        self, generated_code: GeneratedCode
    ) -> FormattedGeneratedCode:
        # Combine the code parts
        full_code = f"{generated_code.function_signature}\n{generated_code.function_args_with_docstring_within_triple_quotes}\n{generated_code.code_logic}"

        # Ensure proper indentation
        lines = full_code.split("\n")
        indented_lines = []
        for i, line in enumerate(lines):
            if i == 0:  # Function signature
                indented_lines.append(line)
            elif i == 1:  # Function arguments (docstring)
                indented_lines.append("    " + line)
            else:  # Function body
                indented_lines.append("    " + line)
        full_code = "\n".join(indented_lines)

        # Lint the code
        full_code = self.lint_code(full_code)

        # Add imports
        cleaned_code = self.add_imports(full_code)

        return FormattedGeneratedCode(full_code=cleaned_code)
```

This `CodeFormatter` class provides several Weave operations to clean and format the generated code:

* Replacing escaped newlines with actual newlines.
* Removing unused imports and variables.
* Sorting imports.
* Applying PEP 8 formatting.
* Adding missing imports.

## Define the `CodeGenerationPipeline`

<Frame>
  <img alt="Weave trace of a code generation pipeline run" />
</Frame>

With the formatter in place, the next step is to implement the core code generation logic that ties the prompt, the LLM call, and the formatter together.

This example uses a `weave.Model` so that the model is automatically versioned when it changes. The `model_name` is kept as an attribute so that you can experiment with it and diff and compare it in Weave. Function calls are tracked with `@weave.op` so the inputs and outputs are logged to help with error tracking and debugging.

```python lines theme={null}
class CodeGenerationPipeline(weave.Model):
    model_name: str
    formatter: CodeFormatter

    def __init__(
        self, model_name: str = "gpt-4o", formatter: CodeFormatter | None = None
    ):
        if formatter is None:
            formatter = CodeFormatter()
        super().__init__(model_name=model_name, formatter=formatter)
        self.model_name = model_name
        self.formatter = formatter

    @weave.op()
    async def predict(self, prompt: str):
        generated_code = self.generate_code(prompt)
        formatted_generated_code = self.formatter.format_generated_code(generated_code)

        return formatted_generated_code.full_code

    @weave.op()
    def generate_code(self, prompt: str) -> GeneratedCode:
        completion = client.beta.chat.completions.parse(
            model=self.model_name,
            messages=[
                {
                    "role": "system",
                    "content": "You are an expert Python code generator.",
                },
                {"role": "user", "content": prompt},
            ],
            response_format=GeneratedCode,
        )
        message = completion.choices[0].message
        if message.parsed:
            return message.parsed
        else:
            raise ValueError(message.refusal)
```

This `CodeGenerationPipeline` class encapsulates the code generation logic as a Weave Model, providing several benefits:

* Automatic experiment tracking: Weave captures inputs, outputs, and parameters for each run of the model.
* Versioning: Changes to the model's attributes or code are automatically versioned, creating a history of how your code generation pipeline evolves over time.
* Reproducibility: The versioning and tracking let you reproduce any previous result or configuration of your code generation pipeline.
* Hyperparameter management: Model attributes (like `model_name`) are defined and tracked across different runs, facilitating experimentation.
* Integration with Weave ecosystem: Using `weave.Model` connects your pipeline to other Weave tools, such as evaluations and serving capabilities.

## Implement evaluation metrics

To assess the quality of the generated code, implement evaluation metrics using a `weave.Scorer` subclass. This runs `score` on every `model_output` from the dataset. `model_output` comes from the output of the `predict` function in the `weave.Model`. `prompt` is taken from the `human-eval` dataset.

```python lines theme={null}
CODE_TEMPLATE = """
{model_output}

{test}

if __name__ == "__main__":
    check({entry_point})
"""
python
@weave.op()
async def score_humaneval_test(test: str, entry_point: str, output: str):
    generated_code = output

    # Extract test cases from the test string
    test_cases = re.findall(r"assert.*", test)
    test_cases_str = "\n            ".join(test_cases)

    # Generate the full source code
    full_code = CODE_TEMPLATE.format(
        model_output=generated_code,
        test=test,
        test_cases=test_cases_str,
        entry_point=entry_point,
    )

    # Create a temporary file to store the code
    with tempfile.NamedTemporaryFile(delete=False, suffix=".py") as tmp_file:
        # Write the generated code to the temporary file
        tmp_file.write(full_code.encode())
        tmp_file_path = tmp_file.name

    try:
        # Run the temporary Python file as a subprocess with a timeout
        result = subprocess.run(
            ["python", tmp_file_path],
            capture_output=True,
            text=True,
            timeout=10,  # Timeout of 10 seconds
        )

        print(result)

        if result.returncode == 0:
            return {"correct": True}
        else:
            return {"correct": False, "error": result.stderr, "output": result.stdout}
    except subprocess.TimeoutExpired:
        return {"correct": False, "error": "TimeoutExpired"}
    except Exception as e:
        return {"correct": False, "error": traceback.format_exc()}
    finally:
        # Ensure the temporary file is removed after execution
        os.remove(tmp_file_path)
```

These evaluation functions run the generated code and return a boolean value indicating whether the code passed the test provided from the dataset.

<Frame>
  <img alt="Weave trace of a HumanEval scorer evaluating generated code" />
</Frame>

## Create a Weave Dataset and run evaluation

With the pipeline and scorer defined, the final step is to assemble the evaluation dataset and run it end-to-end. To evaluate the pipeline, create a Weave Dataset and run an evaluation:

```python lines theme={null}
formatted_selected_examples = [
    {
        "task_id": task_id,
        "prompt": prompt,
        "canonical_solution": solution,
        "test": test,
        "entry_point": entry_point,
    }
    for task_id, prompt, solution, test, entry_point in zip(
        selected_examples["task_id"],
        selected_examples["prompt"],
        selected_examples["canonical_solution"],
        selected_examples["test"],
        selected_examples["entry_point"],
    )
]
python
prompt_dataset = Dataset(
    name="humaneval_code_gen_example",
    rows=[
        {
            "prompt": example["prompt"],
            "test": example["test"],
            "entry_point": example["entry_point"],
        }
        for example in formatted_selected_examples
    ],
)
weave.publish(prompt_dataset)
python
EVAL_RUN = True
python
for model_name in ["gpt-4o-2024-08-06"]:
    pipeline = CodeGenerationPipeline(model_name=model_name)
    if not EVAL_RUN:
        dataset = prompt_dataset.rows[2]
        result = await pipeline.predict(dataset["prompt"])
        score_result = await score_humaneval_test(
            dataset["test"], dataset["entry_point"], result["generated_code"].full_code
        )
    else:
        evaluation = Evaluation(
            name="minimal_code_gen_evaluation",
            dataset=prompt_dataset,
            scorers=[score_humaneval_test],
        )
        results = await evaluation.evaluate(pipeline)
```

This code creates a dataset with the sample prompts, defines the HumanEval test scorer, and runs an evaluation of the code generation pipeline. After the evaluation completes, results are available in the Weave UI for inspection and comparison across runs.

<Frame>
  <img alt="Weave evaluation dashboard showing HumanEval scorer results" />
</Frame>

## Conclusion

This example demonstrates how to implement a code generation pipeline using Weave and OpenAI's language models. You learned how to:

* Create Weave operations for each step of the code generation process.
* Wrap the pipeline in a Weave Model for streamlined tracking and evaluation.
* Implement custom evaluation metrics using Weave operations.
* Create a dataset and run an evaluation of the pipeline.

Weave tracks inputs, outputs, and intermediate steps throughout the code generation process, making it easier to debug, optimize, and evaluate your LLM application.

For more information about Weave and its capabilities, see the [Weave documentation](/weave). You can extend this example to handle larger datasets, implement more sophisticated evaluation metrics, or integrate with other LLM workflows.


# Custom Model Cost
Source: https://docs.wandb.ai/weave/cookbooks/custom_model_cost

Define and track custom model costs in W&B Weave to monitor spending across different LLM providers and models.

<Note>
  This is an interactive notebook. You can run it locally or use the links below:

  * [Open in Google Colab](https://colab.research.google.com/github/wandb/docs/blob/main/weave/cookbooks/source/custom_model_cost.ipynb)
  * [View source on GitHub](https://github.com/wandb/docs/blob/main/weave/cookbooks/source/custom_model_cost.ipynb)
</Note>

## Set up a custom cost model

This guide shows you how to define a custom model in Weave that reports its own token usage, register a custom per-token cost for that model, and then retrieve calls with cost information attached. Use this when you're working with a model that isn't covered by Weave's built-in cost data and you want to track spending alongside your traces.

Weave calculates costs based on the number of tokens used and the model used.
Weave grabs this usage and model from the output and associates them with the call.

Set up a simple custom model that calculates its own token usage and stores that in Weave.

### Set up the environment

Install and import all needed packages.
Set `WANDB_API_KEY` in your environment so that you can log in with `wandb.login()` (this should be given to the colab as a secret).

Set the project in W\&B you want to log this into in `name_of_wandb_project`.

<Note>
  `name_of_wandb_project` can also be in the format of `{team_name}/{project_name}` to specify a team to log the traces into.
</Note>

Then fetch a Weave client by calling `weave.init()`.

```python lines theme={null}
%pip install wandb weave datetime --quiet
python
import os

import wandb
from google.colab import userdata

import weave

os.environ["WANDB_API_KEY"] = userdata.get("WANDB_API_KEY")
name_of_wandb_project = "custom-cost-model"

wandb.login()
python
weave_client = weave.init(name_of_wandb_project)
```

### Set up a model with Weave

Next, define a Weave `Model` subclass that performs its own token accounting. Returning the usage counts and model name in the output dictionary is what lets Weave later associate the call with the custom cost defined in the next section.

```python lines theme={null}
from weave import Model

class YourModel(Model):
    attribute1: str
    attribute2: int

    def simple_token_count(self, text: str) -> int:
        return len(text) // 3

    # This is a custom op that we are defining
    # It takes in a string, and outputs a dict with the usage counts, model name, and the output
    @weave.op()
    def custom_model_generate(self, input_data: str) -> dict:
        # Model logic goes here
        # Here is where you would have a custom generate function
        prediction = self.attribute1 + " " + input_data

        # Usage counts
        prompt_tokens = self.simple_token_count(input_data)
        completion_tokens = self.simple_token_count(prediction)

        # We return a dictionary with the usage counts, model name, and the output
        # Weave will automatically associate this with the trace
        # This object {usage, model, output} matches the output of a OpenAI Call
        return {
            "usage": {
                "input_tokens": prompt_tokens,
                "output_tokens": completion_tokens,
                "total_tokens": prompt_tokens + completion_tokens,
            },
            "model": "your_model_name",
            "output": prediction,
        }

    # In our predict function we call our custom generate function, and return the output.
    @weave.op()
    def predict(self, input_data: str) -> dict:
        # Here is where you would do any post processing of the data
        outputs = self.custom_model_generate(input_data)
        return outputs["output"]
```

### Add a custom cost

Add a custom cost. Once you have added a custom cost and your calls have usage, you can fetch the calls with `include_cost` and review the costs under `summary.weave.costs`.

```python lines theme={null}
model = YourModel(attribute1="Hello", attribute2=1)
model.predict("world")

# We then add a custom cost to our project
weave_client.add_cost(
    llm_id="your_model_name", prompt_token_cost=0.1, completion_token_cost=0.2
)

# We can then query for the calls, and with include_costs=True
# we receive the costs back attached to the calls
calls = weave_client.get_calls(filter={"trace_roots_only": True}, include_costs=True)

list(calls)
```

You now have a custom model that records its own token usage, a custom cost registered against that model's `llm_id`, and a way to retrieve calls with per-call cost data attached under `summary.weave.costs`.


# DSPy Prompt Optimization
Source: https://docs.wandb.ai/weave/cookbooks/dspy_prompt_optimization

Learn how to use dspy prompt optimization with W&B Weave

<Note>
  This is an interactive notebook. You can run it locally or use the following links:

  * [Open in Google Colab](https://colab.research.google.com/github/wandb/docs/blob/main/weave/cookbooks/source/dspy_prompt_optimization.ipynb)
  * [View source on GitHub](https://github.com/wandb/docs/blob/main/weave/cookbooks/source/dspy_prompt_optimization.ipynb)
</Note>

The [BIG-bench (Beyond the Imitation Game Benchmark)](https://github.com/google/BIG-bench) is a collaborative benchmark that probes large language models and extrapolates their future capabilities across more than 200 tasks. The [BIG-Bench Hard (BBH)](https://github.com/suzgunmirac/BIG-Bench-Hard) is a suite of the 23 most challenging BIG-Bench tasks that the current generation of language models can find difficult to solve.

This tutorial demonstrates how to improve the performance of an LLM workflow implemented on the causal judgement task from the BIG-bench Hard benchmark and evaluate prompting strategies. You use [DSPy](https://dspy.ai) to implement the LLM workflow and optimize the prompting strategy. You also use [Weave](/weave) to track the LLM workflow and evaluate prompting strategies.

By the end of this tutorial, you have built a baseline DSPy program for causal reasoning, evaluated it with Weave, applied a DSPy optimizer to improve its prompting strategy, and compared the optimized program against the baseline. This tutorial is intended for practitioners who want to apply prompt optimization to their own LLM workflows and use Weave to track and compare results.

## Install the dependencies

Before starting, install the libraries used throughout the tutorial. This tutorial uses the following libraries:

* [DSPy](https://dspy.ai) to build and optimize the LLM workflow.
* [Weave](/weave) to track the LLM workflow and evaluate prompting strategies.
* [datasets](https://huggingface.co/docs/datasets/index) to access the BIG-Bench Hard dataset from HuggingFace Hub.

```python lines theme={null}
!pip install -qU dspy weave "datasets<4"
```

Because this tutorial uses the [OpenAI API](https://openai.com/index/openai-api/) as the LLM vendor, you also need an OpenAI API key. You can [sign up](https://platform.openai.com/signup) on the OpenAI platform to get your own API key.

```python lines theme={null}
import os
from getpass import getpass

api_key = getpass("Enter you OpenAI API key: ")
os.environ["OPENAI_API_KEY"] = api_key
```

## Enable tracking using Weave

This section configures Weave so that subsequent DSPy calls in the tutorial are automatically traced and viewable in the Weave UI.

Weave integrates with DSPy. Including [`weave.init`](/weave/reference/python-sdk/trace/weave_client#method-init) at the start of your code automatically traces your DSPy functions, which you can then explore in the Weave UI. For more information, see the [Weave integration docs for DSPy](/weave/guides/integrations/dspy).

```python lines theme={null}
import weave

weave.init(project_name="dspy-bigbench-hard")
```

This tutorial uses a metadata class inherited from [`weave.Object`](/weave/reference/python-sdk#class-object) to manage metadata.

```python lines theme={null}
class Metadata(weave.Object):
    dataset_address: str = "maveriq/bigbenchhard"
    big_bench_hard_task: str = "causal_judgement"
    num_train_examples: int = 50
    openai_model: str = "gpt-4o-mini"
    openai_max_tokens: int = 2048
    max_bootstrapped_demos: int = 8
    max_labeled_demos: int = 8

metadata = Metadata()
```

<Tip>
  Object versioning: The `Metadata` objects are automatically versioned and traced when functions consuming them are traced.
</Tip>

## Load the BIG-Bench Hard dataset

With Weave tracking enabled, the next step is to prepare the data used to train and evaluate the DSPy program.

Load this dataset from HuggingFace Hub, split it into training and validation sets, and [publish](/weave/guides/core-types/datasets) them on Weave. Publishing lets you version the datasets and also use [`weave.Evaluation`](/weave/guides/core-types/evaluations) to evaluate your prompting strategy.

```python lines theme={null}
import dspy
from datasets import load_dataset

@weave.op()
def get_dataset(metadata: Metadata):
    # load the BIG-Bench Hard dataset corresponding to the task from Huggingface Hug
    dataset = load_dataset(metadata.dataset_address, metadata.big_bench_hard_task)[
        "train"
    ]

    # create the training and validation datasets
    rows = [{"question": data["input"], "answer": data["target"]} for data in dataset]
    train_rows = rows[0 : metadata.num_train_examples]
    val_rows = rows[metadata.num_train_examples :]

    # create the training and validation examples consisting of `dspy.Example` objects
    dspy_train_examples = [
        dspy.Example(row).with_inputs("question") for row in train_rows
    ]
    dspy_val_examples = [dspy.Example(row).with_inputs("question") for row in val_rows]

    # publish the datasets to the Weave, this would let us version the data and use for evaluation
    weave.publish(
        weave.Dataset(
            name=f"bigbenchhard_{metadata.big_bench_hard_task}_train", rows=train_rows
        )
    )
    weave.publish(
        weave.Dataset(
            name=f"bigbenchhard_{metadata.big_bench_hard_task}_val", rows=val_rows
        )
    )

    return dspy_train_examples, dspy_val_examples

dspy_train_examples, dspy_val_examples = get_dataset(metadata)
```

<Frame>
  <img alt="DSPy dataset loading interface with dataset preparation steps and data structure" />
</Frame>

## The DSPy program

With the dataset published to Weave, you can now define the baseline DSPy program that you later evaluate and optimize.

[DSPy](https://dspy.ai) is a framework that pushes building new LM pipelines away from manipulating free-form strings and closer to programming (composing modular operators to build text transformation graphs), where a compiler automatically generates optimized LM invocation strategies and prompts from a program.

Use [`dspy.LM`](https://dspy.ai/learn/programming/language_models) to configure the language model and [`dspy.configure`](https://dspy.ai/api/utils/configure/) to set it as the default.

```python lines theme={null}
llm = dspy.LM("openai/gpt-4o-mini")
dspy.configure(lm=llm)
```

### Write the causal reasoning signature

A [signature](https://dspy.ai/learn/programming/signatures) is a declarative specification of input/output behavior of a [DSPy module](https://dspy.ai/learn/programming/modules). DSPy modules are task-adaptive components (akin to neural network layers) that abstract any particular text transformation.

```python lines theme={null}
class CausalReasoning(dspy.Signature):
    """You are an expert in causal reasoning. Analyze the given question carefully
    and answer Yes or No. Provide a detailed explanation justifying your answer."""

    question: str = dspy.InputField(desc="The question to be answered")
    answer: str = dspy.OutputField(desc="Yes or No")
    confidence: float = dspy.OutputField(desc="Confidence score between 0 and 1")
    explanation: str = dspy.OutputField(desc="Detailed explanation for the answer")

class CausalReasoningModule(dspy.Module):
    def __init__(self):
        self.prog = dspy.Predict(CausalReasoning)

    @weave.op()
    def forward(self, question: str) -> dict:
        result = self.prog(question=question)
        return {
            "answer": result.answer,
            "confidence": result.confidence,
            "explanation": result.explanation,
        }
```

Test the LLM workflow (that is, the `CausalReasoningModule`) on an example from the causal reasoning subset of BIG-Bench Hard.

```python lines theme={null}
import rich

baseline_module = CausalReasoningModule()

prediction = baseline_module(dspy_train_examples[0]["question"])
rich.print(prediction)
```

<Frame>
  <img alt="Baseline DSPy program evaluation results with performance metrics and output examples" />
</Frame>

## Evaluate the DSPy program

Now that you have a baseline prompting strategy, evaluate it on the validation set using [`weave.Evaluation`](/weave/guides/core-types/evaluations) with a metric that matches the predicted answer with the ground truth. Weave takes each example, passes it through your application, and scores the output on multiple custom scoring functions. This gives you a view of the performance of your application, and a rich UI to drill into individual outputs and scores.

First, create a scoring function that determines whether the predicted answer matches the ground truth. Weave scoring functions receive the model's return value as `output` and any matching keys from the dataset example as additional arguments. Here, `answer` comes from the dataset and `output` is the dict returned by `CausalReasoningModule.forward`.

```python lines theme={null}
@weave.op()
def weave_evaluation_scorer(answer: str, output: dict) -> dict:
    return {"match": int(answer.lower() == output["answer"].lower())}
```

Next, wrap the module in a traced function that `weave.Evaluation` can call. The wrapper's argument names must match the dataset column names that the model consumes.

```python lines theme={null}
@weave.op()
def predict(question: str) -> dict:
    return baseline_module(question=question)
```

Now you can define the evaluation and run it.

```python lines theme={null}
validation_dataset = weave.ref(
    f"bigbenchhard_{metadata.big_bench_hard_task}_val:v0"
).get()

evaluation = weave.Evaluation(
    name="baseline_causal_reasoning_module",
    dataset=validation_dataset,
    scorers=[weave_evaluation_scorer],
)

await evaluation.evaluate(predict)
```

<Frame>
  <img alt="Weave evaluation dashboard with DSPy program performance metrics, traces, and comparison results" />
</Frame>

<Note>
  If you're running from a Python script, you can use the following code to run the evaluation:

  ```python lines theme={null}
  import asyncio
  asyncio.run(evaluation.evaluate(predict))
  ```
</Note>

<Warning>
  Running the evaluation causal reasoning dataset costs approximately \$0.24 in OpenAI credits.
</Warning>

## Optimize the DSPy program

With the baseline performance measured, you can now apply a DSPy optimizer and compare the result to the baseline.

Now that you have a baseline DSPy program, improve its performance for causal reasoning using the [BootstrapFewShot](https://dspy.ai/api/optimizers/BootstrapFewShot/) optimizer, which can tune the parameters of a DSPy program to maximize the specified metrics.

```python lines theme={null}
from dspy.teleprompt import BootstrapFewShot

@weave.op()
def get_optimized_program(model: dspy.Module, metadata: Metadata) -> dspy.Module:
    @weave.op()
    def dspy_evaluation_metric(true, prediction, trace=None):
        return prediction["answer"].lower() == true.answer.lower()

    teleprompter = BootstrapFewShot(
        metric=dspy_evaluation_metric,
        max_bootstrapped_demos=metadata.max_bootstrapped_demos,
        max_labeled_demos=metadata.max_labeled_demos,
    )
    return teleprompter.compile(model, trainset=dspy_train_examples)

optimized_module = get_optimized_program(baseline_module, metadata)
```

<Frame>
  <img alt="DSPy program optimization process interface with teleprompter configuration and optimization progress" />
</Frame>

<Warning>
  Running the evaluation causal reasoning dataset costs approximately \$0.04 in OpenAI credits.
</Warning>

Now that you have the optimized program (the optimized prompting strategy), evaluate it again on the validation set and compare it with the baseline DSPy program.

```python lines theme={null}
@weave.op()
def predict_optimized(question: str) -> dict:
    return optimized_module(question=question)

evaluation = weave.Evaluation(
    name="optimized_causal_reasoning_module",
    dataset=validation_dataset,
    scorers=[weave_evaluation_scorer],
)

await evaluation.evaluate(predict_optimized)
```

<Frame>
  <img alt="Optimized DSPy program evaluation results with improved performance metrics and output quality" />
</Frame>

Comparing the evaluation of the baseline program with the optimized one shows that the optimized program answers the causal reasoning questions with more accuracy.

## Conclusion

In this tutorial, you learned how to use DSPy for prompt optimization alongside Weave for tracking and evaluation to compare the original and optimized programs.


# Feedback Prod
Source: https://docs.wandb.ai/weave/cookbooks/feedback_prod

Collect and analyze user feedback on LLM outputs in production using W&B Weave's feedback tracking features.

<Note>
  This is an interactive notebook. You can run it locally or use the links below:

  * [Open in Google Colab](https://colab.research.google.com/github/wandb/docs/blob/main/weave/cookbooks/source/feedback_prod.ipynb)
  * [View source on GitHub](https://github.com/wandb/docs/blob/main/weave/cookbooks/source/feedback_prod.ipynb)
</Note>

Automatically evaluating a generated LLM response can be difficult. To better evaluate and improve responses, you can additionally gather direct user feedback to identify problem areas.

This notebook demonstrates how to collect user feedback about the responses from a custom chatbot. It uses Streamlit to build the interface and capture the LLM interactions and feedback in W\&B Weave. By the end, you have a running chatbot that logs each response to Weave, along with thumbs-up or thumbs-down reactions and free-text feedback from your users, so you can review production interactions and identify areas for improvement.

## Setup

First, install the required packages and set your OpenAI and W\&B API keys so that the chatbot can call the OpenAI API and log calls to Weave.

```python lines theme={null}
!pip install weave openai streamlit wandb
!pip install set-env-colab-kaggle-dotenv -q # for env var
python
# Add a .env file with your OpenAI and WandB API keys
from set_env import set_env

_ = set_env("OPENAI_API_KEY")
_ = set_env("WANDB_API_KEY")
```

Next, create a file called `chatbot.py` with the following contents. This file defines the Streamlit chat interface, wraps the OpenAI call with a Weave op so each response is tracked, and renders the feedback controls that attach reactions and notes back to the corresponding Weave call.

```python lines {13,15,27} theme={null}
# chatbot.py

import openai
import streamlit as st
import wandb
from set_env import set_env

import weave

_ = set_env("OPENAI_API_KEY")
_ = set_env("WANDB_API_KEY")

wandb.login()

weave_client = weave.init("feedback-example")
oai_client = openai.OpenAI()

def init_states():
    """Set up session_state keys if they don't exist yet."""
    if "messages" not in st.session_state:
        st.session_state["messages"] = []
    if "calls" not in st.session_state:
        st.session_state["calls"] = []
    if "session_id" not in st.session_state:
        st.session_state["session_id"] = "123abc"

@weave.op
def chat_response(full_history):
    """
    Calls the OpenAI API in streaming mode given the entire conversation history so far.
    full_history is a list of dicts: [{"role":"user"|"assistant","content":...}, ...]
    """
    stream = oai_client.chat.completions.create(
        model="gpt-4", messages=full_history, stream=True
    )
    response_text = st.write_stream(stream)
    return {"response": response_text}

def render_feedback_buttons(call_idx):
    """Renders thumbs up/down and text feedback for the call."""
    col1, col2, col3 = st.columns([1, 1, 4])

    # Thumbs up button
    with col1:
        if st.button("👍", key=f"thumbs_up_{call_idx}"):
            st.session_state.calls[call_idx].feedback.add_reaction("👍")
            st.success("Thanks for the feedback!")

    # Thumbs down button
    with col2:
        if st.button("👎", key=f"thumbs_down_{call_idx}"):
            st.session_state.calls[call_idx].feedback.add_reaction("👎")
            st.success("Thanks for the feedback!")

    # Text feedback
    with col3:
        feedback_text = st.text_input("Feedback", key=f"feedback_input_{call_idx}")
        if (
            st.button("Submit Feedback", key=f"submit_feedback_{call_idx}")
            and feedback_text
        ):
            st.session_state.calls[call_idx].feedback.add_note(feedback_text)
            st.success("Feedback submitted!")

def display_old_messages():
    """Displays the conversation stored in st.session_state.messages with feedback buttons"""
    for idx, message in enumerate(st.session_state.messages):
        with st.chat_message(message["role"]):
            st.markdown(message["content"])

            # If it's an assistant message, show feedback form
            if message["role"] == "assistant":
                # Figure out index of this assistant message in st.session_state.calls
                assistant_idx = (
                    len(
                        [
                            m
                            for m in st.session_state.messages[: idx + 1]
                            if m["role"] == "assistant"
                        ]
                    )
                    - 1
                )
                # Render thumbs up/down & text feedback
                if assistant_idx < len(st.session_state.calls):
                    render_feedback_buttons(assistant_idx)

def display_chat_prompt():
    """Displays the chat prompt input box."""
    if prompt := st.chat_input("Ask me anything!"):
        # Immediately render new user message
        with st.chat_message("user"):
            st.markdown(prompt)

        # Save user message in session
        st.session_state.messages.append({"role": "user", "content": prompt})

        # Prepare chat history for the API
        full_history = [
            {"role": msg["role"], "content": msg["content"]}
            for msg in st.session_state.messages
        ]

        with st.chat_message("assistant"):
            # Attach Weave attributes for tracking of conversation instances
            with weave.attributes(
                {"session": st.session_state["session_id"], "env": "prod"}
            ):
                # Call the OpenAI API (stream)
                result, call = chat_response.call(full_history)

                # Store the assistant message
                st.session_state.messages.append(
                    {"role": "assistant", "content": result["response"]}
                )

                # Store the weave call object to link feedback to the specific response
                st.session_state.calls.append(call)

                # Render feedback buttons for the new message
                new_assistant_idx = (
                    len(
                        [
                            m
                            for m in st.session_state.messages
                            if m["role"] == "assistant"
                        ]
                    )
                    - 1
                )

                # Render feedback buttons
                if new_assistant_idx < len(st.session_state.calls):
                    render_feedback_buttons(new_assistant_idx)

def main():
    st.title("Chatbot with immediate feedback forms")
    init_states()
    display_old_messages()
    display_chat_prompt()

if __name__ == "__main__":
    main()
```

You can run this with `streamlit run chatbot.py`.

You can now interact with this application and click the feedback buttons after each response. Visit the Weave UI to see the attached feedback. Each chat exchange is recorded as a Weave call, and any reactions or notes you submit are linked to the call that produced the response.

## Explanation

The following section walks through the key Weave APIs used in the chatbot, so you can apply the same pattern to your own application.

Consider the following decorated prediction function:

```python lines {5} theme={null}
import weave

weave.init("feedback-example")

@weave.op
def predict(input_data):
    # Your prediction logic here
    some_result = "hello world"
    return some_result
```

You can use it as usual to deliver a model response to the user:

```python lines theme={null}
with weave.attributes(
    {"session": "123abc", "env": "prod"}
):  # attach arbitrary attributes to the call alongside inputs & outputs
    result = predict(input_data="your data here")  # user question through the App UI
```

To attach feedback, you need the `call` object, which you obtain by using the `.call()` method instead of calling the function as normal:

```python lines theme={null}
result, call = predict.call(input_data="your data here")
```

You need this `call` object to attach feedback to the specific response. After you make the call, the output of the operation is available as `result`.

With the `call` object in hand, you can then record user feedback against that specific response:

```python lines theme={null}
call.feedback.add_reaction("👍")  # user reaction through the App UI
```

## Conclusion

In this tutorial, you built a chat UI with Streamlit that captures inputs and outputs in Weave, alongside thumbs-up and thumbs-down buttons to capture user feedback.


# HuggingFace Dataset Evaluations
Source: https://docs.wandb.ai/weave/cookbooks/hf_dataset_evals

Learn how to use HuggingFace dataset evaluations with W&B Weave

<Note>
  This is an interactive notebook. You can run it locally or use the links below:

  * [Open in Google Colab](https://colab.research.google.com/github/wandb/docs/blob/main/weave/cookbooks/source/hf_dataset_evals.ipynb)
  * [View source on GitHub](https://github.com/wandb/docs/blob/main/weave/cookbooks/source/hf_dataset_evals.ipynb)
</Note>

This notebook shows you how to use a HuggingFace dataset as the input source for a Weave `Evaluation`. By the end, you'll have a working evaluation pipeline that references rows in a HuggingFace dataset by index, preprocesses each row into the format your model expects, and tracks the scored results in Weave. This pattern is useful when you want to evaluate models against existing HuggingFace datasets without first converting them into a Weave-native dataset.

<Note>
  This guide demonstrates a workaround for using HuggingFace datasets with Weave evaluations. The approach works today, and more seamless integrations for external datasets are in development.
</Note>

## Setup and imports

First, initialize Weave and connect to W\&B so that Weave tracks the evaluation runs and their results in your Weave project.

```python lines theme={null}
!pip install datasets wandb weave

# Initialize variables
HUGGINGFACE_DATASET = "wandb/ragbench-test-sample"
WANDB_KEY = ""
WEAVE_TEAM = ""
WEAVE_PROJECT = ""

# Init weave and required libraries
import asyncio

import nest_asyncio
import wandb
from datasets import load_dataset

import weave
from weave import Evaluation

# Login to wandb and initialize weave
wandb.login(key=WANDB_KEY)
client = weave.init(f"{WEAVE_TEAM}/{WEAVE_PROJECT}")

# Apply nest_asyncio to allow nested event loops (needed for some notebook environments)
nest_asyncio.apply()
```

## Load and prepare HuggingFace dataset

Next, load the HuggingFace dataset and build a lightweight index that the evaluation iterates over. Rather than passing the dataset rows directly to Weave, pass a list of index references and resolve them to full rows during preprocessing. This approach keeps the evaluation tied to the original HuggingFace dataset and maintains references to it.

<Note>
  In the index, encode the `hf_hub_name` along with the `hf_id` to ensure each row has a unique identifier. Weave uses this unique digest value to track and reference specific dataset entries during evaluations.
</Note>

```python lines theme={null}
# Load the HuggingFace dataset
ds = load_dataset(HUGGINGFACE_DATASET)
row_count = ds["train"].num_rows

# Create an index mapping for the dataset
# This creates a list of dictionaries with HF dataset indices
# Example: [{"hf_id": 0}, {"hf_id": 1}, {"hf_id": 2}, ...]
hf_index = [{"hf_id": i, "hf_hub_name": HUGGINGFACE_DATASET} for i in range(row_count)]
```

## Define processing and evaluation functions

With the index in place, define the three functions that make up the evaluation pipeline: one to turn each index reference into a usable example, one to score the model's output, and one representing the model being evaluated.

The processing pipeline uses the following functions:

* `preprocess_example`: Transforms the index reference into the actual data needed for evaluation.
* `hf_eval`: Defines how to score the model outputs.
* `function_to_evaluate`: The actual function or model being evaluated.

```python lines theme={null}
@weave.op()
def preprocess_example(example):
    """
    Preprocesses each example before evaluation.
    Args:
        example: Dict containing hf_id
    Returns:
        Dict containing the prompt from the HF dataset
    """
    hf_row = ds["train"][example["hf_id"]]
    return {"prompt": hf_row["question"], "answer": hf_row["response"]}

@weave.op()
def hf_eval(hf_id: int, output: dict) -> dict:
    """
    Scoring function for evaluating model outputs.
    Args:
        hf_id: Index in the HF dataset
        output: The output from the model to evaluate
    Returns:
        Dict containing evaluation scores
    """
    hf_row = ds["train"][hf_id]
    return {"scorer_value": True}

@weave.op()
def function_to_evaluate(prompt: str):
    """
    The function that will be evaluated (e.g., your model or pipeline).
    Args:
        prompt: Input prompt from the dataset
    Returns:
        Dict containing model output
    """
    return {"generated_text": "testing "}
```

## Create and run the evaluation

Finally, wire the index, scorer, and preprocessing function into a Weave `Evaluation` and run it against the model. For each entry in `hf_index`, Weave performs the following steps:

1. `preprocess_example` gets the corresponding data from the HuggingFace dataset.
2. Weave passes the preprocessed data to `function_to_evaluate`.
3. `hf_eval` scores the output.
4. Weave tracks the results.

After the evaluation finishes, the run and its per-row scores are available in your Weave project for inspection.

```python lines theme={null}
# Create evaluation object
evaluation = Evaluation(
    dataset=hf_index,  # Use our index mapping
    scorers=[hf_eval],  # List of scoring functions
    preprocess_model_input=preprocess_example,  # Function to prepare inputs
)

# Run evaluation asynchronously
async def main():
    await evaluation.evaluate(function_to_evaluate)

asyncio.run(main())
```


# Import from CSV
Source: https://docs.wandb.ai/weave/cookbooks/import_from_csv

Import datasets from CSV files into W&B Weave for use in evaluations, tracing, and model comparison workflows.

<Note>
  This is an interactive notebook. You can run it locally or use the following links:

  * [Open in Google Colab](https://colab.research.google.com/github/wandb/docs/blob/main/weave/cookbooks/source/import_from_csv.ipynb)
  * [View source on GitHub](https://github.com/wandb/docs/blob/main/weave/cookbooks/source/import_from_csv.ipynb)
</Note>

## Import traces from third-party systems

This notebook shows you how to import historical conversation traces from a CSV file into W\&B Weave so you can analyze them, compare model behavior, and run evaluations on data that was generated outside of a Weave-instrumented application.

Sometimes you can't instrument your Python or JavaScript code with the Weave integration to obtain real-time traces of your GenAI application. Often, these traces are available to you later in CSV or JSON format.

This notebook uses the lower-level Weave Python API to extract data from a CSV file and import it into Weave so you can analyze and evaluate it.

The sample dataset assumed in this cookbook has the following structure:

```text theme={null}
conversation_id,turn_index,start_time,user_input,ground_truth,answer_text
1234,1,2024-09-04 13:05:39,This is the beginning, ['This was the beginning'], That was the beginning
1235,1,2024-09-04 13:02:11,This is another trace,, That was another trace
1235,2,2024-09-04 13:04:19,This is the next turn,, That was the next turn
1236,1,2024-09-04 13:02:10,This is a 3 turn conversation,, Woah thats a lot of turns
1236,2,2024-09-04 13:02:30,This is the second turn, ['That was definitely the second turn'], You are correct
1236,3,2024-09-04 13:02:53,This is the end,, Well good riddance!

```

To understand the decisions for import in this notebook, remember that Weave traces have parent-child relationships that are 1:Many and continuous. A single parent can have multiple children, and that parent can itself be a child of another parent.

This notebook uses `conversation_id` as the parent identifier and `turn_index` as the child identifier to provide complete conversation logging.

You must modify the variables in the following sections to match your own dataset, file paths, and W\&B project.

## Set up the environment

Install and import all needed packages.
Set `WANDB_API_KEY` in your environment so that you can log in with `wandb.login()` (provide this to Colab as a secret).

Set the name of the file you upload to Colab in `name_of_file`, and set the W\&B project you want to log into in `name_of_wandb_project`.

<Note>
  `name_of_wandb_project` can also be in the format `[TEAM_NAME]/[PROJECT_NAME]` to specify a team to log the traces into.
</Note>

Then, fetch a Weave client by calling `weave.init()`.

```python lines theme={null}
%pip install wandb weave pandas datetime --quiet
python
import os

import pandas as pd
import wandb
from google.colab import userdata

import weave

## Write samples file to disk
with open("/content/import_cookbook_data.csv", "w") as f:
    f.write(
        "conversation_id,turn_index,start_time,user_input,ground_truth,answer_text\n"
    )
    f.write(
        '1234,1,2024-09-04 13:05:39,This is the beginning, ["This was the beginning"], That was the beginning\n'
    )
    f.write(
        "1235,1,2024-09-04 13:02:11,This is another trace,, That was another trace\n"
    )
    f.write(
        "1235,2,2024-09-04 13:04:19,This is the next turn,, That was the next turn\n"
    )
    f.write(
        "1236,1,2024-09-04 13:02:10,This is a 3 turn conversation,, Woah thats a lot of turns\n"
    )
    f.write(
        '1236,2,2024-09-04 13:02:30,This is the second turn, ["That was definitely the second turn"], You are correct\n'
    )
    f.write("1236,3,2024-09-04 13:02:53,This is the end,, Well good riddance!\n")

os.environ["WANDB_API_KEY"] = userdata.get("WANDB_API_KEY")
name_of_file = "/content/import_cookbook_data.csv"
name_of_wandb_project = "import-weave-traces-cookbook"

wandb.login()
python
weave_client = weave.init(name_of_wandb_project)
```

## Load the data

With the environment ready, you can load and shape the CSV data so that it matches the parent-child structure that Weave expects.

Load the data into a pandas DataFrame, and sort it by `conversation_id` and `turn_index` to ensure the parents and children are correctly ordered.

This results in a two-column pandas DataFrame with the conversation turns as an array under `conversation_data`.

```python lines theme={null}
## Load data and shape it
df = pd.read_csv(name_of_file)

sorted_df = df.sort_values(["conversation_id", "turn_index"])

# Function to create an array of dictionaries for each conversation
def create_conversation_dict_array(group):
    return group.drop("conversation_id", axis=1).to_dict("records")

# Group the dataframe by conversation_id and apply the aggregation
result_df = (
    sorted_df.groupby("conversation_id")
    .apply(create_conversation_dict_array)
    .reset_index()
)
result_df.columns = ["conversation_id", "conversation_data"]

# Show how our aggregation looks
result_df.head()
```

## Log the traces to Weave

With the data shaped into conversations and turns, the next step is to write those records into Weave as parent and child calls.

Iterate through the pandas DataFrame:

* Create a parent call for every `conversation_id`.
* Iterate through the turn array to create child calls sorted by their `turn_index`.

Important concepts of the lower-level Python API:

* A Weave call is equivalent to a Weave trace. This call can have a parent or children associated with it.
* A Weave call can have other things associated with it, such as feedback and metadata. This example only associates inputs and outputs, but you can add these other items in your import if the data provides them.
* A Weave call is `created` and `finished` because these are meant to be tracked in real time. Because this is an after-the-fact import, you create and finish once the objects are defined and tied to one another.
* The `op` value of a call is how Weave categorizes calls of the same makeup. In this example, all parent calls are of `Conversation` type, and all child calls are of `Turn` type. You can modify this as you see fit.
* A call can have `inputs` and `output`. `inputs` are defined at creation, and `output` is defined when the call is finished.

```python lines theme={null}
# Log traces to weave

# Iterate through our aggregated conversations
for _, row in result_df.iterrows():
    # Define our conversation parent,
    # we are now creating a "call" with the weave_client we defined before
    parent_call = weave_client.create_call(
        # The Op value will register this as a Weave Op, which will allow us to retrieve these as a group easily in the future
        op="Conversation",
        # We set the inputs of our high level conversation as all the turns under it
        inputs={
            "conversation_data": row["conversation_data"][:-1]
            if len(row["conversation_data"]) > 1
            else row["conversation_data"]
        },
        # Our Conversation parent does not have a further parent
        parent=None,
        # The name of how this specific conversation will appear in the UI
        display_name=f"conversation-{row['conversation_id']}",
    )

    # We set the output of the parent to be the last trace in the conversation
    parent_output = row["conversation_data"][len(row["conversation_data"]) - 1]

    # We now iterate through all the conversation turns for the parent
    # and log them as children of the conversation
    for item in row["conversation_data"]:
        item_id = f"{row['conversation_id']}-{item['turn_index']}"

        # We create a call again here to be categorized under the conversation
        call = weave_client.create_call(
            # We qualify a single conversation trace as a "Turn"
            op="Turn",
            # We provide all inputs of the turn, including RAG 'ground_truth'
            inputs={
                "turn_index": item["turn_index"],
                "start_time": item["start_time"],
                "user_input": item["user_input"],
                "ground_truth": item["ground_truth"],
            },
            # We set this to be a child of the parent we defined
            parent=parent_call,
            # We provide it a name to be id'ed by in Weave
            display_name=item_id,
        )

        # We set the output of the call as the answer
        output = {
            "answer_text": item["answer_text"],
        }

        # Because these are traces that already happened, we finish the single turn call
        weave_client.finish_call(call=call, output=output)
    # Now that we have logged all its children, we also finish the parent call
    weave_client.finish_call(call=parent_call, output=parent_output)
```

## Result: traces logged to Weave

At this point, your CSV data has been imported into Weave. You can now browse the conversations and their turns in the Weave UI, grouped under the `Conversation` and `Turn` operations you defined.

Traces:

<Frame>
  <img alt="Imported conversation traces in the Weave UI" />
</Frame>

Operations:

<Frame>
  <img alt="Conversation and Turn operations in the Weave UI" />
</Frame>

## Optional: export your traces to run evaluations

Once the traces are in Weave and you understand how the conversations look, you can export them to another process to run Weave Evaluations.

<Frame>
  <img alt="Exporting traces from a Weave project" />
</Frame>

To do this, fetch all conversations from W\&B through the query API and create a dataset from them.

```python lines theme={null}
## This cell does not run by default, comment the below line to execute this script
%%script false --no-raise-error
## Get all Conversation traces for evaluation and prepare dataset for eval

# We create a query filter that brings us all our Conversation objects
# The ref shown below is specific to your project, and you can obtain it by
# going into your project's Operations in the UI, clicking on the "Conversations"
# object, then the "Use" tab in the side panel.
weave_ref_for_conversation_op = "weave://wandb-smle/import-weave-traces-cookbook/op/Conversation:tzUhDyzVm5bqQsuqh5RT4axEXSosyLIYZn9zbRyenaw"
filter = weave.trace_server.trace_server_interface.CallsFilter(
    op_names=[weave_ref_for_conversation_op],
  )

# We execute the query
conversation_traces = weave_client.get_calls(filter=filter)

rows = []

# We go through our conversation traces and construct dataset rows from it
for single_conv in conversation_traces:
  # In this example, we may only care for conversations that utilized our RAG
  # pipeline, so we filter for such types of conversations
  is_rag = False
  for single_trace in single_conv.inputs['conversation_data']:
    if single_trace['ground_truth'] is not None:
      is_rag = True
      break
  if single_conv.output['ground_truth'] is not None:
      is_rag = True

  # Once we've identified a converation to have used RAG, we add it to our dataset
  if is_rag:
    inputs = []
    ground_truths = []
    answers = []

    # We go through every turn in the conversation
    for turn in single_conv.inputs['conversation_data']:
      inputs.append(turn.get('user_input', ''))
      ground_truths.append(turn.get('ground_truth', ''))
      answers.append(turn.get('answer_text', ''))
    ## Account for when conversations are a single turn
    if len(single_conv.inputs) != 1 or single_conv.inputs['conversation_data'][0].get('turn_index') != single_conv.output.get('turn_index'):
      inputs.append(single_conv.output.get('user_input', ''))
      ground_truths.append(single_conv.output.get('ground_truth', ''))
      answers.append(single_conv.output.get('answer_text', ''))

    data = {
        'question': inputs,
        'contexts': ground_truths,
        'answer': answers
    }

    rows.append(data)

# With our dataset rows created, we create the Dataset object and
# publish it back to Weave for later retrieval
dset = weave.Dataset(name = "conv_traces_for_eval", rows=rows)
weave.publish(dset)
```

## Result

The exported dataset is now published back to Weave and is ready to use as input for an evaluation.

<Frame>
  <img alt="Published dataset in the Weave UI, ready for use in an evaluation" />
</Frame>

To learn more about evaluations, see the [Quickstart](/weave/tutorial-rag) on using your newly created dataset to evaluate your RAG application.


# Leaderboard Quickstart
Source: https://docs.wandb.ai/weave/cookbooks/leaderboard_quickstart

Learn how to use leaderboard quickstart with W&B Weave

<Note>
  This is an interactive notebook. You can run it locally or use the links below:

  * [Open in Google Colab](https://colab.research.google.com/github/wandb/docs/blob/main/weave/cookbooks/source/leaderboard_quickstart.ipynb)
  * [View source on GitHub](https://github.com/wandb/docs/blob/main/weave/cookbooks/source/leaderboard_quickstart.ipynb)
</Note>

# Leaderboard Quickstart

This quickstart shows you how to use the W\&B Weave Leaderboard to compare model performance across multiple datasets and scoring functions. By the end, you will have published a leaderboard that ranks several models against a shared set of evaluations. You can then identify which model performs best on each metric. This guide is for developers familiar with running Weave evaluations who want to compare results side by side.

Specifically, you will:

1. Generate a dataset of fake zip code data.
2. Author some scoring functions and evaluate a baseline model.
3. Use these techniques to evaluate a matrix of models versus evaluations.
4. Review the leaderboard in the Weave UI.

## Step 1: Generate a dataset of fake zip code data

First, create a function `generate_dataset_rows` that generates a list of fake zip code data. This synthetic dataset gives the leaderboard a consistent set of inputs and expected values to score each model against.

```python lines theme={null}
import json

from openai import OpenAI
from pydantic import BaseModel

class Row(BaseModel):
    zip_code: str
    city: str
    state: str
    avg_temp_f: float
    population: int
    median_income: int
    known_for: str

class Rows(BaseModel):
    rows: list[Row]

def generate_dataset_rows(
    location: str = "United States", count: int = 5, year: int = 2022
):
    client = OpenAI()

    completion = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {"role": "system", "content": "You are a helpful assistant."},
            {
                "role": "user",
                "content": f"Please generate {count} rows of data for random zip codes in {location} for the year {year}.",
            },
        ],
        response_format={
            "type": "json_schema",
            "json_schema": {
                "name": "response_format",
                "schema": Rows.model_json_schema(),
            },
        },
    )

    return json.loads(completion.choices[0].message.content)["rows"]
python
import weave

weave.init("leaderboard-demo")
```

## Step 2: Author scoring functions

Next, author three scoring functions. Each scorer evaluates a different aspect of the model output so the leaderboard can rank models on distinct dimensions of quality:

1. `check_concrete_fields`: Checks if the model output matches the expected city and state.
2. `check_value_fields`: Checks if the model output is within 10% of the expected population and median income.
3. `check_subjective_fields`: Uses an LLM to check if the model output matches the expected "known for" field.

```python lines theme={null}
@weave.op
def check_concrete_fields(city: str, state: str, output: dict):
    return {
        "city_match": city == output["city"],
        "state_match": state == output["state"],
    }

@weave.op
def check_value_fields(
    avg_temp_f: float, population: int, median_income: int, output: dict
):
    return {
        "avg_temp_f_err": abs(avg_temp_f - output["avg_temp_f"]) / avg_temp_f,
        "population_err": abs(population - output["population"]) / population,
        "median_income_err": abs(median_income - output["median_income"])
        / median_income,
    }

@weave.op
def check_subjective_fields(zip_code: str, known_for: str, output: dict):
    client = OpenAI()

    class Response(BaseModel):
        correct_known_for: bool

    completion = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {"role": "system", "content": "You are a helpful assistant."},
            {
                "role": "user",
                "content": f"My student was asked what the zip code {zip_code} is best known best for. The right answer is '{known_for}', and they said '{output['known_for']}'. Is their answer correct?",
            },
        ],
        response_format={
            "type": "json_schema",
            "json_schema": {
                "name": "response_format",
                "schema": Response.model_json_schema(),
            },
        },
    )

    return json.loads(completion.choices[0].message.content)
```

## Step 3: Create an evaluation

Next, define an evaluation using the fake data and scoring functions. The `Evaluation` object pairs the dataset with the scorers, so you can run any model against the same benchmark.

```python lines theme={null}
rows = generate_dataset_rows()
evaluation = weave.Evaluation(
    name="United States - 2022",
    dataset=rows,
    scorers=[
        check_concrete_fields,
        check_value_fields,
        check_subjective_fields,
    ],
)
```

## Step 4: Evaluate a baseline model

Now, evaluate a baseline model that returns a static response. Establishing a baseline gives you a reference point on the leaderboard so you can measure how much each subsequent model improves over a static implementation.

```python lines theme={null}
@weave.op
def baseline_model(zip_code: str):
    return {
        "city": "New York",
        "state": "NY",
        "avg_temp_f": 50.0,
        "population": 1000000,
        "median_income": 100000,
        "known_for": "The Big Apple",
    }

await evaluation.evaluate(baseline_model)
```

## Step 5: Create more models

Now, create two more models to compare against the baseline. One model receives the zip code with no additional prompting, while the other receives a structured prompt. Comparing them on the leaderboard shows the impact of prompt context on answer quality.

```python lines theme={null}
@weave.op
def gpt_4o_mini_no_context(zip_code: str):
    client = OpenAI()

    completion = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": f"""Zip code {zip_code}"""}],
        response_format={
            "type": "json_schema",
            "json_schema": {
                "name": "response_format",
                "schema": Row.model_json_schema(),
            },
        },
    )

    return json.loads(completion.choices[0].message.content)

await evaluation.evaluate(gpt_4o_mini_no_context)
python
@weave.op
def gpt_4o_mini_with_context(zip_code: str):
    client = OpenAI()

    completion = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {
                "role": "user",
                "content": f"""Please answer the following questions about the zip code {zip_code}:
                   1. What is the city?
                   2. What is the state?
                   3. What is the average temperature in Fahrenheit?
                   4. What is the population?
                   5. What is the median income?
                   6. What is the most well known thing about this zip code?
                   """,
            }
        ],
        response_format={
            "type": "json_schema",
            "json_schema": {
                "name": "response_format",
                "schema": Row.model_json_schema(),
            },
        },
    )

    return json.loads(completion.choices[0].message.content)

await evaluation.evaluate(gpt_4o_mini_with_context)
```

## Step 6: Create more evaluations

Now, evaluate a matrix of models versus evaluations. Running every model against several datasets (different regions and years) produces the data the leaderboard needs to rank models across multiple conditions.

```python lines theme={null}
scorers = [
    check_concrete_fields,
    check_value_fields,
    check_subjective_fields,
]
evaluations = [
    weave.Evaluation(
        name="United States - 2022",
        dataset=weave.Dataset(
            name="United States - 2022",
            rows=generate_dataset_rows("United States", 5, 2022),
        ),
        scorers=scorers,
    ),
    weave.Evaluation(
        name="California - 2022",
        dataset=weave.Dataset(
            name="California - 2022", rows=generate_dataset_rows("California", 5, 2022)
        ),
        scorers=scorers,
    ),
    weave.Evaluation(
        name="United States - 2000",
        dataset=weave.Dataset(
            name="United States - 2000",
            rows=generate_dataset_rows("United States", 5, 2000),
        ),
        scorers=scorers,
    ),
]
models = [
    baseline_model,
    gpt_4o_mini_no_context,
    gpt_4o_mini_with_context,
]

for evaluation in evaluations:
    for model in models:
        await evaluation.evaluate(
            model, __weave={"display_name": evaluation.name + ":" + model.__name__}
        )
```

## Step 7: Review the leaderboard

With the evaluation results published, you can now assemble them into a leaderboard for side-by-side comparison.

You can create a new leaderboard by navigating to the leaderboard tab in the UI and clicking **Create Leaderboard**.

You can also generate a leaderboard directly from Python:

```python lines theme={null}
from weave.flow import leaderboard
from weave.trace.ref_util import get_ref

spec = leaderboard.Leaderboard(
    name="Zip Code World Knowledge",
    description="""
This leaderboard compares the performance of models in terms of world knowledge about zip codes.

### Columns

1. **State Match against `United States - 2022`**: The fraction of zip codes that the model correctly identified the state for.
2. **Avg Temp F Error against `California - 2022`**: The mean absolute error of the model's average temperature prediction.
3. **Correct Known For against `United States - 2000`**: The fraction of zip codes that the model correctly identified the most well known thing about the zip code.
""",
    columns=[
        leaderboard.LeaderboardColumn(
            evaluation_object_ref=get_ref(evaluations[0]).uri(),
            scorer_name="check_concrete_fields",
            summary_metric_path="state_match.true_fraction",
        ),
        leaderboard.LeaderboardColumn(
            evaluation_object_ref=get_ref(evaluations[1]).uri(),
            scorer_name="check_value_fields",
            should_minimize=True,
            summary_metric_path="avg_temp_f_err.mean",
        ),
        leaderboard.LeaderboardColumn(
            evaluation_object_ref=get_ref(evaluations[2]).uri(),
            scorer_name="check_subjective_fields",
            summary_metric_path="correct_known_for.true_fraction",
        ),
    ],
)

ref = weave.publish(spec)
```

You now have a published leaderboard in Weave that ranks each model against the three evaluations and scoring metrics you defined. From the Weave UI, you can inspect per-model scores, drill into individual evaluation runs, and compare future model iterations against the same baseline.


# Multi-agent structured output
Source: https://docs.wandb.ai/weave/cookbooks/multi-agent-structured-output

Learn how to use multi-agent structured output with W&B Weave

<Note>
  This is an interactive notebook. You can run it locally or use the following links:

  * [Open in Google Colab](https://colab.research.google.com/github/wandb/docs/blob/main/weave/cookbooks/source/multi-agent-structured-output.ipynb)
  * [View source on GitHub](https://github.com/wandb/docs/blob/main/weave/cookbooks/source/multi-agent-structured-output.ipynb)
</Note>

This tutorial demonstrates how to build a multi-agent system that uses OpenAI's structured outputs feature, and how to trace agent interactions with [Weave](/weave). By the end, you have a four-agent data analysis pipeline whose intermediate inputs and outputs are visible in the Weave UI.

OpenAI released [Structured Outputs](https://openai.com/index/introducing-structured-outputs-in-the-api/) so the model always generates responses that adhere to your supplied JSON Schema without strongly worded prompts. With Structured Outputs, you don't need to validate or retry incorrectly formatted responses.

By using the parameter `strict: true`, you can guarantee the response abides by a provided schema.

Structured outputs in a multi-agent system produce consistent, predictably processed data between agents. They also support explicit refusals and remove the need for retries or response validation.

<Tip>
  **Source**: This cookbook is based on [sample code from OpenAI's structured outputs](https://cookbook.openai.com/examples/structured_outputs_multi_agent), with some modifications added for improved visualization using Weave.
</Tip>

## Install the dependencies

First, install the libraries the tutorial depends on. This tutorial uses the following libraries:

* [OpenAI](https://openai.com/index/openai-api/) to create the multi-agent system.
* [Weave](/weave) to track the LLM workflow and evaluate prompting strategies.

```python lines theme={null}
!pip install -qU openai weave wandb
python
%%capture
# Temporary workaround to fix bug in openai:
# TypeError: Client.__init__() got an unexpected keyword argument 'proxies'
# See https://community.openai.com/t/error-with-openai-1-56-0-client-init-got-an-unexpected-keyword-argument-proxies/1040332/15
!pip install "httpx<0.28"
```

Next, configure credentials and initialize Weave so traces are logged to your W\&B project.

Set `WANDB_API_KEY` in your environment so you can log in with `wandb.login()`. Provide this to Colab as a secret.

Set the W\&B project to log into in `name_of_wandb_project`.

<Note>
  `name_of_wandb_project` may also be in the format of `[YOUR-TEAM]/[YOUR-PROJECT]` to specify a team to log the traces into.
</Note>

Then fetch a Weave client by calling `weave.init()`.

Because this tutorial uses the [OpenAI API](https://openai.com/index/openai-api/), you also need an OpenAI API key. [Sign up](https://platform.openai.com/signup) on the OpenAI platform to get your own API key. Provide this to Colab as a secret too.

```python lines theme={null}
import base64
import json
import os
from io import BytesIO, StringIO

import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import wandb
from google.colab import userdata
from openai import OpenAI

import weave
python
os.environ["WANDB_API_KEY"] = userdata.get("WANDB_API_KEY")
os.environ["OPENAI_API_KEY"] = userdata.get("OPENAI_API_KEY")

wandb.login()
name_of_wandb_project = "multi-agent-structured-output"
weave.init(name_of_wandb_project)

client = OpenAI()
MODEL = "gpt-4o-2024-08-06"
```

## Set up the agents

With Weave initialized, define the agents that make up the system. The use case for this tutorial is a data analysis task.
First, set up the four-agent system:

* Triaging agent: Decides which agents to call.
* Data pre-processing agent: Prepares data for analysis, for example by cleaning it up.
* Data analysis agent: Performs analysis on the data.
* Data visualization agent: Visualizes the output of the analysis to extract insights.

Start by defining the system prompts for each of these agents. These prompts establish each agent's role and the tools it's allowed to call.

```python lines theme={null}
triaging_system_prompt = """You are a Triaging Agent. Your role is to assess the user's query and route it to the relevant agents. The agents available are:
- Data Processing Agent: Cleans, transforms, and aggregates data.
- Analysis Agent: Performs statistical, correlation, and regression analysis.
- Visualization Agent: Creates bar charts, line charts, and pie charts.

Use the send_query_to_agents tool to forward the user's query to the relevant agents. Also, use the speak_to_user tool to get more information from the user if needed."""

processing_system_prompt = """You are a Data Processing Agent. Your role is to clean, transform, and aggregate data using the following tools:
- clean_data
- transform_data
- aggregate_data"""

analysis_system_prompt = """You are an Analysis Agent. Your role is to perform statistical, correlation, and regression analysis using the following tools:
- stat_analysis
- correlation_analysis
- regression_analysis"""

visualization_system_prompt = """You are a Visualization Agent. Your role is to create bar charts, line charts, and pie charts using the following tools:
- create_bar_chart
- create_line_chart
- create_pie_chart"""
```

Next, define the tools for each agent.

Apart from the triaging agent, each agent has tools specific to its role:

**Data pre-processing agent**: Clean data, transform data, and aggregate data.

**Data analysis agent**: Statistical analysis, correlation analysis, and regression analysis.

**Data visualization agent**: Create bar chart, create line chart, and create pie chart.

```python lines theme={null}
triage_tools = [
    {
        "type": "function",
        "function": {
            "name": "send_query_to_agents",
            "description": "Sends the user query to relevant agents based on their capabilities.",
            "parameters": {
                "type": "object",
                "properties": {
                    "agents": {
                        "type": "array",
                        "items": {"type": "string"},
                        "description": "An array of agent names to send the query to.",
                    },
                    "query": {
                        "type": "string",
                        "description": "The user query to send.",
                    },
                },
                "required": ["agents", "query"],
            },
        },
        "strict": True,
    }
]

preprocess_tools = [
    {
        "type": "function",
        "function": {
            "name": "clean_data",
            "description": "Cleans the provided data by removing duplicates and handling missing values.",
            "parameters": {
                "type": "object",
                "properties": {
                    "data": {
                        "type": "string",
                        "description": "The dataset to clean. Should be in a suitable format such as JSON or CSV.",
                    }
                },
                "required": ["data"],
                "additionalProperties": False,
            },
        },
        "strict": True,
    },
    {
        "type": "function",
        "function": {
            "name": "transform_data",
            "description": "Transforms data based on specified rules.",
            "parameters": {
                "type": "object",
                "properties": {
                    "data": {
                        "type": "string",
                        "description": "The data to transform. Should be in a suitable format such as JSON or CSV.",
                    },
                    "rules": {
                        "type": "string",
                        "description": "Transformation rules to apply, specified in a structured format.",
                    },
                },
                "required": ["data", "rules"],
                "additionalProperties": False,
            },
        },
        "strict": True,
    },
    {
        "type": "function",
        "function": {
            "name": "aggregate_data",
            "description": "Aggregates data by specified columns and operations.",
            "parameters": {
                "type": "object",
                "properties": {
                    "data": {
                        "type": "string",
                        "description": "The data to aggregate. Should be in a suitable format such as JSON or CSV.",
                    },
                    "group_by": {
                        "type": "array",
                        "items": {"type": "string"},
                        "description": "Columns to group by.",
                    },
                    "operations": {
                        "type": "string",
                        "description": "Aggregation operations to perform, specified in a structured format.",
                    },
                },
                "required": ["data", "group_by", "operations"],
                "additionalProperties": False,
            },
        },
        "strict": True,
    },
]

analysis_tools = [
    {
        "type": "function",
        "function": {
            "name": "stat_analysis",
            "description": "Performs statistical analysis on the given dataset.",
            "parameters": {
                "type": "object",
                "properties": {
                    "data": {
                        "type": "string",
                        "description": "The dataset to analyze. Should be in a suitable format such as JSON or CSV.",
                    }
                },
                "required": ["data"],
                "additionalProperties": False,
            },
        },
        "strict": True,
    },
    {
        "type": "function",
        "function": {
            "name": "correlation_analysis",
            "description": "Calculates correlation coefficients between variables in the dataset.",
            "parameters": {
                "type": "object",
                "properties": {
                    "data": {
                        "type": "string",
                        "description": "The dataset to analyze. Should be in a suitable format such as JSON or CSV.",
                    },
                    "variables": {
                        "type": "array",
                        "items": {"type": "string"},
                        "description": "List of variables to calculate correlations for.",
                    },
                },
                "required": ["data", "variables"],
                "additionalProperties": False,
            },
        },
        "strict": True,
    },
    {
        "type": "function",
        "function": {
            "name": "regression_analysis",
            "description": "Performs regression analysis on the dataset.",
            "parameters": {
                "type": "object",
                "properties": {
                    "data": {
                        "type": "string",
                        "description": "The dataset to analyze. Should be in a suitable format such as JSON or CSV.",
                    },
                    "dependent_var": {
                        "type": "string",
                        "description": "The dependent variable for regression.",
                    },
                    "independent_vars": {
                        "type": "array",
                        "items": {"type": "string"},
                        "description": "List of independent variables.",
                    },
                },
                "required": ["data", "dependent_var", "independent_vars"],
                "additionalProperties": False,
            },
        },
        "strict": True,
    },
]

visualization_tools = [
    {
        "type": "function",
        "function": {
            "name": "create_bar_chart",
            "description": "Creates a bar chart from the provided data.",
            "parameters": {
                "type": "object",
                "properties": {
                    "data": {
                        "type": "string",
                        "description": "The data for the bar chart. Should be in a suitable format such as JSON or CSV.",
                    },
                    "x": {"type": "string", "description": "Column for the x-axis."},
                    "y": {"type": "string", "description": "Column for the y-axis."},
                },
                "required": ["data", "x", "y"],
                "additionalProperties": False,
            },
        },
        "strict": True,
    },
    {
        "type": "function",
        "function": {
            "name": "create_line_chart",
            "description": "Creates a line chart from the provided data.",
            "parameters": {
                "type": "object",
                "properties": {
                    "data": {
                        "type": "string",
                        "description": "The data for the line chart. Should be in a suitable format such as JSON or CSV.",
                    },
                    "x": {"type": "string", "description": "Column for the x-axis."},
                    "y": {"type": "string", "description": "Column for the y-axis."},
                },
                "required": ["data", "x", "y"],
                "additionalProperties": False,
            },
        },
        "strict": True,
    },
    {
        "type": "function",
        "function": {
            "name": "create_pie_chart",
            "description": "Creates a pie chart from the provided data.",
            "parameters": {
                "type": "object",
                "properties": {
                    "data": {
                        "type": "string",
                        "description": "The data for the pie chart. Should be in a suitable format such as JSON or CSV.",
                    },
                    "labels": {
                        "type": "string",
                        "description": "Column for the labels.",
                    },
                    "values": {
                        "type": "string",
                        "description": "Column for the values.",
                    },
                },
                "required": ["data", "labels", "values"],
                "additionalProperties": False,
            },
        },
        "strict": True,
    },
]
```

## Enable multi-agent tracking with Weave

With the agents and their tools defined, the next step is to wire them together and enable Weave tracing. Write the code logic to:

* Handle passing the user query to the multi-agent system.
* Handle the internal workings of the multi-agent system.
* Execute the tool calls.

```python lines theme={null}
# Example query

user_query = """
Below is some data. I want you to first remove the duplicates then analyze the statistics of the data as well as plot a line chart.

house_size (m3), house_price ($)
90, 100
80, 90
100, 120
90, 100
"""
```

From the user query, you can infer that the tools to call are `clean_data`, `start_analysis`, and `use_line_chart`.

Begin by defining the execution function responsible for running tool calls.

By decorating Python functions with `@weave.op()`, you can log and debug language model inputs, outputs, and traces.

A multi-agent system involves many functions, but it's sufficient to add `@weave.op()` on top of each one.

```python lines theme={null}
@weave.op()
def clean_data(data):
    data_io = StringIO(data)
    df = pd.read_csv(data_io, sep=",")
    df_deduplicated = df.drop_duplicates()
    return df_deduplicated

@weave.op()
def stat_analysis(data):
    data_io = StringIO(data)
    df = pd.read_csv(data_io, sep=",")
    return df.describe()

@weave.op()
def plot_line_chart(data):
    data_io = StringIO(data)
    df = pd.read_csv(data_io, sep=",")

    x = df.iloc[:, 0]
    y = df.iloc[:, 1]

    coefficients = np.polyfit(x, y, 1)
    polynomial = np.poly1d(coefficients)
    y_fit = polynomial(x)

    plt.figure(figsize=(10, 6))
    plt.plot(x, y, "o", label="Data Points")
    plt.plot(x, y_fit, "-", label="Best Fit Line")
    plt.title("Line Chart with Best Fit Line")
    plt.xlabel(df.columns[0])
    plt.ylabel(df.columns[1])
    plt.legend()
    plt.grid(True)

    # Save the plot to a BytesIO buffer before showing it
    buf = BytesIO()
    plt.savefig(buf, format="png")
    buf.seek(0)

    # Display the plot
    plt.show()

    # Encode the image in base64 for the data URL
    image_data = buf.getvalue()
    base64_encoded_data = base64.b64encode(image_data)
    base64_string = base64_encoded_data.decode("utf-8")
    data_url = f"data:image/png;base64,{base64_string}"

    return data_url

# Define the function to execute the tools
@weave.op()
def execute_tool(tool_calls, messages):
    for tool_call in tool_calls:
        tool_name = tool_call.function.name
        tool_arguments = json.loads(tool_call.function.arguments)

        if tool_name == "clean_data":
            # Simulate data cleaning
            cleaned_df = clean_data(tool_arguments["data"])
            cleaned_data = {"cleaned_data": cleaned_df.to_dict()}
            messages.append(
                {"role": "tool", "name": tool_name, "content": json.dumps(cleaned_data)}
            )
            print("Cleaned data: ", cleaned_df)
        elif tool_name == "transform_data":
            # Simulate data transformation
            transformed_data = {"transformed_data": "sample_transformed_data"}
            messages.append(
                {
                    "role": "tool",
                    "name": tool_name,
                    "content": json.dumps(transformed_data),
                }
            )
        elif tool_name == "aggregate_data":
            # Simulate data aggregation
            aggregated_data = {"aggregated_data": "sample_aggregated_data"}
            messages.append(
                {
                    "role": "tool",
                    "name": tool_name,
                    "content": json.dumps(aggregated_data),
                }
            )
        elif tool_name == "stat_analysis":
            # Simulate statistical analysis
            stats_df = stat_analysis(tool_arguments["data"])
            stats = {"stats": stats_df.to_dict()}
            messages.append(
                {"role": "tool", "name": tool_name, "content": json.dumps(stats)}
            )
            print("Statistical Analysis: ", stats_df)
        elif tool_name == "correlation_analysis":
            # Simulate correlation analysis
            correlations = {"correlations": "sample_correlations"}
            messages.append(
                {"role": "tool", "name": tool_name, "content": json.dumps(correlations)}
            )
        elif tool_name == "regression_analysis":
            # Simulate regression analysis
            regression_results = {"regression_results": "sample_regression_results"}
            messages.append(
                {
                    "role": "tool",
                    "name": tool_name,
                    "content": json.dumps(regression_results),
                }
            )
        elif tool_name == "create_bar_chart":
            # Simulate bar chart creation
            bar_chart = {"bar_chart": "sample_bar_chart"}
            messages.append(
                {"role": "tool", "name": tool_name, "content": json.dumps(bar_chart)}
            )
        elif tool_name == "create_line_chart":
            # Simulate line chart creation
            line_chart = {"line_chart": plot_line_chart(tool_arguments["data"])}
            messages.append(
                {"role": "tool", "name": tool_name, "content": json.dumps(line_chart)}
            )
        elif tool_name == "create_pie_chart":
            # Simulate pie chart creation
            pie_chart = {"pie_chart": "sample_pie_chart"}
            messages.append(
                {"role": "tool", "name": tool_name, "content": json.dumps(pie_chart)}
            )
    return messages
```

Next, create the tool handlers for each of the sub-agents. These have a unique prompt and tool set passed to the model. The output is then passed to an execution function that runs the tool calls.

```python lines theme={null}
# Define the functions to handle each agent's processing
@weave.op()
def handle_data_processing_agent(query, conversation_messages):
    messages = [{"role": "system", "content": processing_system_prompt}]
    messages.append({"role": "user", "content": query})

    response = client.chat.completions.create(
        model=MODEL,
        messages=messages,
        temperature=0,
        tools=preprocess_tools,
    )

    conversation_messages.append(
        [tool_call.function for tool_call in response.choices[0].message.tool_calls]
    )
    execute_tool(response.choices[0].message.tool_calls, conversation_messages)

@weave.op()
def handle_analysis_agent(query, conversation_messages):
    messages = [{"role": "system", "content": analysis_system_prompt}]
    messages.append({"role": "user", "content": query})

    response = client.chat.completions.create(
        model=MODEL,
        messages=messages,
        temperature=0,
        tools=analysis_tools,
    )

    conversation_messages.append(
        [tool_call.function for tool_call in response.choices[0].message.tool_calls]
    )
    execute_tool(response.choices[0].message.tool_calls, conversation_messages)

@weave.op()
def handle_visualization_agent(query, conversation_messages):
    messages = [{"role": "system", "content": visualization_system_prompt}]
    messages.append({"role": "user", "content": query})

    response = client.chat.completions.create(
        model=MODEL,
        messages=messages,
        temperature=0,
        tools=visualization_tools,
    )

    conversation_messages.append(
        [tool_call.function for tool_call in response.choices[0].message.tool_calls]
    )
    execute_tool(response.choices[0].message.tool_calls, conversation_messages)
```

Finally, create the overarching tool to handle processing the user query. This function takes the user query, gets a response from the model, and passes it to the other agents to execute.

```python lines theme={null}
# Function to handle user input and triaging
@weave.op()
def handle_user_message(user_query, conversation_messages=None):
    if conversation_messages is None:
        conversation_messages = []
    user_message = {"role": "user", "content": user_query}
    conversation_messages.append(user_message)

    messages = [{"role": "system", "content": triaging_system_prompt}]
    messages.extend(conversation_messages)

    response = client.chat.completions.create(
        model=MODEL,
        messages=messages,
        temperature=0,
        tools=triage_tools,
    )

    conversation_messages.append(
        [tool_call.function for tool_call in response.choices[0].message.tool_calls]
    )

    for tool_call in response.choices[0].message.tool_calls:
        if tool_call.function.name == "send_query_to_agents":
            agents = json.loads(tool_call.function.arguments)["agents"]
            query = json.loads(tool_call.function.arguments)["query"]
            for agent in agents:
                if agent == "Data Processing Agent":
                    handle_data_processing_agent(query, conversation_messages)
                elif agent == "Analysis Agent":
                    handle_analysis_agent(query, conversation_messages)
                elif agent == "Visualization Agent":
                    handle_visualization_agent(query, conversation_messages)

    outputs = extract_tool_contents(conversation_messages)

    return outputs

functions = [
    "clean_data",
    "transform_data",
    "stat_analysis",
    "aggregate_data",
    "correlation_analysis",
    "regression_analysis",
    "create_bar_chart",
    "create_line_chart",
    "create_pie_chart",
]

@weave.op()
def extract_tool_contents(data):
    contents = {}
    contents["all"] = data
    for element in data:
        if (
            isinstance(element, dict)
            and element.get("role") == "tool"
            and element.get("name") in functions
        ):
            name = element["name"]
            content_str = element["content"]
            try:
                content_json = json.loads(content_str)
                if "chart" not in element.get("name"):
                    contents[name] = [content_json]
                else:
                    first_key = next(iter(content_json))
                    second_level = content_json[first_key]
                    if isinstance(second_level, dict):
                        second_key = next(iter(second_level))
                        contents[name] = second_level[second_key]
                    else:
                        contents[name] = second_level
            except json.JSONDecodeError:
                print(f"Error decoding JSON for {name}")
                contents[name] = None

    return contents
```

## Execute the multi-agent system and visualize in Weave

With every agent, tool, and handler in place, the system is ready to run. Finally, execute the primary `handle_user_message` function using the user's input and observe the results.

```python lines theme={null}
handle_user_message(user_query)
```

When you click the Weave URL, you can see the execution trace. On the **Traces** page, you can check the input and output. For clarity, the diagram includes screenshots of the results displayed when you click each output. Weave integrates with the OpenAI API and automatically calculates costs. Cost and latency are also displayed alongside each trace.

<img alt="Weave Traces page showing the multi-agent execution with cost and latency" />

Click a line to see the intermediate processes that ran within the multi-agent system. For example, the input and output of the `analysis_agent` appear in a structured output format. OpenAI's structured outputs help agents collaborate, but as the system grows more complex, the format of these interactions becomes harder to follow. Weave lets you inspect these intermediate processes and their inputs and outputs in detail.

<Frame>
  <img alt="Weave trace detail showing the analysis agent's structured input and output" />
</Frame>

Take a closer look at how tracing is handled in Weave.

## Conclusion

In this tutorial, you learned how to develop a multi-agent system using OpenAI's structured output and Weave for tracking inputs, final outputs, and intermediate output formats. You now have a working example you can extend with additional agents, tools, or structured response schemas.


# Not Diamond custom routing
Source: https://docs.wandb.ai/weave/cookbooks/notdiamond_custom_routing

Learn how to use Not Diamond custom routing with W&B Weave

<Note>
  This is an interactive notebook. You can run it locally or use the following links:

  * [Open in Google Colab](https://colab.research.google.com/github/wandb/docs/blob/main/weave/cookbooks/source/notdiamond_custom_routing.ipynb)
  * [View source on GitHub](https://github.com/wandb/docs/blob/main/weave/cookbooks/source/notdiamond_custom_routing.ipynb)
</Note>

This notebook demonstrates how to use Weave with [Not Diamond's custom routing](https://docs.notdiamond.ai/docs/router-training-quickstart) to route LLM prompts to the most appropriate model based on evaluation results. By the end, you have trained a custom router for coding prompts, used it to select a target model for a new prompt, and evaluated its performance against the best individual model.

## Routing prompts

This section explains why custom routing is useful when working with multiple LLMs.

When building complex LLM workflows, you may need to prompt different models according to accuracy, cost, or call latency.
You can use [Not Diamond](https://www.notdiamond.ai/) to route prompts in these workflows to the right model for your needs, helping maximize accuracy while saving on model costs.

For any given distribution of data, rarely does a single model outperform every other model on every query. By combining multiple models into a "meta-model" that learns when to call each LLM, you can beat every individual model's performance and even drive down costs and latency.

## Custom routing

The following inputs are required to train a custom router for your prompts:

1. **A set of LLM prompts**: Prompts must be strings and should be representative of the prompts used in your application.
2. **LLM responses**: The responses from candidate LLMs for each input. Candidate LLMs can include both supported LLMs and your own custom models.
3. **Evaluation scores for responses to the inputs from candidate LLMs**: Scores are numbers, and can be any metric that fits your needs.

By submitting these to the Not Diamond API, you can then train a custom router tuned to each of your workflows.

## Set up the training data

In this section, you prepare the training and test data that the custom router learns from.

In practice, you use your own Evaluations to train a custom router. For this example notebook, however, you use LLM responses
for [the HumanEval dataset](https://github.com/openai/human-eval) to train a custom router for coding tasks.

Start by downloading the dataset prepared for this example, then parse LLM responses into `EvaluationResults` for each model. This split of training and test data is used in later sections to train the router and to evaluate its out-of-sample performance.

```python lines theme={null}
!curl -L "https://drive.google.com/uc?export=download&id=1q1zNZHioy9B7M-WRjsJPkfvFosfaHX38" -o humaneval.csv
python
import random

import weave
from weave.flow.dataset import Dataset
from weave.flow.eval import EvaluationResults
from weave.integrations.notdiamond.util import get_model_evals

pct_train = 0.8
pct_test = 1 - pct_train

# In practice, you will build an Evaluation on your dataset and call
# `evaluation.get_eval_results(model)`
model_evals = get_model_evals("./humaneval.csv")
model_train = {}
model_test = {}
for model, evaluation_results in model_evals.items():
    n_results = len(evaluation_results.rows)
    all_idxs = list(range(n_results))
    train_idxs = random.sample(all_idxs, k=int(n_results * pct_train))
    test_idxs = [idx for idx in all_idxs if idx not in train_idxs]

    model_train[model] = EvaluationResults(
        rows=weave.Table([evaluation_results.rows[idx] for idx in train_idxs])
    )
    model_test[model] = Dataset(
        rows=weave.Table([evaluation_results.rows[idx] for idx in test_idxs])
    )
    print(
        f"Found {len(train_idxs)} train rows and {len(test_idxs)} test rows for {model}."
    )
```

## Train a custom router

With the `EvaluationResults` prepared, you can now submit them to Not Diamond to train a custom router. Make sure you have [created an account](https://app.notdiamond.ai/keys) and
[generated an API key](https://app.notdiamond.ai/keys), then insert your API key in the following code. The API key authenticates the training request.

<Frame>
  <img alt="Create an API key" />
</Frame>

```python lines theme={null}
import os

from weave.integrations.notdiamond.custom_router import train_router

api_key = os.getenv("NOTDIAMOND_API_KEY", "<YOUR_API_KEY>")

preference_id = train_router(
    model_evals=model_train,
    prompt_column="prompt",
    response_column="actual",
    language="en",
    maximize=True,
    api_key=api_key,
    # Leave this commented out to train your first custom router
    # Uncomment this to retrain your custom router in place
    # preference_id=preference_id,
)
```

You can then follow the training process for your custom router through the Not Diamond app.

<Frame>
  <img alt="Check on router training progress" />
</Frame>

Once your custom router has finished training, you can use it to route your prompts.

```python lines theme={null}
from notdiamond import NotDiamond

import weave

weave.init("notdiamond-quickstart")

llm_configs = [
    "anthropic/claude-3-5-sonnet-20240620",
    "openai/gpt-4o-2024-05-13",
    "google/gemini-1.5-pro-latest",
    "openai/gpt-4-turbo-2024-04-09",
    "anthropic/claude-3-opus-20240229",
]
client = NotDiamond(api_key=api_key, llm_configs=llm_configs)

new_prompt = (
    """
You are a helpful coding assistant. Using the provided function signature, write the implementation for the function
in Python. Write only the function. Do not include any other text.

from typing import List

def has_close_elements(numbers: List[float], threshold: float) -> bool:
    """
    """ Check if in given list of numbers, are any two numbers closer to each other than
    given threshold.
    >>> has_close_elements([1.0, 2.0, 3.0], 0.5)
    False
    >>> has_close_elements([1.0, 2.8, 3.0, 4.0, 5.0, 2.0], 0.3)
    True
    """
    """
"""
)
session_id, routing_target_model = client.model_select(
    messages=[{"role": "user", "content": new_prompt}],
    preference_id=preference_id,
)

print(f"Session ID: {session_id}")
print(f"Target Model: {routing_target_model}")
```

This example also used Not Diamond's compatibility with Weave autotracing. You can see the results in the Weave UI.

<img alt="Weave UI for custom routing" />

## Evaluate your custom router

This section shows how to measure whether the custom router improves on the best individual model.

Once you've trained your custom router, you can evaluate its performance in one of the following ways:

* In-sample performance by submitting the training prompts.
* Out-of-sample performance by submitting new or held-out prompts.

In the following example, you submit the test set to the custom router to evaluate its performance.

```python lines theme={null}
from weave.integrations.notdiamond.custom_router import evaluate_router

eval_prompt_column = "prompt"
eval_response_column = "actual"

best_provider_model, nd_model = evaluate_router(
    model_datasets=model_test,
    prompt_column=eval_prompt_column,
    response_column=eval_response_column,
    api_key=api_key,
    preference_id=preference_id,
)
python
@weave.op()
def is_correct(score: int, output: dict) -> dict:
    # We hack score, since we already have model responses
    return {"correct": score}

best_provider_eval = weave.Evaluation(
    dataset=best_provider_model.model_results.to_dict(orient="records"),
    scorers=[is_correct],
)
await best_provider_eval.evaluate(best_provider_model)

nd_eval = weave.Evaluation(
    dataset=nd_model.model_results.to_dict(orient="records"), scorers=[is_correct]
)
await nd_eval.evaluate(nd_model)
```

In this instance, the Not Diamond "meta-model" routes prompts across several different models.

Training the custom router through Weave also runs evaluations and uploads results to the Weave UI. Once the custom router process completes, you can review the results in the Weave UI.

In the UI, you can see that the Not Diamond "meta-model" outperforms the best-performing model by routing prompts to other models with higher likelihood of answering the prompt accurately.

<Frame>
  <img alt="Evaluating Not Diamond" />
</Frame>


# Trace and Evaluate a Computer Vision Pipeline with Weave
Source: https://docs.wandb.ai/weave/cookbooks/ocr-pipeline

Learn how to use trace and evaluate a computer vision pipeline with weave with W&B Weave

<Note>
  This is an interactive notebook. You can run it locally or use the links below:

  * [Open in Google Colab](https://colab.research.google.com/github/wandb/docs/blob/main/weave/cookbooks/source/ocr-pipeline.ipynb)
  * [View source on GitHub](https://github.com/wandb/docs/blob/main/weave/cookbooks/source/ocr-pipeline.ipynb)
</Note>

This tutorial shows you how to build, trace, and evaluate a computer vision pipeline that performs named entity recognition (NER) on images of handwritten patient information. By the end, you'll have a working optical character recognition (OCR) pipeline backed by a vision-language model (VLM) and a W\&B Weave Evaluation that measures how accurately the pipeline extracts structured fields from images. This guide is for developers who want to use Weave to iterate on prompts and systematically measure the quality of multimodal extraction pipelines.

The following sections walk through five stages: creating and iterating on prompts, retrieving the dataset, building the NER pipeline, defining scorers, and running an evaluation.

## Prerequisites

Before you begin, install and import the required libraries, get your W\&B API key, and initialize your Weave project. Completing this step ensures that your environment can authenticate with W\&B and log traces to your Weave project.

```python lines theme={null}
# Install the required dependencies
!pip install openai weave -q
python
import json
import os

from google.colab import userdata
from openai import OpenAI

import weave
python
# Get API Keys
os.environ["OPENAI_API_KEY"] = userdata.get(
    "OPENAI_API_KEY"
)  # please set the keys as collab environment secrets from the menu on the left
os.environ["WANDB_API_KEY"] = userdata.get("WANDB_API_KEY")

# Set project name
# Replace the PROJECT value with your project name
PROJECT = "vlm-handwritten-ner"

# Initiatlize the Weave project
weave.init(PROJECT)
```

## Create and iterate on prompts with Weave

Good prompt engineering is critical to guiding the model to properly extract entities. In this section, you author an initial prompt, publish it to Weave so you can track changes over time, and then refine it with stricter validation rules.

First, create a basic prompt that gives the model the instructions on what to extract from the image data and how to format it. Then, store the prompt in Weave for tracking and iteration.

````python lines theme={null}
# Create your prompt object with Weave
prompt = """
Extract all readable text from this image. Format the extracted entities as a valid JSON.
Do not return any extra text, just the JSON. Do not include ```json```
Use the following format:
{"Patient Name": "James James","Date": "4/22/2025","Patient ID": "ZZZZZZZ123","Group Number": "3452542525"}
"""
system_prompt = weave.StringPrompt(prompt)
# Publish your prompt to Weave
weave.publish(system_prompt, name="NER-prompt")
````

Next, improve the prompt by adding more instructions and validation rules to help reduce errors in the output. Publishing the revised version under the same name lets Weave track the prompt as a new version so you can compare results across iterations.

````python lines theme={null}
better_prompt = """
You are a precision OCR assistant. Given an image of patient information, extract exactly these fields into a single JSON object (and nothing else):

- Patient Name
- Date (MM/DD/YYYY)
- Patient ID
- Group Number

Validation rules:
1. Date must match MM/DD/YY; if not, set Date to "".
2. Patient ID must be alphanumeric; if unreadable, set to "".
3. Always zero-pad months and days (e.g. "04/07/25").
4. Omit any markup, commentary, or code fences.
5. Return strictly valid JSON with only those four keys.

Do not return any extra text, just the JSON. Do not include ```json```
Example output:
{"Patient Name":"James James","Date":"04/22/25","Patient ID":"ZZZZZZZ123","Group Number":"3452542525"}
"""
# Edit the prompt
system_prompt = weave.StringPrompt(better_prompt)
# Publish the edited prompt to Weave
weave.publish(system_prompt, name="NER-prompt")
````

## Get the dataset

With a prompt in place, you next need input data to run through the pipeline. Retrieve the dataset of handwritten notes that serves as input for the OCR pipeline.

The images in the dataset are already `base64` encoded, which means that the LLM can use the data without any pre-processing.

```python lines theme={null}
# Retrieve the dataset from the following Weave project
dataset = weave.ref(
    "weave://wandb-smle/vlm-handwritten-ner/object/NER-eval-dataset:G8MEkqWBtvIxPYAY23sXLvqp8JKZ37Cj0PgcG19dGjw"
).get()

# Access a specific example in the dataset
example_image = dataset.rows[3]["image_base64"]

# Display the example_image
from IPython.display import HTML, display

html = f'<img src="{example_image}" style="max-width: 100%; height: auto;">'
display(HTML(html))
```

## Build the NER pipeline

Now that you have a prompt and a dataset, build the NER pipeline that connects them to the VLM. The pipeline consists of two functions:

* An `encode_image` function that takes a PIL image from the dataset and returns a `base64` encoded string representation of the image that can be passed to the VLM.
* An `extract_named_entities_from_image` function that takes an image and system prompt and returns the extracted entities from that image as described by the system prompt.

```python lines theme={null}
# Traceable function using GPT-4-Vision
def extract_named_entities_from_image(image_base64) -> dict:
    # init LLM Client
    client = OpenAI()

    # Setup the instruction prompt
    # You can optionally use a prompt stored in Weave withweave.ref("weave://wandb-smle/vlm-handwritten-ner/object/NER-prompt:FmCv4xS3RFU21wmNHsIYUFal3cxjtAkegz2ylM25iB8").get().content.strip()
    prompt = better_prompt

    response = client.responses.create(
        model="gpt-4.1",
        input=[
            {
                "role": "user",
                "content": [
                    {"type": "input_text", "text": prompt},
                    {
                        "type": "input_image",
                        "image_url": image_base64,
                    },
                ],
            }
        ],
    )

    return response.output_text
```

Now, create a function called `named_entity_recognation` that:

* Passes the image data to the NER pipeline.
* Returns correctly formatted JSON with the results.

Use the [`@weave.op()` decorator](/weave/reference/python-sdk/trace/op) to automatically track and trace function execution in the W\&B UI.

Every time `named_entity_recognation` runs, the full trace results are visible in the Weave UI. To view the traces, navigate to the **Traces** tab of your Weave project.

```python lines theme={null}
# NER Function for evaluations
@weave.op()
def named_entity_recognation(image_base64, id):
    result = {}
    try:
        # 1) call the vision op, get back a JSON string
        output_text = extract_named_entities_from_image(image_base64)

        # 2) parse JSON exactly once
        result = json.loads(output_text)

        print(f"Processed: {str(id)}")
    except Exception as e:
        print(f"Failed to process {str(id)}: {e}")
    return result
```

Finally, run the pipeline over the dataset, and view the results. This step produces the model outputs that you evaluate in the next section.

The following code loops over the dataset and stores the results in a local file `processing_results.json`. The results are also viewable in the Weave UI.

```python lines theme={null}
# Output results
results = []

# loop over all images in the dataset
for row in dataset.rows:
    result = named_entity_recognation(row["image_base64"], str(row["id"]))
    result["image_id"] = str(row["id"])
    results.append(result)

# Save all results to a JSON file
output_file = "processing_results.json"
with open(output_file, "w") as f:
    json.dump(results, f, indent=2)

print(f"Results saved to: {output_file}")
```

You see something similar to the following in the **Traces** table in the Weave UI.

<Frame>
  <img alt="Weave Traces table showing NER pipeline execution results." />
</Frame>

## Evaluate the pipeline using Weave

Now that you have created a pipeline to perform NER using a VLM, you can use Weave to systematically evaluate it and find out how well it performs. Evaluating the pipeline lets you measure extraction quality across the full dataset rather than relying on spot checks. For more information about Evaluations in Weave, see [Evaluations Overview](/weave/guides/core-types/evaluations).

A fundamental part of a Weave Evaluation is the [Scorer](/weave/guides/evaluation/scorers). Scorers evaluate AI outputs and return evaluation metrics. They take the AI's output, analyze it, and return a dictionary of results. Scorers can use your input data as reference if needed and can also output extra information, such as explanations or reasonings from the evaluation.

In this section, you create two scorers to evaluate the pipeline:

* Programatic scorer.
* LLM-as-a-judge scorer.

### Programatic scorer

The first Scorer is a deterministic check that runs without an LLM. The programmatic scorer, `check_for_missing_fields_programatically`, takes the model output (the output of the `named_entity_recognition` function), and identifies which `keys` are missing or empty in the results.

This check is useful for identifying samples where the model missed capturing any fields.

```python lines theme={null}
# Add weave.op() to track execution of the scorer
@weave.op()
def check_for_missing_fields_programatically(model_output):
    # Required keys for every entry
    required_fields = {"Patient Name", "Date", "Patient ID", "Group Number"}

    for key in required_fields:
        if (
            key not in model_output
            or model_output[key] is None
            or str(model_output[key]).strip() == ""
        ):
            return False  # This entry has a missing or empty field

    return True  # All required fields are present and non-empty
```

### LLM-as-a-judge scorer

Because the programmatic scorer only catches missing or empty fields, you need a second scorer to check whether the extracted values match what appears in the image. In this step of the evaluation, you provide both the image data and the model's output to ensure the assessment reflects actual NER performance. The image content is explicitly referenced, not just the model output.

The scorer used for this step, `check_for_missing_fields_with_llm`, uses an LLM to perform scoring (specifically OpenAI's `gpt-4o`). As specified by the contents of the `eval_prompt`, `check_for_missing_fields_with_llm` outputs a `Boolean` value. If all fields match the information in the image and formatting is correct, the scorer returns `true`. If any field is missing, empty, incorrect, or mismatched, the result is `false`, and the scorer also returns a message explaining the problem.

```python lines theme={null}
# The system prompt for the LLM-as-a-judge

eval_prompt = """
You are an OCR validation system. Your role is to assess whether the structured text extracted from an image accurately reflects the information in that image.
Only validate the structured text and use the image as your source of truth.

Expected input text format:
{"Patient Name": "First Last", "Date": "04/23/25", "Patient ID": "131313JJH", "Group Number": "35453453"}

Evaluation criteria:
- All four fields must be present.
- No field should be empty or contain placeholder/malformed values.
- The "Date" should be in MM/DD/YY format (e.g., "04/07/25") (zero padding the date is allowed)

Scoring:
- Return: {"Correct": true, "Reason": ""} if **all fields** match the information in the image and formatting is correct.
- Return: {"Correct": false, "Reason": "EXPLANATION"} if **any** field is missing, empty, incorrect, or mismatched.

Output requirements:
- Respond with a valid JSON object only.
- "Correct" must be a JSON boolean: true or false (not a string or number).
- "Reason" must be a short, specific string indicating all the problem — e.g., "Patient Name mismatch", "Date not zero-padded", or "Missing Group Number".
- Do not return any additional explanation or formatting.

Your response must be exactly one of the following:
{"Correct": true, "Reason": null}
OR
{"Correct": false, "Reason": "EXPLANATION_HERE"}
"""

# Add weave.op() to track execution of the Scorer
@weave.op()
def check_for_missing_fields_with_llm(model_output, image_base64):
    client = OpenAI()
    response = client.chat.completions.create(
        model="gpt-4o",
        messages=[
            {"role": "developer", "content": [{"text": eval_prompt, "type": "text"}]},
            {
                "role": "user",
                "content": [
                    {
                        "type": "image_url",
                        "image_url": {
                            "url": image_base64,
                        },
                    },
                    {"type": "text", "text": str(model_output)},
                ],
            },
        ],
        response_format={"type": "json_object"},
    )
    response = json.loads(response.choices[0].message.content)
    return response
```

## Run the evaluation

With both Scorers defined, you can now run the evaluation. Define an evaluation call that automatically loops over the `dataset` passed and logs the results together in the Weave UI.

The following code kicks off the evaluation and applies the two Scorers to every output from the NER pipeline. Results are visible in the **Evals** tab in the Weave UI.

```python lines theme={null}
evaluation = weave.Evaluation(
    dataset=dataset,
    scorers=[
        check_for_missing_fields_with_llm,
        check_for_missing_fields_programatically,
    ],
    name="Evaluate_4.1_NER",
)

print(await evaluation.evaluate(named_entity_recognation))
```

When the previous code runs, Weave generates a link to the Evaluation table in the Weave UI. Follow the link to view the results and compare different iterations of the pipeline across models, prompts, and datasets of your choice. The Weave UI automatically creates a visualization like the following for your team.

<Frame>
  <img alt="Weave Evaluation results comparing scorer outputs across the dataset." />
</Frame>


# Online Monitoring
Source: https://docs.wandb.ai/weave/cookbooks/online_monitoring

Set up online monitoring for LLM applications in production using W&B Weave to track performance and quality metrics.

<Note>
  This is an interactive notebook. You can run it locally or use the links below:

  * [Open in Google Colab](https://colab.research.google.com/github/wandb/docs/blob/main/weave/cookbooks/source/online_monitoring.ipynb)
  * [View source on GitHub](https://github.com/wandb/docs/blob/main/weave/cookbooks/source/online_monitoring.ipynb)
</Note>

## Integrate with Weave: production dashboard

This notebook demonstrates how to use Weave's APIs and functions to create a custom dashboard for production monitoring as an extension to the Traces view in Weave. This guide is intended for developers and ML engineers who run LLM applications in production and want tailored visibility into performance, cost, and user feedback beyond what the default Traces view provides. This notebook focuses on:

* Fetching traces, costs, feedback, and other metrics from Weave.
* Creating aggregate views for user feedback and cost distribution.
* Creating visualizations for token usage and latency over time.

By the end of this notebook, you will have a working custom dashboard that aggregates trace data, costs, and feedback from your Weave project into a single view. You can try the dashboard with your own Weave project by installing `streamlit` and running the [production dashboard script](https://github.com/NiWaRe/agent-dev-collection).

<img alt="Example Production Dashboard with Weave" />

## Setup

To begin, install the following packages:

```python lines theme={null}
!pip install streamlit pandas plotly weave
```

## Implementation

The following sections walk through initializing the Weave client, fetching call data, and generating visualizations for the dashboard.

### Initialize the Weave client and define costs

First, set up a function to initialize the Weave client and add costs for each model. This step is required so that downstream cost queries can attribute per-token pricing to each call.

W\&B includes the standard costs for many standard models, and also lets you add your own custom costs and custom models. The following example shows how to add custom costs for a few models and use the standard costs for the rest.
The costs are calculated based on the tracked tokens for each call in Weave. For many LLM vendor libraries, Weave automatically tracks the token usage, but you can also return custom token counts for any call. For more information about defining the token count and cost calculation for a custom model, see the [custom cost cookbook](/weave/cookbooks/custom_model_cost#setting-up-a-model-with-weave).

```python lines theme={null}
PROJECT_NAME = "wandb-smle/weave-cookboook-demo"
python
import weave

MODEL_NAMES = [
    # model name, prompt cost, completion cost
    ("gpt-4o-2024-05-13", 0.03, 0.06),
    ("gpt-4o-mini-2024-07-18", 0.03, 0.06),
    ("gemini/gemini-1.5-flash", 0.00025, 0.0005),
    ("gpt-4o-mini", 0.03, 0.06),
    ("gpt-4-turbo", 0.03, 0.06),
    ("claude-3-haiku-20240307", 0.01, 0.03),
    ("gpt-4o", 0.03, 0.06),
]

def init_weave_client(project_name):
    try:
        client = weave.init(project_name)
        for model, prompt_cost, completion_cost in MODEL_NAMES:
            client.add_cost(
                llm_id=model,
                prompt_token_cost=prompt_cost,
                completion_token_cost=completion_cost,
            )
    except Exception as e:
        print(f"Failed to initialize Weave client for project '{project_name}': {e}")
        return None
    else:
        return client

client = init_weave_client(PROJECT_NAME)
```

### Fetch calls data from Weave

With the client initialized and costs configured, the next step is to pull call data from Weave. There are two options for fetching call data:

* Fetching data call-by-call.
* Using high-level APIs.

The following sections describe each option.

#### Fetch data call-by-call

The first option to access data from Weave is to retrieve a list of filtered calls and extract the wanted data call-by-call. To do this, use the `calls_query_stream` API to fetch the calls data from Weave:

* `calls_query_stream` API: This API fetches the calls data from Weave.
* `filter` dictionary: This dictionary contains the filter parameters to fetch the calls data. See the [CallSchema reference](/weave/reference/python-sdk/trace_server/trace_server_interface#class-callschema) for more details.
* `expand_columns` list: This list contains the columns to expand in the calls data.
* `sort_by` list: This list contains the sorting parameters for the calls data.
* `include_costs` boolean: This boolean indicates whether to include the costs in the calls data.
* `include_feedback` boolean: This boolean indicates whether to include the feedback in the calls data.

```python lines theme={null}
import itertools
from datetime import datetime, timedelta

import pandas as pd

def fetch_calls(client, project_id, start_time, trace_roots_only, limit):
    filter_params = {
        "project_id": project_id,
        "filter": {"started_at": start_time, "trace_roots_only": trace_roots_only},
        "expand_columns": ["inputs.example", "inputs.model"],
        "sort_by": [{"field": "started_at", "direction": "desc"}],
        "include_costs": True,
        "include_feedback": True,
    }
    try:
        calls_stream = client.server.calls_query_stream(filter_params)
        calls = list(
            itertools.islice(calls_stream, limit)
        )  # limit the number of calls to fetch if too many
        print(f"Fetched {len(calls)} calls.")
    except Exception as e:
        print(f"Error fetching calls: {e}")
        return []
    else:
        return calls

calls = fetch_calls(client, PROJECT_NAME, datetime.now() - timedelta(days=1), True, 100)
python
# the raw data is a list of Call objects
pd.DataFrame([call.dict() for call in calls]).head(3)
```

Process the calls using the return from Weave. Extract the relevant information and store it in a list of dictionaries. Then, convert the list of dictionaries to a pandas DataFrame and return it.

```python lines theme={null}
import json
from datetime import datetime

import pandas as pd

def process_calls(calls):
    records = []
    for call in calls:
        feedback = call.summary.get("weave", {}).get("feedback", [])
        thumbs_up = sum(
            1
            for item in feedback
            if isinstance(item, dict) and item.get("payload", {}).get("emoji") == "👍"
        )
        thumbs_down = sum(
            1
            for item in feedback
            if isinstance(item, dict) and item.get("payload", {}).get("emoji") == "👎"
        )
        latency = call.summary.get("weave", {}).get("latency_ms", 0)

        records.append(
            {
                "Call ID": call.id,
                "Trace ID": call.trace_id,  # this is a unique ID for the trace that can be used to retrieve it
                "Display Name": call.display_name,  # this is an optional name you can set in the UI or programatically
                "Latency (ms)": latency,
                "Thumbs Up": thumbs_up,
                "Thumbs Down": thumbs_down,
                "Started At": pd.to_datetime(getattr(call, "started_at", datetime.min)),
                "Inputs": json.dumps(call.inputs, default=str),
                "Outputs": json.dumps(call.output, default=str),
            }
        )
    return pd.DataFrame(records)
python
df_calls = process_calls(calls)
df_calls.head(3)
```

#### Use high-level APIs

Instead of going through every call, Weave also provides high-level APIs to directly access model costs, feedback, and other metrics.
For example, for the cost, use the `query_costs` API to fetch the costs of all LLMs used in the project:

```python lines theme={null}
# Use cost API to get costs
costs = client.query_costs()
df_costs = pd.DataFrame([cost.dict() for cost in costs])
df_costs["total_cost"] = (
    df_costs["prompt_token_cost"] + df_costs["completion_token_cost"]
)

# only show the first row for every unqiue llm_id
df_costs
```

### Gather inputs and generate visualizations

With call data and costs available as DataFrames, you can generate the visualizations using plotly. This is a starter dashboard that you can customize as you like. For a more advanced example, check out the [Streamlit example in the knowledge-worker-weave repo](https://github.com/NiWaRe/knowledge-worker-weave/blob/master/prod_dashboard.py).

```python lines theme={null}
import plotly.express as px
import plotly.graph_objects as go

def plot_feedback_pie_chart(thumbs_up, thumbs_down):
    fig = go.Figure(
        data=[
            go.Pie(
                labels=["Thumbs Up", "Thumbs Down"],
                values=[thumbs_up, thumbs_down],
                marker={"colors": ["#66b3ff", "#ff9999"]},
                hole=0.3,
            )
        ]
    )
    fig.update_traces(textinfo="percent+label", hoverinfo="label+percent")
    fig.update_layout(showlegend=False, title="Feedback Summary")
    return fig

def plot_model_cost_distribution(df):
    fig = px.bar(
        df,
        x="llm_id",
        y="total_cost",
        color="llm_id",
        title="Cost Distribution by Model",
    )
    fig.update_layout(xaxis_title="Model", yaxis_title="Cost (USD)")
    return fig

# See the source code for all the plots
python
plot_feedback_pie_chart(df_calls["Thumbs Up"].sum(), df_calls["Thumbs Down"].sum())
python
plot_model_cost_distribution(df_costs)
```

## Conclusion

This cookbook demonstrated how to create a custom production monitoring dashboard using Weave's APIs and functions. Weave focuses on fast integrations for streamlined input of data as well as extraction of the data for custom processes.

* **Data input:**
  * Framework-agnostic tracing with the [`@weave-op()`](/weave/quickstart#2-log-a-trace-to-a-new-project) decorator and the option to import calls from CSV (see the related [import cookbook](/weave/cookbooks/import_from_csv)).
  * Service API endpoints to log to Weave from various programming frameworks and languages. See the [Service API reference](https://docs.wandb.ai/weave/reference/service-api/calls/call-start) for more details.
* **Data output:**
  * Download the data in CSV, TSV, JSONL, or JSON formats. See the [Service API reference](/weave/reference/service-api) for more details.
  * Export using programmatic access to the data. See the "Use Python" section in the export panel as described in this cookbook. See [Querying and exporting calls](/weave/guides/tracking/tracing#querying-and-exporting-calls) for more details.

This custom dashboard extends Weave's native Traces view to allow tailored monitoring of LLM applications in production. To view a more complex dashboard, check out the [Streamlit example in the agent-dev-collection repo](https://github.com/NiWaRe/agent-dev-collection), where you can add your own Weave project URL.


# PII Data Handling
Source: https://docs.wandb.ai/weave/cookbooks/pii

Handle personally identifiable information (PII) in LLM applications using W&B Weave's data redaction and filtering tools.

<Note>
  This is an interactive notebook. You can run it locally or use the links below:

  * [Open in Google Colab](https://colab.research.google.com/github/wandb/docs/blob/main/weave/cookbooks/source/pii.ipynb)
  * [View source on GitHub](https://github.com/wandb/docs/blob/main/weave/cookbooks/source/pii.ipynb)
</Note>

In this guide, you learn how to use W\&B Weave while ensuring your Personally Identifiable Information (PII) data remains private. Protecting PII helps you meet privacy and compliance requirements while still benefiting from LLM tracing and evaluation. This guide is intended for developers integrating Weave into LLM applications that process sensitive user data.

The guide demonstrates the following methods to identify, redact, and anonymize PII data:

1. **Regular expressions** to identify PII data and redact it.
2. **Microsoft's [Presidio](https://microsoft.github.io/presidio/)**, a Python-based data protection SDK. This tool provides redaction and replacement functionality.
3. **[Faker](https://faker.readthedocs.io/en/master/)**, a Python library to generate fake data, combined with Presidio to anonymize PII data.

Additionally, you learn how to use *`weave.op` input/output logging customization* and *`autopatch_settings`* to integrate PII redaction and anonymization into the workflow. For more information, see [Customize logged inputs and outputs](/weave/guides/tracking/ops/#customize-logged-inputs-and-outputs).

To get started, do the following:

1. Review the [Overview](#overview) section.
2. Complete the [prerequisites](#prerequisites).
3. Review the [available methods](#redaction-methods-overview) for identifying, redacting, and anonymizing PII data.
4. [Apply the methods to Weave calls](#apply-the-methods-to-weave-calls).

## Overview

The following section provides an overview of input and output logging using `weave.op`, as well as best practices for working with PII data in Weave.

### Customize input and output logging using `weave.op`

Weave Ops let you define input and output postprocessing functions. With these functions, you can modify the data that is passed to your LLM call or logged to Weave.

The following example defines two postprocessing functions and passes them as arguments to `weave.op()`.

```python lines theme={null}
from dataclasses import dataclass
from typing import Any

import weave

# Inputs Wrapper Class
@dataclass
class CustomObject:
    x: int
    secret_password: str

# First, define functions for input and output postprocessing:
def postprocess_inputs(inputs: dict[str, Any]) -> dict[str, Any]:
    return {k:v for k,v in inputs.items() if k != "hide_me"}

def postprocess_output(output: CustomObject) -> CustomObject:
    return CustomObject(x=output.x, secret_password="REDACTED")

# Then, when you use the `@weave.op` decorator, pass these processing functions as arguments to the decorator:
@weave.op(
    postprocess_inputs=postprocess_inputs,
    postprocess_output=postprocess_output,
)
def some_llm_call(a: int, hide_me: str) -> CustomObject:
    return CustomObject(x=a, secret_password=hide_me)
```

### Best practices for Weave with PII data

Before using Weave with PII data, review the following best practices. They are grouped by stage of the development lifecycle, with additional guidance on encryption.

#### During testing

* Log anonymized data to check PII detection.
* Track PII handling processes with Weave Traces.
* Measure anonymization performance without exposing real PII.

#### In production

* Never log raw PII.
* Encrypt sensitive fields before logging.

#### Encryption tips

* Use reversible encryption for data you need to decrypt later.
* Apply one-way hashing for unique IDs that you don't need to reverse.
* Consider specialized encryption for data you need to analyze while encrypted.

## Prerequisites

Before applying any of the redaction methods, complete the following setup steps to install dependencies, configure API keys, initialize your Weave project, and load the sample data.

1. First, install the required packages.

```python lines theme={null}
%%capture
# @title required python packages:
!pip install cryptography
!pip install presidio_analyzer
!pip install presidio_anonymizer
!python -m spacy download en_core_web_lg    # Presidio uses spacy NLP engine
!pip install Faker                          # we'll use Faker to replace PII data with fake data
!pip install weave                          # To leverage Traces
!pip install set-env-colab-kaggle-dotenv -q # for env var
!pip install anthropic                      # to use sonnet
!pip install cryptography                   # to encrypt our data
```

2. Create API keys at:

   * [W\&B User Settings](https://wandb.ai/settings)
   * [Anthropic Console](https://platform.claude.com/login?returnTo=%2Fsettings%2Fkeys)

```python lines theme={null}
%%capture
# @title Set up your API keys correctly
# See: https://pypi.org/project/set-env-colab-kaggle-dotenv/ for usage instructions.

from set_env import set_env

_ = set_env("ANTHROPIC_API_KEY")
_ = set_env("WANDB_API_KEY")
```

3. Initialize your Weave project.

```python lines theme={null}
import weave

# Start a new Weave project
WEAVE_PROJECT = "pii_cookbook"
weave.init(WEAVE_PROJECT)
```

4. Load the demo PII dataset, which contains 10 text blocks.

```python lines theme={null}
import requests

url = "https://raw.githubusercontent.com/wandb/docs/main/weave/cookbooks/source/10_pii_data.json"
response = requests.get(url)
pii_data = response.json()

print('PII data first sample: "' + pii_data[0]["text"] + '"')
```

## Redaction methods overview

After you complete the [Prerequisites](#prerequisites), you can choose one of the following methods to detect and protect PII data. Each method identifies and redacts PII, and optionally anonymizes it:

1. **Regular expressions** to identify PII data and redact it.
2. **Microsoft [Presidio](https://microsoft.github.io/presidio/)**, a Python-based data protection SDK that provides redaction and replacement functionality.
3. **[Faker](https://faker.readthedocs.io/en/master/)**, a Python library for generating fake data.

### Method 1: Filter using regular expressions

[Regular expressions (regex)](https://docs.python.org/3/library/re.html) are a straightforward method to identify and redact PII data. Regex lets you define patterns that can match various formats of sensitive information like phone numbers, email addresses, and social security numbers. With regex, you can scan through large volumes of text and replace or redact information without the need for more complex NLP techniques.

```python lines theme={null}
import re

# Define a function to clean PII data using regex
def redact_with_regex(text):
    # Phone number pattern
    # \b         : Word boundary
    # \d{3}      : Exactly 3 digits
    # [-.]?      : Optional hyphen or dot
    # \d{3}      : Another 3 digits
    # [-.]?      : Optional hyphen or dot
    # \d{4}      : Exactly 4 digits
    # \b         : Word boundary
    text = re.sub(r"\b\d{3}[-.]?\d{3}[-.]?\d{4}\b", "<PHONE>", text)

    # Email pattern
    # \b         : Word boundary
    # [A-Za-z0-9._%+-]+ : One or more characters that can be in an email username
    # @          : Literal @ symbol
    # [A-Za-z0-9.-]+ : One or more characters that can be in a domain name
    # \.         : Literal dot
    # [A-Z|a-z]{2,} : Two or more uppercase or lowercase letters (TLD)
    # \b         : Word boundary
    text = re.sub(
        r"\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Z|a-z]{2,}\b", "<EMAIL>", text
    )

    # SSN pattern
    # \b         : Word boundary
    # \d{3}      : Exactly 3 digits
    # -          : Literal hyphen
    # \d{2}      : Exactly 2 digits
    # -          : Literal hyphen
    # \d{4}      : Exactly 4 digits
    # \b         : Word boundary
    text = re.sub(r"\b\d{3}-\d{2}-\d{4}\b", "<SSN>", text)

    # Simple name pattern (this is not comprehensive)
    # \b         : Word boundary
    # [A-Z]      : One uppercase letter
    # [a-z]+     : One or more lowercase letters
    # \s         : One whitespace character
    # [A-Z]      : One uppercase letter
    # [a-z]+     : One or more lowercase letters
    # \b         : Word boundary
    text = re.sub(r"\b[A-Z][a-z]+ [A-Z][a-z]+\b", "<NAME>", text)

    return text
```

To test the function, run the following with a sample text:

```python lines theme={null}
# Test the function
test_text = "My name is John Doe, my email is john.doe@example.com, my phone is 123-456-7890, and my SSN is 123-45-6789."
cleaned_text = redact_with_regex(test_text)
print(f"Raw text:\n\t{test_text}")
print(f"Redacted text:\n\t{cleaned_text}")
```

### Method 2: Redact using Microsoft Presidio

The next method involves complete removal of PII data using [Microsoft Presidio](https://microsoft.github.io/presidio/). Presidio redacts PII and replaces it with a placeholder representing the PII type. For example, Presidio replaces `Alex` in `"My name is Alex"` with `<PERSON>`.

Presidio comes with built-in support for [common entities](https://microsoft.github.io/presidio/supported_entities/). The following example redacts all entities that are a `PHONE_NUMBER`, `PERSON`, `LOCATION`, `EMAIL_ADDRESS`, or `US_SSN`. The Presidio process is encapsulated in a function.

```python lines theme={null}
from presidio_analyzer import AnalyzerEngine
from presidio_anonymizer import AnonymizerEngine

# Set up the Analyzer, which loads an NLP module (spaCy model by default) and other PII recognizers.
analyzer = AnalyzerEngine()

# Set up the Anonymizer, which will use the analyzer results to anonymize the text.
anonymizer = AnonymizerEngine()

# Encapsulate the Presidio redaction process into a function
def redact_with_presidio(text):
    # Analyze the text to identify PII data
    results = analyzer.analyze(
        text=text,
        entities=["PHONE_NUMBER", "PERSON", "LOCATION", "EMAIL_ADDRESS", "US_SSN"],
        language="en",
    )
    # Anonymize the identified PII data
    anonymized_text = anonymizer.anonymize(text=text, analyzer_results=results)
    return anonymized_text.text
```

To test the function, run the following with a sample text:

```python lines theme={null}
text = "My phone number is 212-555-5555 and my name is alex"

# Test the function
anonymized_text = redact_with_presidio(text)

print(f"Raw text:\n\t{text}")
print(f"Redacted text:\n\t{anonymized_text}")
```

### Method 3: Anonymize with replacement using Faker and Presidio

Instead of redacting text, you can anonymize it by using MS Presidio to swap PII like names and phone numbers with fake data generated by the [Faker](https://faker.readthedocs.io/en/master/) Python library. For example, suppose you have the following data:

`"My name is Raphael and I like to fish. My phone number is 212-555-5555"`

After Presidio and Faker process the data, it might look like:

`"My name is Katherine Dixon and I like to fish. My phone number is 667.431.7379"`

To use Presidio and Faker together, you must supply references to your custom operators. These operators direct Presidio to the Faker functions responsible for swapping PII with fake data.

```python lines theme={null}
from faker import Faker
from presidio_anonymizer import AnonymizerEngine
from presidio_anonymizer.entities import OperatorConfig

fake = Faker()

# Create faker functions (note that it has to receive a value)
def fake_name(x):
    return fake.name()

def fake_number(x):
    return fake.phone_number()

# Create custom operator for the PERSON and PHONE_NUMBER" entities
operators = {
    "PERSON": OperatorConfig("custom", {"lambda": fake_name}),
    "PHONE_NUMBER": OperatorConfig("custom", {"lambda": fake_number}),
}

text_to_anonymize = (
    "My name is Raphael and I like to fish. My phone number is 212-555-5555"
)

# Analyzer output
analyzer_results = analyzer.analyze(
    text=text_to_anonymize, entities=["PHONE_NUMBER", "PERSON"], language="en"
)

anonymizer = AnonymizerEngine()

# do not forget to pass the operators from above to the anonymizer
anonymized_results = anonymizer.anonymize(
    text=text_to_anonymize, analyzer_results=analyzer_results, operators=operators
)

print(f"Raw text:\n\t{text_to_anonymize}")
print(f"Anonymized text:\n\t{anonymized_results.text}")
```

To consolidate the code into a single class and expand the list of entities to include the additional ones identified earlier, run the following:

```python lines theme={null}
from typing import ClassVar

from faker import Faker
from presidio_anonymizer import AnonymizerEngine
from presidio_anonymizer.entities import OperatorConfig

# A custom class for generating fake data that extends Faker
class MyFaker(Faker):
    # Create faker functions (note that it has to receive a value)
    def fake_address(self):
        return fake.address()

    def fake_ssn(self):
        return fake.ssn()

    def fake_name(self):
        return fake.name()

    def fake_number(self):
        return fake.phone_number()

    def fake_email(self):
        return fake.email()

    # Create custom operators for the entities
    operators: ClassVar[dict[str, OperatorConfig]] = {
        "PERSON": OperatorConfig("custom", {"lambda": fake_name}),
        "PHONE_NUMBER": OperatorConfig("custom", {"lambda": fake_number}),
        "EMAIL_ADDRESS": OperatorConfig("custom", {"lambda": fake_email}),
        "LOCATION": OperatorConfig("custom", {"lambda": fake_address}),
        "US_SSN": OperatorConfig("custom", {"lambda": fake_ssn}),
    }

    def redact_and_anonymize_with_faker(self, text):
        anonymizer = AnonymizerEngine()
        analyzer_results = analyzer.analyze(
            text=text,
            entities=["PHONE_NUMBER", "PERSON", "LOCATION", "EMAIL_ADDRESS", "US_SSN"],
            language="en",
        )
        anonymized_results = anonymizer.anonymize(
            text=text, analyzer_results=analyzer_results, operators=self.operators
        )
        return anonymized_results.text
```

To test the function, run the following with a sample text:

```python lines theme={null}
faker = MyFaker()
text_to_anonymize = (
    "My name is Raphael and I like to fish. My phone number is 212-555-5555"
)
anonymized_text = faker.redact_and_anonymize_with_faker(text_to_anonymize)

print(f"Raw text:\n\t{text_to_anonymize}")
print(f"Anonymized text:\n\t{anonymized_text}")
```

### Method 4: Use `autopatch_settings`

You can use `autopatch_settings` to configure PII handling directly during initialization for one or more of the supported LLM integrations. W\&B recommends this approach when you want centralized PII handling across all calls to a given integration. The advantages of this method are:

1. PII handling logic is centralized and scoped at initialization, reducing the need for scattered custom logic.
2. You can customize or disable PII processing workflows entirely for specific integrations.

To use `autopatch_settings` to configure PII handling, define `postprocess_inputs` or `postprocess_output` in `op_settings` for any one of the supported LLM integrations.

```python lines theme={null}

def postprocess(inputs: dict) -> dict:
    if "SENSITIVE_KEY" in inputs:
        inputs["SENSITIVE_KEY"] = "REDACTED"
    return inputs

client = weave.init(
    ...,
    autopatch_settings={
        "openai": {
            "op_settings": {
                "postprocess_inputs": postprocess,
                "postprocess_output": ...,
            }
        },
        "anthropic": {
            "op_settings": {
                "postprocess_inputs": ...,
                "postprocess_output": ...,
            }
        }
    },
)
```

## Apply the methods to Weave calls

Now that you've seen each redaction method in isolation, the following examples show how to integrate them into Weave Models and preview the results in Weave Traces.

First, create a [Weave Model](https://docs.wandb.ai/weave/guides/core-types/models). A Weave Model is a combination of information like configuration settings, model weights, and code that defines how the model operates.

The model includes a predict function where the Anthropic API is called. Anthropic's Claude Sonnet performs sentiment analysis while tracing LLM calls using [Traces](https://docs.wandb.ai/weave/quickstart). Claude Sonnet receives a block of text and outputs one of the following sentiment classifications: *positive*, *negative*, or *neutral*. The model also includes the postprocessing functions to ensure that PII data is redacted or anonymized before it's sent to the LLM.

Once you run this code, you receive links to the Weave project page, as well as the specific trace (LLM calls) you ran. Use these links to verify that the postprocessing functions redacted or anonymized the input as expected before it reached the LLM.

### Regex method

As a starting point, you can use regex to identify and redact PII data from the original text.

```python lines theme={null}
import json
from typing import Any

import anthropic

import weave

# Define an input postprocessing function that applies our regex redaction for the model prediction Weave Op
def postprocess_inputs_regex(inputs: dict[str, Any]) -> dict:
    inputs["text_block"] = redact_with_regex(inputs["text_block"])
    return inputs

# Weave model / predict function
class SentimentAnalysisRegexPiiModel(weave.Model):
    model_name: str
    system_prompt: str
    temperature: int

    @weave.op(
        postprocess_inputs=postprocess_inputs_regex,
    )
    async def predict(self, text_block: str) -> dict:
        client = anthropic.AsyncAnthropic()
        response = await client.messages.create(
            max_tokens=1024,
            model=self.model_name,
            system=self.system_prompt,
            messages=[
                {"role": "user", "content": [{"type": "text", "text": text_block}]}
            ],
        )
        result = response.content[0].text
        if result is None:
            raise ValueError("No response from model")
        parsed = json.loads(result)
        return parsed
```

```python lines theme={null}
# create our LLM model with a system prompt
model = SentimentAnalysisRegexPiiModel(
    name="claude-3-sonnet",
    model_name="claude-3-5-sonnet-20240620",
    system_prompt='You are a Sentiment Analysis classifier. You will be classifying text based on their sentiment. Your input will be a block of text. You will answer with one the following rating option["positive", "negative", "neutral"]. Your answer should be one word in json format: {classification}. Ensure that it is valid JSON.',
    temperature=0,
)

print("Model: ", model)
# for every block of text, anonymized first and then predict
for entry in pii_data:
    await model.predict(entry["text"])
```

### Presidio redaction method

Next, use Presidio to identify and redact PII data from the original text.

<Frame>
  <img alt="Presidio PII redaction process with identified PII entities and redacted text output" />
</Frame>

```python lines theme={null}
from typing import Any

import weave

# Define an input postprocessing function that applies our Presidio redaction for the model prediction Weave Op
def postprocess_inputs_presidio(inputs: dict[str, Any]) -> dict:
    inputs["text_block"] = redact_with_presidio(inputs["text_block"])
    return inputs

# Weave model / predict function
class SentimentAnalysisPresidioPiiModel(weave.Model):
    model_name: str
    system_prompt: str
    temperature: int

    @weave.op(
        postprocess_inputs=postprocess_inputs_presidio,
    )
    async def predict(self, text_block: str) -> dict:
        client = anthropic.AsyncAnthropic()
        response = await client.messages.create(
            max_tokens=1024,
            model=self.model_name,
            system=self.system_prompt,
            messages=[
                {"role": "user", "content": [{"type": "text", "text": text_block}]}
            ],
        )
        result = response.content[0].text
        if result is None:
            raise ValueError("No response from model")
        parsed = json.loads(result)
        return parsed
```

```python lines theme={null}
# create our LLM model with a system prompt
model = SentimentAnalysisPresidioPiiModel(
    name="claude-3-sonnet",
    model_name="claude-3-5-sonnet-20240620",
    system_prompt='You are a Sentiment Analysis classifier. You will be classifying text based on their sentiment. Your input will be a block of text. You will answer with one the following rating option["positive", "negative", "neutral"]. Your answer should be one word in json format: {classification}. Ensure that it is valid JSON.',
    temperature=0,
)

print("Model: ", model)
# for every block of text, anonymized first and then predict
for entry in pii_data:
    await model.predict(entry["text"])
```

### Faker and Presidio replacement method

In this example, you use Faker to generate anonymized replacement PII data and use Presidio to identify and replace the PII data in the original text.

<Frame>
  <img alt="Faker and Presidio PII replacement process with original text, identified PII, and anonymized replacement values" />
</Frame>

```python lines theme={null}
from typing import Any

import weave

# Define an input postprocessing function that applies our Faker anonymization and Presidio redaction for the model prediction Weave Op
faker = MyFaker()

def postprocess_inputs_faker(inputs: dict[str, Any]) -> dict:
    inputs["text_block"] = faker.redact_and_anonymize_with_faker(inputs["text_block"])
    return inputs

# Weave model / predict function
class SentimentAnalysisFakerPiiModel(weave.Model):
    model_name: str
    system_prompt: str
    temperature: int

    @weave.op(
        postprocess_inputs=postprocess_inputs_faker,
    )
    async def predict(self, text_block: str) -> dict:
        client = anthropic.AsyncAnthropic()
        response = await client.messages.create(
            max_tokens=1024,
            model=self.model_name,
            system=self.system_prompt,
            messages=[
                {"role": "user", "content": [{"type": "text", "text": text_block}]}
            ],
        )
        result = response.content[0].text
        if result is None:
            raise ValueError("No response from model")
        parsed = json.loads(result)
        return parsed
```

```python lines theme={null}
# create our LLM model with a system prompt
model = SentimentAnalysisFakerPiiModel(
    name="claude-3-sonnet",
    model_name="claude-3-5-sonnet-20240620",
    system_prompt='You are a Sentiment Analysis classifier. You will be classifying text based on their sentiment. Your input will be a block of text. You will answer with one the following rating option["positive", "negative", "neutral"]. Your answer should be one word in json format: {classification}. Ensure that it is valid JSON.',
    temperature=0,
)

print("Model: ", model)
# for every block of text, anonymized first and then predict
for entry in pii_data:
    await model.predict(entry["text"])
```

### `autopatch_settings` method

In the following example, `postprocess_inputs` for `anthropic` is set to the `postprocess_inputs_regex()` function at initialization. The `postprocess_inputs_regex` function applies the `redact_with_regex` method defined in [Method 1: Filter using regular expressions](#method-1-filter-using-regular-expressions). As a result, `redact_with_regex` is applied to all inputs to any `anthropic` models.

```python lines theme={null}
from typing import Any

import weave

client = weave.init(
    ...,
    autopatch_settings={
        "anthropic": {
            "op_settings": {
                "postprocess_inputs": postprocess_inputs_regex,
            }
        }
    },
)

# Define an input postprocessing function that applies our regex redaction for the model prediction Weave Op
def postprocess_inputs_regex(inputs: dict[str, Any]) -> dict:
    inputs["text_block"] = redact_with_regex(inputs["text_block"])
    return inputs

# Weave model / predict function
class SentimentAnalysisRegexPiiModel(weave.Model):
    model_name: str
    system_prompt: str
    temperature: int

    async def predict(self, text_block: str) -> dict:
        client = anthropic.AsyncAnthropic()
        response = await client.messages.create(
            max_tokens=1024,
            model=self.model_name,
            system=self.system_prompt,
            messages=[
                {"role": "user", "content": [{"type": "text", "text": text_block}]}
            ],
        )
        result = response.content[0].text
        if result is None:
            raise ValueError("No response from model")
        parsed = json.loads(result)
        return parsed
```

```python lines theme={null}
# create our LLM model with a system prompt
model = SentimentAnalysisRegexPiiModel(
    name="claude-3-sonnet",
    model_name="claude-3-5-sonnet-20240620",
    system_prompt='You are a Sentiment Analysis classifier. You will be classifying text based on their sentiment. Your input will be a block of text. You will answer with one the following rating option["positive", "negative", "neutral"]. Your answer should be one word in json format: {classification}. Ensure that it is valid JSON.',
    temperature=0,
)

print("Model: ", model)
# for every block of text, anonymized first and then predict
for entry in pii_data:
    await model.predict(entry["text"])
```

### Optional: Encrypt your data

<Frame>
  <img alt="PII data encryption process with encrypted text output and encryption key management" />
</Frame>

In addition to anonymizing PII, you can add an extra layer of security by encrypting your data with the cryptography library's [Fernet](https://cryptography.io/en/latest/fernet/) symmetric encryption. This approach ensures that even if the anonymized data is intercepted, it remains unreadable without the encryption key. The following example shows how to encrypt input text before it's logged and decrypt it inside the model's `predict` method.

```python lines theme={null}
import os
from cryptography.fernet import Fernet
from pydantic import BaseModel, ValidationInfo, model_validator

def get_fernet_key():
    # Check if the key exists in environment variables
    key = os.environ.get('FERNET_KEY')

    if key is None:
        # If the key doesn't exist, generate a new one
        key = Fernet.generate_key()
        # Save the key to an environment variable
        os.environ['FERNET_KEY'] = key.decode()
    else:
        # If the key exists, ensure it's in bytes
        key = key.encode()

    return key

cipher_suite = Fernet(get_fernet_key())

class EncryptedSentimentAnalysisInput(BaseModel):
    encrypted_text: str = None

    @model_validator(mode="before")
    def encrypt_fields(cls, values):
        if "text" in values and values["text"] is not None:
            values["encrypted_text"] = cipher_suite.encrypt(values["text"].encode()).decode()
            del values["text"]
        return values

    @property
    def text(self):
        if self.encrypted_text:
            return cipher_suite.decrypt(self.encrypted_text.encode()).decode()
        return None

    @text.setter
    def text(self, value):
        self.encrypted_text = cipher_suite.encrypt(str(value).encode()).decode()

    @classmethod
    def encrypt(cls, text: str):
        return cls(text=text)

    def decrypt(self):
        return self.text

# Modified sentiment_analysis_model to use the new EncryptedSentimentAnalysisInput
class sentiment_analysis_model(weave.Model):
    model_name: str
    system_prompt: str
    temperature: int

    @weave.op()
    async def predict(self, encrypted_input: EncryptedSentimentAnalysisInput) -> dict:
        client = AsyncAnthropic()

        decrypted_text = encrypted_input.decrypt() # We use the custom class to decrypt the text

        response = await client.messages.create(
            max_tokens=1024,
            model=self.model_name,
            system=self.system_prompt,
            messages=[
                {   "role": "user",
                    "content":[
                        {
                            "type": "text",
                            "text": decrypted_text
                        }
                    ]
                }
            ]
        )
        result = response.content[0].text
        if result is None:
            raise ValueError("No response from model")
        parsed = json.loads(result)
        return parsed

model = sentiment_analysis_model(
    name="claude-3-sonnet",
    model_name="claude-3-5-sonnet-20240620",
    system_prompt="You are a Sentiment Analysis classifier. You will be classifying text based on their sentiment. Your input will be a block of text. You will answer with one the following rating option[\"positive\", \"negative\", \"neutral\"]. Your answer should one word in json format dict where the key is classification.",
    temperature=0
)

for entry in pii_data:
    encrypted_input = EncryptedSentimentAnalysisInput.encrypt(entry["text"])
    await model.predict(encrypted_input)
```


# Scorers as guardrails
Source: https://docs.wandb.ai/weave/cookbooks/scorers_as_guardrails

Learn how to use scorers as guardrails with W&B Weave

<Note>
  This is an interactive notebook. You can run it locally or use the links below:

  * [Open in Google Colab](https://colab.research.google.com/github/wandb/docs/blob/main/weave/cookbooks/source/scorers_as_guardrails.ipynb)
  * [View source on GitHub](https://github.com/wandb/docs/blob/main/weave/cookbooks/source/scorers_as_guardrails.ipynb)
</Note>

# Scorers as guardrails

Weave Scorers are classes with a `score` method that evaluates the performance of a call. They can range from basic rules to LLMs as judges.

This notebook is for Weave users who want to apply Scorers at runtime to block or filter unwanted model outputs. By following along, you build a content safety guardrail that intercepts an LLM response, evaluates it with a Scorer, and returns a fallback message when the Scorer flags the response as unsafe. Using Scorers this way lets you reuse the same evaluation logic that powers offline experiments to enforce safety policies in production.

By the end of this notebook, you have a working `ContentSafetyScorer` that wraps an LLM call and returns either the original response or a safe fallback, depending on the Scorer's verdict.

```python lines theme={null}
%pip install weave --quiet
python
"""
Example demonstrating how to implement guardrails in Weave.
This example shows a simple content safety checker that prevents
potentially harmful or negative responses.
"""

import weave

# Initialize Weave with a descriptive project name
weave.init("content-safety-guardrails")

class ContentSafetyScorer(weave.Scorer):
    """A scorer that evaluates content safety based on presence of specified phrases."""

    unsafe_phrases: list[str]
    case_sensitive: bool = False

    @weave.op
    def score(self, output: str) -> bool:
        """
        Evaluate output safety based on presence of unsafe phrases.

        Args:
            output: The text output to evaluate

        Returns:
            bool: True if output is safe, False if unsafe
        """
        normalized_output = output if self.case_sensitive else output.lower()

        for phrase in self.unsafe_phrases:
            normalized_phrase = phrase if self.case_sensitive else phrase.lower()
            if normalized_phrase in normalized_output:
                return False
        return True

@weave.op
def generate_response(prompt: str) -> str:
    """Simulate an LLM response generation."""
    if "test" in prompt.lower():
        return "I'm sorry, I cannot process that request."
    elif "help" in prompt.lower():
        return "I'd be happy to help you with that!"
    else:
        return "Here's what you requested: " + prompt

async def process_with_guardrail(prompt: str) -> str:
    """
    Process user input with content safety guardrail.
    Returns the response if safe, or a fallback message if unsafe.
    """
    # Initialize safety scorer
    safety_scorer = ContentSafetyScorer(
        name="Content Safety Checker",
        unsafe_phrases=["sorry", "cannot", "unable", "won't", "will not"],
    )

    # Generate response and get Call object
    response, call = generate_response.call(prompt)

    # Apply safety scoring
    evaluation = await call.apply_scorer(safety_scorer)

    # Return response or fallback based on safety check
    if evaluation.result:
        return response
    else:
        return "I cannot provide that response."
python
"""Example usage of the guardrail system."""
test_prompts = [
    "Please help me with my homework",
    "Can you run a test for me?",
    "Tell me a joke",
]

print("Testing content safety guardrails:\n")

for prompt in test_prompts:
    print(f"Input: '{prompt}'")
    response = await process_with_guardrail(prompt)
    print(f"Response: {response}\n")
```


# Limits and expected behaviors
Source: https://docs.wandb.ai/weave/details/limits

A list of Weave's limitations, known issues, and expected behaviors

This page lists Weave's limitations, known issues, and expected behaviors. Use it as a reference when you encounter unexpected results or want to understand how Weave handles edge cases.

* For retryable requests, Weave retries requests starting at 1 second after receiving the first error and then doubles the amount of time between attempts up to 5 minutes. Requests time out after 36 hours.

* Instead of raising exceptions, [`.call()`](/weave/guides/tracking/tracing#creating-calls) captures exceptions and stores them in the `call.exception`. If you need to raise exceptions during execution, set the [`__should_raise` parameter](/weave/reference/python-sdk/trace/op#function-call), like this:

  ```python showLineNumbers theme={null}
  # This raises exceptions
  result, call = foo.call(__should_raise=True)
  ```

* Dedicated Weave instances use a different OpenTelemetry ingress URL. For more information about authoritative endpoints, see [Send OpenTelemetry traces](/weave/guides/tracking/otel).

* Weave sometimes truncates large trace data objects. This occurs because the default trace output is a raw, custom Python object that Weave can't serialize automatically. To return all of your trace data, define a dictionary of strings, like this:

  ```python theme={null}
  import weave

  class MyObj:
      """An object with a large string attribute."""
      def __init__(self, x: int):
          self.x = x

      def __repr__(self):
          return f"MyObj(x={self.x})"

      def to_dict(self):
          return {"x": self.x}

  @weave.op()
  def make_my_obj():
      x = "a" * 10_000
      return MyObj(x)

  def main():
      weave.init("<entity/project>")

      # Define a traceable operation that processes MyObj
      @weave.op()
      def process_obj(obj: MyObj) -> int:
          return len(obj.x)

      # Create and process a large MyObj instance
      large_obj = make_my_obj()
      length = process_obj(large_obj)
      print(f"Length of x in MyObj: {length}")
      print("dict:", large_obj.to_dict())

  if __name__ == "__main__":
      main()
  ```


# Support: Weave
Source: https://docs.wandb.ai/weave/details/support-weave





# Configure Weave environment variables
Source: https://docs.wandb.ai/weave/guides/core-types/env-vars

Configure Weave's behavior at runtime using environment variables

Weave provides a set of environment variables to configure and optimize its behavior. This page is for developers using the Weave SDK who want to adjust runtime behavior such as tracing, logging, caching, and retries without modifying application code. You can set these variables in your shell or within scripts to control specific functionality.

The following examples show two common ways to set Weave environment variables. Use the shell approach to configure Weave for an entire session, or the Python approach to set values from within your script.

```bash theme={null}
# Example of setting environment variables in the shell
export WEAVE_PARALLELISM=10  # Controls the number of parallel workers
export WEAVE_PRINT_CALL_LINK=false  # Disables call link output
```

```python lines theme={null}
# Example of setting environment variables in Python
import os

os.environ["WEAVE_PARALLELISM"] = "10"
os.environ["WEAVE_PRINT_CALL_LINK"] = "false"
```

## Available environment variables

The following table lists each supported environment variable, its type, default value, and the behavior it controls. For accepted values when setting boolean variables, see the note about boolean values.

| Variable                        | Type     | Default         | Description                                                                                                                                                                                                                                                                                                                                              |
| ------------------------------- | -------- | --------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `WANDB_API_KEY`                 | `string` | `None`          | If set, automatically logs you into W\&B Weave without prompting for your API key. To generate an API key, log in to your W\&B account and go to [User Settings](https://wandb.ai/settings).                                                                                                                                                             |
| `WEAVE_DISABLED`                | `bool`   | `false`         | When set to `true`, disables all Weave tracing. Weave ops behave like regular functions.                                                                                                                                                                                                                                                                 |
| `WEAVE_PRINT_CALL_LINK`         | `bool`   | `true`          | Controls whether to print a link to the Weave UI when calling a Weave op. You can also set this directly in your code by configuring the `settings` argument for `weave.init()` like this: `weave.init("your-project-name", settings={"print_call_link": False})`                                                                                        |
| `WEAVE_LOG_LEVEL`               | `str`    | `INFO`          | Controls the log level of the Weave logger.                                                                                                                                                                                                                                                                                                              |
| `WEAVE_CAPTURE_CODE`            | `bool`   | `true`          | Controls whether to save code for ops so you can reload it for later use.                                                                                                                                                                                                                                                                                |
| `WEAVE_DEBUG_HTTP`              | `bool`   | `false`         | When set to `true`, turns on HTTP request and response logging for debugging.                                                                                                                                                                                                                                                                            |
| `WEAVE_PARALLELISM`             | `int`    | `20`            | In evaluations, controls the number of examples to evaluate in parallel. Set to `1` to run examples sequentially.                                                                                                                                                                                                                                        |
| `WEAVE_TRACE_LANGCHAIN`         | `bool`   | `true`          | Controls global tracing for LangChain. Set to `false` to explicitly disable LangChain tracing.                                                                                                                                                                                                                                                           |
| `WEAVE_USE_SERVER_CACHE`        | `bool`   | `true`          | Enables server response caching. When enabled, Weave caches server responses to disk so repeated queries can be served from the cache instead of refetching from the server.                                                                                                                                                                             |
| `WEAVE_SERVER_CACHE_SIZE_LIMIT` | `int`    | `1,000,000,000` | Sets the maximum size limit for the server cache in bytes. When the cache reaches this size, Weave automatically removes older entries to make space for new ones. Important: the underlying implementation uses SQLite, which has a Write-Ahead Log (WAL) that grows to 4 MB regardless of this setting. Weave removes this WAL when the program exits. |
| `WEAVE_SERVER_CACHE_DIR`        | `str`    | `None`          | Specifies the directory where Weave stores cache files. If not set, Weave uses a temporary directory.                                                                                                                                                                                                                                                    |
| `WEAVE_MAX_CALLS_QUEUE_SIZE`    | `int`    | `100000`        | Sets the maximum size of the calls queue. Defaults to `100000`. Setting a value of `0` lets the queue grow unbounded.                                                                                                                                                                                                                                    |
| `WEAVE_ENABLE_WAL`              | `bool`   | `false`         | Enables the Weave [Write-Ahead Log (WAL)](/weave/guides/tracking/write-ahead-log). When set to `true`, Weave writes API requests to disk before sending them to the server, so requests can be recovered if the process exits before they are sent.                                                                                                      |
| `WEAVE_DISABLE_WAL_SENDER`      | `bool`   | `false`         | Disables the Weave Write-Ahead Log (WAL) sender. When set to `true`, Weave writes requests to disk but doesn't flush them to the server. Useful for debugging.                                                                                                                                                                                           |
| `WEAVE_RETRY_MAX_ATTEMPTS`      | `int`    | `3`             | Sets the maximum number of retry attempts for failed requests.                                                                                                                                                                                                                                                                                           |
| `WEAVE_RETRY_MAX_INTERVAL`      | `float`  | `300.0`         | Sets the maximum interval between retry attempts in seconds.                                                                                                                                                                                                                                                                                             |
| `WANDB_BASE_URL`                | `string` | `None`          | Sets the Weave host URL. Equivalent to entering the host URL when prompted by `wandb.login()`. You can specify `WANDB_BASE_URL` and `WANDB_API_KEY` before using `weave.init()` to automatically log in and authenticate to Weave.                                                                                                                       |

<Note>
  All boolean environment variables accept the following values (case-insensitive):

  * `true`, `1`, `yes`, `on` for `True`
  * `false`, `0`, `no`, `off` for `False`
</Note>


# Agno
Source: https://docs.wandb.ai/weave/guides/integrations/agno

Trace Agno multi-agent systems with Weave using OpenTelemetry to capture agent interactions, tool calls, and multi-modal workflows.

You can trace [Agno](https://docs.agno.com/) agent and tool calls in Weave using [OpenTelemetry (OTEL)](https://opentelemetry.io/). Agno is a Python framework for building multi-agent systems with shared memory, knowledge, and reasoning. It's designed to be lightweight and model-independent, and supports multi-modal capabilities including text, image, audio, and video processing.

This guide explains how to trace Agno agent and tool calls using OTEL, and visualize those traces in Weave. You'll learn how to install the required dependencies, configure an OTEL tracer to send data to Weave, and instrument your Agno agents and tools.

<Tip>
  For more information on OTEL tracing in Weave, see [Send OTEL Traces to Weave](../tracking/otel).
</Tip>

## Prerequisites

1. Install the required dependencies:

   ```bash theme={null}
   pip install agno openinference-instrumentation-agno opentelemetry-sdk opentelemetry-exporter-otlp-proto-http
   ```

2. Set your OpenAI API key (or other model provider) as an environment variable:

   ```bash theme={null}
   export OPENAI_API_KEY=your_api_key_here
   ```

3. [Configure OTEL tracing in Weave](#configure-otel-tracing-in-weave).

### Configure OTEL tracing in Weave

To send traces from Agno to Weave, configure OTEL with a `TracerProvider` and an `OTLPSpanExporter`. Set the exporter to the [correct endpoint and HTTP headers for authentication and project identification](#required-configuration).

<Note>
  Store sensitive environment variables like your API key and project info in an environment file (for example, `.env`), and load them using `os.environ`. This keeps your credentials secure and out of your codebase.
</Note>

#### Required configuration

Configure the OTLP exporter with the following endpoint and headers:

* **Endpoint:** `https://trace.wandb.ai/otel/v1/traces`
* **Headers:**
  * `Authorization`: Basic auth using your W\&B API key.
  * `project_id`: Your W\&B entity/project name (for example, `myteam/myproject`).

## Send OTEL traces from Agno to Weave

After you complete the [prerequisites](#prerequisites), you can send OTEL traces from Agno to Weave. The following code snippet demonstrates how to configure an OTLP span exporter and tracer provider to send OTEL traces from an Agno application to Weave.

<Warning>
  To ensure that Weave traces Agno properly, set the global tracer provider *before* using Agno components in your code.
</Warning>

```python lines theme={null}
# tracing.py

import base64
import os
from openinference.instrumentation.agno import AgnoInstrumentor
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk import trace as trace_sdk
from opentelemetry.sdk.trace.export import SimpleSpanProcessor
from opentelemetry import trace

# Load sensitive values from environment variables
WANDB_BASE_URL = "https://trace.wandb.ai"
# Your W&B entity/project name e.g. "myteam/myproject"
PROJECT_ID = os.environ.get("WANDB_PROJECT_ID")
# Create a W&B API key at https://wandb.ai/settings
WANDB_API_KEY = os.environ.get("WANDB_API_KEY")

OTEL_EXPORTER_OTLP_ENDPOINT = f"{WANDB_BASE_URL}/otel/v1/traces"
AUTH = base64.b64encode(f"api:{WANDB_API_KEY}".encode()).decode()

OTEL_EXPORTER_OTLP_HEADERS = {
    "Authorization": f"Basic {AUTH}",
    "project_id": PROJECT_ID,
}

# Create the OTLP span exporter with endpoint and headers
exporter = OTLPSpanExporter(
    endpoint=OTEL_EXPORTER_OTLP_ENDPOINT,
    headers=OTEL_EXPORTER_OTLP_HEADERS,
)

# Create a tracer provider and add the exporter
tracer_provider = trace_sdk.TracerProvider()
tracer_provider.add_span_processor(SimpleSpanProcessor(exporter))

# Set the global tracer provider BEFORE importing/using Agno
trace.set_tracer_provider(tracer_provider)
```

## Trace Agno agents with OTEL

After you set up the tracer provider, you can create and run Agno agents with automatic tracing. Instrument Agno before you create agents to ensure that Weave captures all subsequent agent activity. The following example demonstrates how to create an agent with tools:

```python lines theme={null}
from agno.agent import Agent
from agno.models.openai import OpenAIChat
from agno.tools.yfinance import YFinanceTools

from dotenv import load_dotenv
load_dotenv()

# Load AgnoInstrumentor from the file created above
from tracing import AgnoInstrumentor

# Start instrumenting Agno
AgnoInstrumentor().instrument()

# Create a finance agent
finance_agent = Agent(
    name="Finance Agent",
    model=OpenAIChat(id="gpt-4o-mini"),
    tools=[
        YFinanceTools(
            stock_price=True,
            analyst_recommendations=True,
            company_info=True,
            company_news=True
        )
    ],
    instructions=["Use tables to display data"],
    show_tool_calls=True,
    markdown=True,
)

# Use the agent - Weave automatically traces this call
finance_agent.print_response(
    "What is the current stock price of Apple and what are the latest analyst recommendations?",
    stream=True
)
```

Weave automatically traces all agent operations, so you can visualize the execution flow, model calls, reasoning steps, and tool invocations.

<Frame>
  <img alt="A trace visualization of an Agno agent" />
</Frame>

## Trace Agno tools with OTEL

When you define and use tools with Agno, the trace also captures these tool calls. The OTEL integration automatically instruments both the agent's reasoning process and the individual tool executions, which provides a comprehensive view of your agent's behavior.

Here's an example with multiple tools:

```python lines theme={null}
from agno.agent import Agent
from agno.models.openai import OpenAIChat
from agno.tools.duckduckgo import DuckDuckGoTools
from agno.tools.yfinance import YFinanceTools

from dotenv import load_dotenv
load_dotenv()

# Load AgnoInstrumentor from the file created above
from tracing import AgnoInstrumentor

# Start instrumenting Agno
AgnoInstrumentor().instrument()

# Create an agent with multiple tools
research_agent = Agent(
    name="Research Agent",
    model=OpenAIChat(id="gpt-4o-mini"),
    tools=[
        DuckDuckGoTools(),
        YFinanceTools(stock_price=True, company_info=True),
    ],
    instructions=[
        "Search for current information and financial data",
        "Always include sources",
        "Use tables to display financial data"
    ],
    show_tool_calls=True,
    markdown=True,
)

# Use the agent - Weave traces the tool calls
research_agent.print_response(
    "Research Tesla's recent performance and news. Include stock price and any recent developments.",
    stream=True
)
```

<Frame>
  <img alt="A trace visualization of Agno tool calls" />
</Frame>

## Trace multi-agent teams with OTEL

Agno's multi-agent architecture lets you create teams of agents that collaborate and share context. Weave also traces these team interactions in full:

```python lines theme={null}
from agno.agent import Agent
from agno.models.openai import OpenAIChat
from agno.tools.duckduckgo import DuckDuckGoTools
from agno.tools.yfinance import YFinanceTools

from dotenv import load_dotenv
load_dotenv()

# Load AgnoInstrumentor from the tracing.py file
from tracing import AgnoInstrumentor

# Start instrumenting Agno
AgnoInstrumentor().instrument()

# Create specialized agents
web_agent = Agent(
    name="Web Agent",
    role="Search the web for information",
    model=OpenAIChat(id="gpt-4o-mini"),
    tools=[DuckDuckGoTools()],
    instructions="Always include sources",
    show_tool_calls=True,
    markdown=True,
)

finance_agent = Agent(
    name="Finance Agent",
    role="Get financial data",
    model=OpenAIChat(id="gpt-4o-mini"),
    tools=[YFinanceTools(stock_price=True, analyst_recommendations=True)],
    instructions="Use tables to display data",
    show_tool_calls=True,
    markdown=True,
)

# Create a team of agents
agent_team = Agent(
    team=[web_agent, finance_agent],
    model=OpenAIChat(id="gpt-4o"),
    instructions=["Always include sources", "Use tables to display data"],
    show_tool_calls=True,
    markdown=True,
)

# Use the team - Weave traces all agent interactions
agent_team.print_response(
    "What's the current market sentiment around NVIDIA? Include both news analysis and financial metrics.",
    stream=True
)
```

This multi-agent trace shows the coordination between different agents in Weave, which provides visibility into how tasks are distributed and executed across your agent team.

<Frame>
  <img alt="A trace visualization of Agno multi-agent team" />
</Frame>

## Work with reasoning agents

Agno provides built-in reasoning capabilities that help agents process problems step-by-step. Traces also capture these reasoning processes:

```python lines theme={null}
from agno.agent import Agent
from agno.models.openai import OpenAIChat
from agno.tools.reasoning import ReasoningTools
from agno.tools.yfinance import YFinanceTools

from dotenv import load_dotenv
load_dotenv()

# Load AgnoInstrumentor from the tracing.py file
from tracing import AgnoInstrumentor

# Start instrumenting Agno
AgnoInstrumentor().instrument()

# Create a reasoning agent
reasoning_agent = Agent(
    name="Reasoning Finance Agent",
    model=OpenAIChat(id="gpt-4o"),
    tools=[
        ReasoningTools(add_instructions=True),
        YFinanceTools(
            stock_price=True,
            analyst_recommendations=True,
            company_info=True,
            company_news=True
        ),
    ],
    instructions="Use tables to display data and show your reasoning process",
    show_tool_calls=True,
    markdown=True,
)

# Use the reasoning agent
reasoning_agent.print_response(
    "Should I invest in Apple stock right now? Analyze the current situation and provide a reasoned recommendation.",
    stream=True
)
```

The trace displays the reasoning steps that show how the agent breaks down complex problems and makes decisions.

<Frame>
  <img alt="A trace visualization of Agno reasoning agent" />
</Frame>

## Work with memory and knowledge

Agno agents can maintain memory and access knowledge bases, which lets agents persist context across interactions. Weave also traces these operations:

```python lines theme={null}
from agno.agent import Agent
from agno.models.openai import OpenAIChat
from agno.memory import AgentMemory
from agno.storage.sqlite import SqliteStorage

from dotenv import load_dotenv
load_dotenv()

# Load AgnoInstrumentor from the tracing.py file
from tracing import AgnoInstrumentor

# Start instrumenting Agno
AgnoInstrumentor().instrument()


# Create an agent with memory
memory_agent = Agent(
    name="Memory Agent",
    model=OpenAIChat(id="gpt-4o-mini"),
    memory=AgentMemory(),
    storage=SqliteStorage(
        table_name="agent_sessions",
        db_file="agent_memory.db"
    ),
    instructions="Remember our conversation history",
    show_tool_calls=True,
    markdown=True,
)

# First interaction
memory_agent.print_response("My name is John and I'm interested in AI investing strategies.")

# Second interaction - agent remembers the previous context
memory_agent.print_response("What specific AI companies would you recommend for my portfolio?")
```

<Frame>
  <img alt="A trace visualization of Agno memory usage" />
</Frame>

Memory operations, including storing and retrieving conversation history, appear in the trace.

## Learn more

* [Official Agno documentation](https://docs.agno.com/)
* [Official OTEL documentation](https://opentelemetry.io/)
* [Agno GitHub repository](https://github.com/agno-agi/agno)
* [OpenInference Agno instrumentation](https://pypi.org/project/openinference-instrumentation-agno/)


# AutoGen
Source: https://docs.wandb.ai/weave/guides/integrations/autogen

Use Weave to track and monitor AutoGen agents and multi-agent systems

AutoGen is a framework from Microsoft for building AI agents and applications. It simplifies the creation of complex multi-agent systems. AutoGen offers components for conversational AI (`AgentChat`), core multi-agent functionalities (`Core`), and integrations with external services (`Extensions`). AutoGen also provides a `Studio` for no-code agent prototyping. For more details, visit the [official AutoGen documentation](https://microsoft.github.io/autogen/stable//index.html).

<Note>
  This guide assumes you have a basic understanding of [AutoGen](https://microsoft.github.io/autogen/stable/index.html).
</Note>

Weave integrates with [AutoGen](https://microsoft.github.io/autogen/stable/index.html) to help you trace and visualize the execution of your multi-agent applications. When you initialize Weave, it automatically tracks interactions within `autogen_agentchat`, `autogen_core`, and `autogen_ext`. This guide walks you through how to configure Weave with AutoGen and shows worked examples that cover model clients, agents with tools, group chats, memory, RAG workflows, agent runtimes, sequential workflows, and code executors. By the end, you can capture detailed traces of your AutoGen applications in Weave to debug agent behavior, monitor LLM usage, and understand how agents interact across complex workflows.

## Prerequisites

Before you begin, you must have AutoGen and Weave installed. You also need any SDKs for the LLM providers you intend to use (for example, OpenAI or Anthropic).

```bash theme={null}
pip install autogen_agentchat "autogen_ext[openai,anthropic]" weave
```

Set up your API keys as environment variables so the model clients can authenticate with the LLM providers used in the examples that follow:

```python lines theme={null}
import os

os.environ["OPENAI_API_KEY"] = "[YOUR-OPENAI-API-KEY]"
os.environ["ANTHROPIC_API_KEY"] = "[YOUR-ANTHROPIC-API-KEY]"
```

## Basic setup

Initialize Weave at the beginning of your script to start capturing traces. After you initialize Weave, it automatically instruments AutoGen calls in the rest of your script.

```python lines {2} theme={null}
import weave
weave.init("autogen-demo")
```

After you run this snippet, Weave is configured to send traces to the `autogen-demo` project, and Weave automatically captures any subsequent AutoGen activity from your script.

## Traces for a model client

The following sections describe how Weave traces calls made directly to model clients within AutoGen, including single calls, streamed responses, and cached responses.

### Traces for a client create call

This example demonstrates tracing a call to an `OpenAIChatCompletionClient`.

```python lines theme={null}
import asyncio
from autogen_core.models import UserMessage
from autogen_ext.models.openai import OpenAIChatCompletionClient
# from autogen_ext.models.anthropic import AnthropicChatCompletionClient

async def simple_client_call(model_name = "gpt-4o"):
    model_client = OpenAIChatCompletionClient(
        model=model_name,
    )
    # Alternatively, you can use Anthropic or other model clients
    # model_client = AnthropicChatCompletionClient(
        # model="claude-3-haiku-20240307"
    # )
    response = await model_client.create(
        [UserMessage(content="Hello, how are you?", source="user")]
    )
    print(response)

asyncio.run(simple_client_call())

```

[<img alt="autogen-simple-client.png" />](https://wandb.ai/parambharat/autogen-demo/weave/traces?view=traces_default\&peekPath=%2Fparambharat%2Fautogen-demo%2Fcalls%2F0196ee09-8dcf-7b72-8cdc-7699608cd6ef%3FhideTraceTree%3D0)

### Traces for a client create call with streaming

Weave also supports tracing streamed responses.

```python lines theme={null}

async def simple_client_call_stream(model_name = "gpt-4o"):
    openai_model_client = OpenAIChatCompletionClient(model=model_name)
    async for item in openai_model_client.create_stream(
        [UserMessage(content="Hello, how are you?", source="user")]
    ):
      print(item, flush=True, end="")

asyncio.run(simple_client_call_stream())

```

[<img alt="autogen-streaming-client.png" />](https://wandb.ai/parambharat/autogen-demo/weave/traces?view=traces_default\&peekPath=%2Fparambharat%2Fautogen-demo%2Fcalls%2F0196ee0e-24be-7523-b15e-04f87c03ac68%3FhideTraceTree%3D0)

### Traces for cached client calls

You can use AutoGen's `ChatCompletionCache`, and Weave traces these interactions to show whether a response came from the cache or a new call.

```python lines theme={null}

from autogen_ext.models.cache import ChatCompletionCache

async def run_cache_client(model_name = "gpt-4o"):
      openai_model_client = OpenAIChatCompletionClient(model=model_name)
      cache_client = ChatCompletionCache(openai_model_client,)

      response = await cache_client.create(
          [UserMessage(content="Hello, how are you?", source="user")]
      )
      print(response)  # Should print response from OpenAI
      response = await cache_client.create(
          [UserMessage(content="Hello, how are you?", source="user")]
      )
      print(response)  # Should print cached response

asyncio.run(run_cache_client())

```

[<img alt="autogen-cached-client.png" />](https://wandb.ai/parambharat/autogen-demo/weave/traces?view=traces_default\&peekPath=%2Fparambharat%2Fautogen-demo%2Fcalls%2F0196ee11-fded-72c2-baaa-7c0ba2a7cd3b%3FhideTraceTree%3D0)

## Traces for an agent with tool calls

This section shows how Weave traces agents and their tool usage to give you visibility into how agents select and execute tools. The following example defines a weather tool and attaches it to an `AssistantAgent`.

```python lines theme={null}
from autogen_agentchat.agents import AssistantAgent

async def get_weather(city: str) -> str:
    return f"The weather in {city} is 73 degrees and Sunny."

async def run_agent_with_tools(model_name = "gpt-4o"):
    model_client = OpenAIChatCompletionClient(model=model_name)

    agent = AssistantAgent(
        name="weather_agent",
        model_client=model_client,
        tools=[get_weather],
        system_message="You are a helpful assistant.",
        reflect_on_tool_use=True,
    )
    # For streaming output to console:
    # await Console(agent.run_stream(task="What is the weather in New York?"))
    res = await agent.run(task="What is the weather in New York?")
    print(res)
    await model_client.close()

asyncio.run(run_agent_with_tools())

```

[<img alt="autogen-agent-tools.png" />](https://wandb.ai/parambharat/autogen-demo/weave/traces?view=traces_default\&peekPath=%2Fparambharat%2Fautogen-demo%2Fcalls%2F0196ee13-e5ca-72a1-b7b6-4b263fad89e3%3FhideTraceTree%3D0)

## Traces for a round-robin GroupChat

Weave traces interactions within group chats, such as `RoundRobinGroupChat`, so you can follow the conversation flow between agents. To group all agent turns under a single parent trace for streamlined inspection, wrap the group chat function with `@weave.op`. This step is optional but recommended.

```python lines theme={null}

from autogen_agentchat.conditions import TextMentionTermination
from autogen_agentchat.teams import RoundRobinGroupChat

# we add this weave op here because we want to trace the entire group chat
# it's completely optional but highly recommended to use it

@weave.op
async def run_round_robin_group_chat(model_name="gpt-4o"):
    model_client = OpenAIChatCompletionClient(model=model_name)

    primary_agent = AssistantAgent(
        "primary",
        model_client=model_client,
        system_message="You are a helpful AI assistant.",
    )

    critic_agent = AssistantAgent(
        "critic",
        model_client=model_client,
        system_message="Provide constructive feedback. Respond with 'APPROVE' to when your feedbacks are addressed.",
    )

    text_termination = TextMentionTermination("APPROVE")

    team = RoundRobinGroupChat(
        [primary_agent, critic_agent], termination_condition=text_termination
    )
    await team.reset()
    # For streaming output to console:
    # await Console(team.run_stream(task="Write a short poem about the fall season."))
    result = await team.run(task="Write a short poem about the fall season.")
    print(result)
    await model_client.close()


asyncio.run(run_round_robin_group_chat())

```

[<img alt="round_robin_group_chat.png" />](https://wandb.ai/parambharat/autogen-demo/weave/traces?filter=%7B%22opVersionRefs%22%3A%5B%22weave%3A%2F%2F%2Fparambharat%2Fautogen-demo%2Fop%2Frun_round_robin_group_chat%3A*%22%5D%7D\&peekPath=%2Fparambharat%2Fautogen-demo%2Fcalls%2F0196f16c-26ce-7b32-8f0c-2366d29038a3%3FdescendentCallId%3D0196f16c-26ce-7b32-8f0c-2366d29038a3%26hideTraceTree%3D0)

## Traces for memory

Weave can trace AutoGen's memory components. Use `@weave.op()` to group memory operations under a single trace for better readability, so that memory add and retrieval calls appear together with the agent run that uses them.

```python lines theme={null}

from autogen_core.memory import ListMemory, MemoryContent, MemoryMimeType

# We add this weave op here because we want to trace 
# the memory add calls along with the memory get calls under a single trace
# it's completely optional but highly recommended to use it

@weave.op
async def run_memory_agent(model_name="gpt-4o"):
    user_memory = ListMemory()

    await user_memory.add(
        MemoryContent(
            content="The weather should be in metric units",
            mime_type=MemoryMimeType.TEXT,
        )
    )

    await user_memory.add(
        MemoryContent(
            content="Meal recipe must be vegan", mime_type=MemoryMimeType.TEXT
        )
    )

    async def get_weather(city: str, units: str = "imperial") -> str:
        if units == "imperial":
            return f"The weather in {city} is 73 °F and Sunny."
        elif units == "metric":
            return f"The weather in {city} is 23 °C and Sunny."
        else:
            return f"Sorry, I don't know the weather in {city}."

    model_client = OpenAIChatCompletionClient(model=model_name)
    assistant_agent = AssistantAgent(
        name="assistant_agent",
        model_client=model_client,
        tools=[get_weather],
        memory=[user_memory],
    )

    # For streaming output to console:
    # stream = assistant_agent.run_stream(task="What is the weather in New York?")
    # await Console(stream)
    result = await assistant_agent.run(task="What is the weather in New York?")
    print(result)
    await model_client.close()


asyncio.run(run_memory_agent())

```

[<img alt="autogen-memory.png" />](https://wandb.ai/parambharat/autogen-demo/weave/traces?view=traces_default\&peekPath=%2Fparambharat%2Fautogen-demo%2Fcalls%2F0196ee18-28b6-7063-90df-77aaedf88dc9%3FhideTraceTree%3D0)

## Traces for RAG workflows

Weave can trace Retrieval Augmented Generation (RAG) workflows, including document indexing and retrieval with memory systems like `ChromaDBVectorMemory`. To visualize the entire flow so that indexing, retrieval, and the resulting LLM call appear together in a single trace, decorate the RAG process with `@weave.op()`.

<Note>
  The RAG example requires `chromadb`. Install it with `pip install chromadb`.
</Note>

```python lines {1,65} theme={null}
# !pip install -q chromadb 
# Ensure chromadb is installed in your environment: `pip install chromadb`

import re
from typing import List
import os
from pathlib import Path

import aiofiles
import aiohttp

from autogen_core.memory import Memory, MemoryContent, MemoryMimeType
from autogen_ext.memory.chromadb import (
    ChromaDBVectorMemory,
    PersistentChromaDBVectorMemoryConfig,
)

class SimpleDocumentIndexer:
    def __init__(self, memory: Memory, chunk_size: int = 1500) -> None:
        self.memory = memory
        self.chunk_size = chunk_size

    async def _fetch_content(self, source: str) -> str:
        if source.startswith(("http://", "https://")):
            async with aiohttp.ClientSession() as session:
                async with session.get(source) as response:
                    return await response.text()
        else:
            async with aiofiles.open(source, "r", encoding="utf-8") as f:
                return await f.read()

    def _strip_html(self, text: str) -> str:
        text = re.sub(r"<[^>]*>", " ", text)
        text = re.sub(r"\\s+", " ", text)
        return text.strip()

    def _split_text(self, text: str) -> List[str]:
        chunks: list[str] = []
        for i in range(0, len(text), self.chunk_size):
            chunk = text[i : i + self.chunk_size]
            chunks.append(chunk.strip())
        return chunks

    async def index_documents(self, sources: List[str]) -> int:
        total_chunks = 0
        for source in sources:
            try:
                content = await self._fetch_content(source)
                if "<" in content and ">" in content:
                    content = self._strip_html(content)
                chunks = self._split_text(content)
                for i, chunk in enumerate(chunks):
                    await self.memory.add(
                        MemoryContent(
                            content=chunk,
                            mime_type=MemoryMimeType.TEXT,
                            metadata={"source": source, "chunk_index": i},
                        )
                    )
                total_chunks += len(chunks)
            except Exception as e:
                print(f"Error indexing {source}: {str(e)}")
        return total_chunks

@weave.op
async def run_rag_agent(model_name="gpt-4o"):
    rag_memory = ChromaDBVectorMemory(
        config=PersistentChromaDBVectorMemoryConfig(
            collection_name="autogen_docs",
            persistence_path=os.path.join(str(Path.home()), ".chromadb_autogen_weave"),
            k=3,
            score_threshold=0.4,
        )
    )
    # await rag_memory.clear() # Uncomment to clear existing memory if needed

    async def index_autogen_docs() -> None:
        indexer = SimpleDocumentIndexer(memory=rag_memory)
        sources = [
            "https://raw.githubusercontent.com/microsoft/autogen/main/README.md",
            "https://microsoft.github.io/autogen/dev/user-guide/agentchat-user-guide/tutorial/agents.html",
        ]
        chunks: int = await indexer.index_documents(sources)
        print(f"Indexed {chunks} chunks from {len(sources)} AutoGen documents")
    
    # Only index if the collection is empty or you want to re-index
    # For demo purposes, we might index it each time or check if already indexed.
    # This example will try to index each run. Consider adding a check.
    await index_autogen_docs()

    model_client = OpenAIChatCompletionClient(model=model_name)
    rag_assistant = AssistantAgent(
        name="rag_assistant",
        model_client=model_client,
        memory=[rag_memory],
    )
    
    # For streaming output to console:
    # stream = rag_assistant.run_stream(task="What is AgentChat?")
    # await Console(stream)
    result = await rag_assistant.run(task="What is AgentChat?")
    print(result)

    await rag_memory.close()
    await model_client.close()

asyncio.run(run_rag_agent())
```

[<img alt="autogen-rag.png" />](https://wandb.ai/parambharat/autogen-demo/weave/traces?view=traces_default\&peekPath=%2Fparambharat%2Fautogen-demo%2Fcalls%2F0196ee1b-bac5-7b80-8be7-6e6ea7d1d63d%3FhideTraceTree%3D0)

## Traces for agent runtimes

Weave can trace operations within AutoGen's agent runtimes, like `SingleThreadedAgentRuntime`. Wrap the runtime execution function with `@weave.op()` to group related traces so you can see the full sequence of message handlers that fire during a runtime run.

```python lines {50} theme={null}
from dataclasses import dataclass
from typing import Callable

from autogen_core import (
    DefaultTopicId,
    MessageContext,
    RoutedAgent,
    default_subscription,
    message_handler,
    AgentId,
    SingleThreadedAgentRuntime
)

@dataclass
class Message:
    content: int

@default_subscription
class Modifier(RoutedAgent):
    def __init__(self, modify_val: Callable[[int], int]) -> None:
        super().__init__("A modifier agent.")
        self._modify_val = modify_val

    @message_handler
    async def handle_message(self, message: Message, ctx: MessageContext) -> None:
        val = self._modify_val(message.content)
        print(f"{'-'*80}\\nModifier:\\nModified {message.content} to {val}")
        await self.publish_message(Message(content=val), DefaultTopicId())

@default_subscription
class Checker(RoutedAgent):
    def __init__(self, run_until: Callable[[int], bool]) -> None:
        super().__init__("A checker agent.")
        self._run_until = run_until

    @message_handler
    async def handle_message(self, message: Message, ctx: MessageContext) -> None:
        if not self._run_until(message.content):
            print(f"{'-'*80}\\nChecker:\\n{message.content} passed the check, continue.")
            await self.publish_message(
                Message(content=message.content), DefaultTopicId()
            )
        else:
            print(f"{'-'*80}\\nChecker:\\n{message.content} failed the check, stopping.")

# we add this weave op here because we want to trace 
# the entire agent runtime call under a single trace
# it's completely optional but highly recommended to use it

@weave.op
async def run_agent_runtime() -> None:
    runtime = SingleThreadedAgentRuntime()

    await Modifier.register(
        runtime,
        "modifier",
        lambda: Modifier(modify_val=lambda x: x - 1),
    )

    await Checker.register(
        runtime,
        "checker",
        lambda: Checker(run_until=lambda x: x <= 1),
    )

    runtime.start()
    await runtime.send_message(Message(content=3), AgentId("checker", "default"))
    await runtime.stop_when_idle()

asyncio.run(run_agent_runtime())

```

[<img alt="autogen-runtime.png" />](https://wandb.ai/parambharat/autogen-demo/weave/traces?view=traces_default\&peekPath=%2Fparambharat%2Fautogen-demo%2Fcalls%2F0196ee1d-6246-7f11-afb1-3a1874f79023%3FhideTraceTree%3D0)

## Traces for sequential workflows

You can trace complex agent workflows that define sequences of agent interactions. Use `@weave.op()` to provide a high-level trace for the entire workflow so that each agent's contribution is nested under a single parent call. The following example chains a concept extractor, a writer, a format-and-proof agent, and a user agent to produce a polished marketing copy.

```python lines {108} theme={null}

from autogen_core import TopicId, type_subscription
from autogen_core.models import ChatCompletionClient, SystemMessage, UserMessage

@dataclass
class WorkflowMessage:
    content: str

concept_extractor_topic_type = "ConceptExtractorAgent"
writer_topic_type = "WriterAgent"
format_proof_topic_type = "FormatProofAgent"
user_topic_type = "User"

@type_subscription(topic_type=concept_extractor_topic_type)
class ConceptExtractorAgent(RoutedAgent):
    def __init__(self, model_client: ChatCompletionClient) -> None:
        super().__init__("A concept extractor agent.")
        self._system_message = SystemMessage(
            content=(
                "You are a marketing analyst. Given a product description, identify:\n"
                "- Key features\n"
                "- Target audience\n"
                "- Unique selling points\n\n"
            )
        )
        self._model_client = model_client

    @message_handler
    async def handle_user_description(self, message: WorkflowMessage, ctx: MessageContext) -> None:
        prompt = f"Product description: {message.content}"
        llm_result = await self._model_client.create(
            messages=[self._system_message, UserMessage(content=prompt, source=self.id.key)],
            cancellation_token=ctx.cancellation_token,
        )
        response = llm_result.content
        assert isinstance(response, str)
        print(f"{'-'*80}\\n{self.id.type}:\\n{response}")
        await self.publish_message(
            WorkflowMessage(response), topic_id=TopicId(writer_topic_type, source=self.id.key)
        )

@type_subscription(topic_type=writer_topic_type)
class WriterAgent(RoutedAgent):
    def __init__(self, model_client: ChatCompletionClient) -> None:
        super().__init__("A writer agent.")
        self._system_message = SystemMessage(
            content=(
                "You are a marketing copywriter. Given a block of text describing features, audience, and USPs, "
                "compose a compelling marketing copy (like a newsletter section) that highlights these points. "
                "Output should be short (around 150 words), output just the copy as a single text block."
            )
        )
        self._model_client = model_client
    
    @message_handler
    async def handle_intermediate_text(self, message: WorkflowMessage, ctx: MessageContext) -> None:
        prompt = f"Below is the info about the product:\\n\\n{message.content}"
        llm_result = await self._model_client.create(
            messages=[self._system_message, UserMessage(content=prompt, source=self.id.key)],
            cancellation_token=ctx.cancellation_token,
        )
        response = llm_result.content
        assert isinstance(response, str)
        print(f"{'-'*80}\\n{self.id.type}:\\n{response}")
        await self.publish_message(
            WorkflowMessage(response), topic_id=TopicId(format_proof_topic_type, source=self.id.key)
        )

@type_subscription(topic_type=format_proof_topic_type)
class FormatProofAgent(RoutedAgent):
    def __init__(self, model_client: ChatCompletionClient) -> None:
        super().__init__("A format & proof agent.")
        self._system_message = SystemMessage(
            content=(
                "You are an editor. Given the draft copy, correct grammar, improve clarity, ensure consistent tone, "
                "give format and make it polished. Output the final improved copy as a single text block."
            )
        )
        self._model_client = model_client

    @message_handler
    async def handle_intermediate_text(self, message: WorkflowMessage, ctx: MessageContext) -> None:
        prompt = f"Draft copy:\\n{message.content}."
        llm_result = await self._model_client.create(
            messages=[self._system_message, UserMessage(content=prompt, source=self.id.key)],
            cancellation_token=ctx.cancellation_token,
        )
        response = llm_result.content
        assert isinstance(response, str)
        print(f"{'-'*80}\\n{self.id.type}:\\n{response}")
        await self.publish_message(
            WorkflowMessage(response), topic_id=TopicId(user_topic_type, source=self.id.key)
        )

@type_subscription(topic_type=user_topic_type)
class UserAgent(RoutedAgent):
    def __init__(self) -> None:
        super().__init__("A user agent that outputs the final copy to the user.")

    @message_handler
    async def handle_final_copy(self, message: WorkflowMessage, ctx: MessageContext) -> None:
        print(f"\\n{'-'*80}\\n{self.id.type} received final copy:\\n{message.content}")

# we add this weave op here because we want to trace 
# the entire agent workflow under a single trace
# it's completely optional but highly recommended to use it

@weave.op(call_display_name="Sequential Agent Workflow")
async def run_agent_workflow(model_name="gpt-4o"):
    model_client = OpenAIChatCompletionClient(model=model_name)
    runtime = SingleThreadedAgentRuntime()

    await ConceptExtractorAgent.register(runtime, type=concept_extractor_topic_type, factory=lambda: ConceptExtractorAgent(model_client=model_client))
    await WriterAgent.register(runtime, type=writer_topic_type, factory=lambda: WriterAgent(model_client=model_client))
    await FormatProofAgent.register(runtime, type=format_proof_topic_type, factory=lambda: FormatProofAgent(model_client=model_client))
    await UserAgent.register(runtime, type=user_topic_type, factory=lambda: UserAgent())

    runtime.start()
    await runtime.publish_message(
        WorkflowMessage(
            content="An eco-friendly stainless steel water bottle that keeps drinks cold for 24 hours"
        ),
        topic_id=TopicId(concept_extractor_topic_type, source="default"),
    )
    await runtime.stop_when_idle()
    await model_client.close()

asyncio.run(run_agent_workflow())

```

[<img alt="autogen-sequential-workflow.png" />](https://wandb.ai/parambharat/autogen-demo/weave/traces?view=traces_default\&peekPath=%2Fparambharat%2Fautogen-demo%2Fcalls%2F0196ee1f-dd53-73f2-9119-2a44da92c5ae%3FhideTraceTree%3D0)

## Traces for code executors

<Warning>
  **Docker required**
  This example involves code execution that uses Docker and might not work in all environments (for example, Colab directly). Ensure Docker is running locally if you try this.
</Warning>

Weave traces the generation and execution of code by AutoGen agents. You can inspect both the code an assistant agent produces and the output an executor agent returns when that code runs.

````python lines {69} theme={null}

import tempfile
from autogen_core import DefaultTopicId
from autogen_core.code_executor import CodeBlock, CodeExecutor
from autogen_core.models import (
    AssistantMessage,
    ChatCompletionClient,
    LLMMessage,
    SystemMessage,
    UserMessage,
)
from autogen_ext.code_executors.docker import DockerCommandLineCodeExecutor


@dataclass
class CodeGenMessage:
    content: str

@default_subscription
class Assistant(RoutedAgent):
    def __init__(self, model_client: ChatCompletionClient) -> None:
        super().__init__("An assistant agent.")
        self._model_client = model_client
        self._chat_history: List[LLMMessage] = [
           SystemMessage(
                content="""Write Python script in markdown block, and it will be executed.
Always save figures to file in the current directory. Do not use plt.show(). All code required to complete this task must be contained within a single response.""",
            )
        ]

    @message_handler
    async def handle_message(self, message: CodeGenMessage, ctx: MessageContext) -> None:
        self._chat_history.append(UserMessage(content=message.content, source="user"))
        result = await self._model_client.create(self._chat_history)
        print(f"\\n{'-'*80}\\nAssistant:\\n{result.content}")
        self._chat_history.append(AssistantMessage(content=result.content, source="assistant"))
        await self.publish_message(CodeGenMessage(content=result.content), DefaultTopicId())

def extract_markdown_code_blocks(markdown_text: str) -> List[CodeBlock]:
    pattern = re.compile(r"```(?:\\s*([\\w\\+\\-]+))?\\n([\\s\\S]*?)```")
    matches = pattern.findall(markdown_text)
    code_blocks: List[CodeBlock] = []
    for match in matches:
        language = match[0].strip() if match[0] else ""
        code_content = match[1]
        code_blocks.append(CodeBlock(code=code_content, language=language))
    return code_blocks

@default_subscription
class Executor(RoutedAgent):
    def __init__(self, code_executor: CodeExecutor) -> None:
        super().__init__("An executor agent.")
        self._code_executor = code_executor

    @message_handler
    async def handle_message(self, message: CodeGenMessage, ctx: MessageContext) -> None:
        code_blocks = extract_markdown_code_blocks(message.content)
        if code_blocks:
            result = await self._code_executor.execute_code_blocks(
                code_blocks, cancellation_token=ctx.cancellation_token
            )
            print(f"\\n{'-'*80}\\nExecutor:\\n{result.output}")
            await self.publish_message(CodeGenMessage(content=result.output), DefaultTopicId())

# we add this weave op here because we want to trace 
# the entire code gen workflow under a single trace
# it's completely optional but highly recommended to use it

@weave.op(call_display_name="CodeGen Agent Workflow")
async def run_codegen(model_name="gpt-4o"): # Updated model
    work_dir = tempfile.mkdtemp()
    runtime = SingleThreadedAgentRuntime()

    # Ensure Docker is running for this example
    try:
        async with DockerCommandLineCodeExecutor(work_dir=work_dir) as executor:
            model_client = OpenAIChatCompletionClient(model=model_name)
            await Assistant.register(runtime, "assistant", lambda: Assistant(model_client=model_client))
            await Executor.register(runtime, "executor", lambda: Executor(executor))

            runtime.start()
            await runtime.publish_message(
                CodeGenMessage(content="Create a plot of NVDA vs TSLA stock returns YTD from 2024-01-01."),
                DefaultTopicId(),
            )
            await runtime.stop_when_idle()
            await model_client.close()
    except Exception as e:
        print(f"Could not run Docker code executor example: {e}")
        print("Please ensure Docker is installed and running.")
    finally:
        import shutil
        shutil.rmtree(work_dir)


asyncio.run(run_codegen())
````

[<img alt="autogen-codegen.png" />](https://wandb.ai/parambharat/autogen-demo/weave/traces?view=traces_default\&peekPath=%2Fparambharat%2Fautogen-demo%2Fcalls%2F0196f173-21c2-7540-9dc7-fbab0b94ce0e%3FhideTraceTree%3D0)

## Learn more

Read the following guides to get a better understanding of Weave tracing and Ops.

* **Weave**:
  * [Tracing guide](/weave/guides/tracking/tracing)
  * [Op decorator](/weave/guides/tracking/ops)
* **AutoGen**:
  * [Official documentation](https://microsoft.github.io/autogen/stable//index.html)
  * [AutoGen GitHub](https://github.com/microsoft/autogen)

This guide provides a starting point for integrating Weave with AutoGen. Explore the Weave UI to see detailed traces of your agent interactions, model calls, and tool usage.


# Claude Code
Source: https://docs.wandb.ai/weave/guides/integrations/claude_code

Use W&B Weave to automatically trace and inspect your Claude Code sessions, including every conversation turn, tool call, and subagent interaction.

The `weave-claude-plugin` is a plugin for the [Claude Code](https://docs.anthropic.com/en/docs/claude-code/overview) command-line interface (CLI) that automatically captures traces from your Claude Code sessions and sends them to W\&B Weave. Every conversation turn, tool call, and subagent interaction is recorded without changes to your code or workflow. This is useful for debugging agent behavior, auditing session history, and monitoring Claude Code usage across your team.

## Prerequisites

Before installing the plugin, make sure you have the following:

* [Claude Code](https://docs.anthropic.com/en/docs/claude-code/overview) installed.
* [Node.js](https://nodejs.org/) and `npm` installed.
* A W\&B account with a valid API key. To get your API key, see [API keys](/platform/app/settings-page/user-settings#api-keys).

## W\&B Dedicated Cloud and Self-Managed instances

If your organization uses W\&B Dedicated Cloud or W\&B Self-Managed, the plugin requires your deployment's base URL to send traces. You must set your deployment's base URL before launching Claude Code by doing one of the following:

* Set the `WANDB_BASE_URL` environment variable to your deployment's base URL:

```bash lines theme={null}
export WANDB_BASE_URL=https://your-instance.wandb.io
```

* Run the `wandb login` command with the `host` flag to persist the base URL across sessions. This writes the host URL to `$HOME/.config/wandb/settings`, which the Weave client reads automatically without requiring the environment variable to be set.

```bash lines theme={null}
wandb login --host https://your-instance.wandb.io
```

If you need to change `WANDB_BASE_URL` or the `host` flag after the plugin's daemon is already running, restart the daemon for the change to take effect:

```bash lines theme={null}
printf '{"command":"shutdown"}' | nc -U -w1 ~/.weave_claude_plugin/daemon.sock
```

Then restart Claude Code. The daemon starts with the updated base URL.

## Install the plugin

Install the CLI globally, then run the installer once:

```bash lines theme={null}
npm install -g weave-claude-plugin
weave-claude-plugin install
```

The installer creates `~/.weave_claude_plugin/settings.json`, registers the plugin with Claude Code, and prompts you for your Weave project name and W\&B API key. If `WEAVE_PROJECT` and `WANDB_API_KEY` are set in your environment before you run `install`, the installer uses them and you can skip entering those values at the prompts. When you're done, launch or restart Claude Code. The plugin starts a background daemon when the Claude Code CLI starts, and sends traces to Weave.

For CI, bootstrap scripts, or other automation where you must not be prompted, set the `--non-interactive` flag. Set your environment variables first so the installer can authenticate with W\&B and configure your Weave project, then run the installer. The installer configures the plugin and registers Claude Code the same way. It uses the environment variables when they are set and warns if either is missing:

```bash lines theme={null}
export WEAVE_PROJECT=[YOUR-TEAM-NAME/YOUR-PROJECT-NAME]
export WANDB_API_KEY=[YOUR-API-KEY]
weave-claude-plugin install --non-interactive
```

Replace `[YOUR-TEAM-NAME/YOUR-PROJECT-NAME]` and `[YOUR-API-KEY]` with your own values.

## Data disclosure

The plugin sends Claude Code session data to Weave. That data can include sensitive content, such as:

* User prompts and model responses.
* Tool inputs and tool outputs.
* File paths and file contents read by Claude Code tools.
* Shell commands and shell output.
* Fetched URLs and content from fetched pages.

If Claude Code touches secrets, credentials, proprietary source code, personal data, or other confidential material during a session, that information may be logged to Weave as part of the trace.

<Note>
  PII scrubbing and sensitive-data redaction aren't implemented in the current plugin. If you can't send this class of data to Weave under your security or compliance requirements, don't install or enable the plugin.
</Note>

## View traces in Weave

After you install the plugin and have Claude Code running, open the **Traces** tab of your Weave project to view your traces. Each Claude Code session appears as a top-level trace containing individual turns and tool calls as child entries. Each trace includes token usage, model name, tool inputs and outputs, timing, and the textual content associated with prompts and responses.

```text theme={null}
claude_code.session
  └─ claude_code.turn                          (One per user message.)
       ├─ claude_code.tool.*                   (Each tool call such as Read, Bash, or Grep.)
       │    └─ claude_code.permission_request  (if user approval was needed.)
       └─ claude_code.subagent                 (If Claude spawned a subagent.)
            └─ claude_code.tool.*
```

## Check status and logs

Use these commands to confirm the plugin is running and to troubleshoot issues. Check status to verify that the daemon is active and Claude Code is registered. Check logs if traces aren't appearing in Weave or if `status` reports that action is needed.

From a terminal, run:

```bash lines theme={null}
weave-claude-plugin status
```

Each line in the output is marked with one of the following labels:

* `OK`: Everything required for tracing is set up and running.
* `Action needed`: Something is misconfigured or missing. Check the daemon logs as follows for more information.
* `Not yet active`: Not an error. The plugin hasn't recorded any activity yet. Start or continue a Claude Code session, wait a moment, then re-run the command.

If sessions aren't appearing in Weave, you can check the daemon log to help you debug any issues:

```bash lines theme={null}
weave-claude-plugin logs
```

To follow the log in real time:

```bash lines theme={null}
weave-claude-plugin logs --follow
```

You can also open the log file directly at `~/.weave_claude_plugin/logs/daemon.log`.

## Skills in Claude Code

In addition to automatic tracing, the plugin provides skills you can invoke from within a Claude Code session. With the plugin installed, the following skills are available inside any Claude Code session as slash commands. To avoid conflicts with Claude Code's built-in skills, the skills use the `weave:weave-*` naming convention.

### `/weave:weave-install`

Walks through installation and configuration interactively. Use it on a fresh machine or when diagnosing a broken setup: Claude checks for the CLI, runs the installer, prompts for missing values, and verifies that everything works.

```text theme={null}
/weave:weave-install
```

### `/weave:weave-status`

Reports current plugin status and explains what to fix. It corresponds to running `weave-claude-plugin status`, with Claude interpreting the output for you.

```text theme={null}
/weave:weave-status
```

### `/weave:weave-config`

Read or update plugin configuration without leaving Claude Code.

```text theme={null}
# Show current config.
/weave:weave-config

# Set a value directly.
/weave:weave-config set weave_project [YOUR-TEAM-NAME/YOUR-PROJECT-NAME]
/weave:weave-config set wandb_api_key [YOUR-API-KEY]
```

## Uninstall

To remove the plugin and its configuration:

```bash lines theme={null}
weave-claude-plugin uninstall
```

This removes the configuration directory and stops the daemon.


# CrewAI
Source: https://docs.wandb.ai/weave/guides/integrations/crewai

Integrate CrewAI with Weave for monitoring and tracing multi-agent applications

<a href="https://github.com/wandb/examples/blob/master/weave/docs/quickstart_crewai.ipynb">
  <img alt="Open In Colab" />
</a>

This guide shows you how to use Weave to monitor and trace [CrewAI](https://docs.crewai.com/introduction) multi-agent applications, including both [Crews](https://docs.crewai.com/guides/crews/first-crew) and [Flows](https://docs.crewai.com/guides/flows/first-flow).

CrewAI is a Python framework for building autonomous AI agents. It's independent of LangChain and other agent frameworks, and supports two abstractions: high-level [Crews](https://docs.crewai.com/guides/crews/first-crew) and low-level [Flows](https://docs.crewai.com/guides/flows/first-flow).

CrewAI applications often consist of multiple agents working together, which makes it important to understand how they collaborate and communicate. Weave automatically captures traces for your CrewAI applications so you can monitor and analyze your agents' performance and interactions.

The following sections walk through tracing a Crew, tracking tool usage, tracing a Flow, and wrapping a guardrail function as a Weave op.

## Get started with Crew

To run this example, install CrewAI and Weave. For more information about CrewAI installation, see the [CrewAI installation guide](https://docs.crewai.com/installation).

```bash theme={null}
pip install crewai weave
```

The following example creates a CrewAI Crew and traces the execution with Weave. To enable tracing, call `weave.init()` at the beginning of your script. The argument to `weave.init()` is the project name where Weave logs traces.

```python lines {5} theme={null}
import weave
from crewai import Agent, Task, Crew, LLM, Process

# Initialize Weave with your project name
weave.init(project_name="crewai_demo")

# Create an LLM with a temperature of 0 to ensure deterministic outputs
llm = LLM(model="gpt-4o-mini", temperature=0)

# Create agents
researcher = Agent(
    role='Research Analyst',
    goal='Find and analyze the best investment opportunities',
    backstory='Expert in financial analysis and market research',
    llm=llm,
    verbose=True,
    allow_delegation=False,
)

writer = Agent(
    role='Report Writer',
    goal='Write clear and concise investment reports',
    backstory='Experienced in creating detailed financial reports',
    llm=llm,
    verbose=True,
    allow_delegation=False,
)

# Create tasks
research_task = Task(
    description='Deep research on the {topic}',
    expected_output='Comprehensive market data including key players, market size, and growth trends.',
    agent=researcher
)

writing_task = Task(
    description='Write a detailed report based on the research',
    expected_output='The report should be easy to read and understand. Use bullet points where applicable.',
    agent=writer
)

# Create a crew
crew = Crew(
    agents=[researcher, writer],
    tasks=[research_task, writing_task],
    verbose=True,
    process=Process.sequential,
)

# Run the crew
result = crew.kickoff(inputs={"topic": "AI in material science"})
print(result)
```

Weave tracks and logs all calls made through the CrewAI library, including agent interactions, task executions, and LLM calls. You can view the traces in the Weave web interface.

[<img alt="crew_trace.png" />](https://wandb.ai/ayut/crewai_demo/weave/traces?filter=%7B%22opVersionRefs%22%3A%5B%22weave%3A%2F%2F%2Fayut%2Fcrewai_demo%2Fop%2Fcrewai.Crew.kickoff%3A*%22%5D%7D\&peekPath=%2Fayut%2Fcrewai_demo%2Fcalls%2F0195c7ac-bd52-7390-95a7-309370e9e058%3FhideTraceTree%3D0\&cols=%7B%22wb_run_id%22%3Afalse%2C%22attributes.weave.client_version%22%3Afalse%2C%22attributes.weave.os_name%22%3Afalse%2C%22attributes.weave.os_release%22%3Afalse%2C%22attributes.weave.os_version%22%3Afalse%2C%22attributes.weave.source%22%3Afalse%2C%22attributes.weave.sys_version%22%3Afalse%7D)

<Note>
  CrewAI provides several methods for better control over the kickoff process: `kickoff()`, `kickoff_for_each()`, `kickoff_async()`, and `kickoff_for_each_async()`. The integration supports logging traces from all these methods.
</Note>

## Track tools

CrewAI tools give agents capabilities like web searching, data analysis, collaboration, and delegating tasks among coworkers. The integration traces them as well.

The following example improves the quality of the generated report from the previous example by giving the agent access to a tool that can search the internet and return the most relevant results.

First, install the extra dependency:

```bash theme={null}
pip install 'crewai[tools]'
```

This example uses the `SerperDevTool` to enable the 'Research Analyst' agent to search relevant information on the internet. For more information about this tool and its API requirements, see the [SerperDevTool documentation](https://docs.crewai.com/tools/serperdevtool).

```python lines {12} theme={null}
# .... existing imports ....
from crewai_tools import SerperDevTool

# We provide the agent with the tool.
researcher = Agent(
    role='Research Analyst',
    goal='Find and analyze the best investment opportunities',
    backstory='Expert in financial analysis and market research',
    llm=llm,
    verbose=True,
    allow_delegation=False,
    tools=[SerperDevTool()],
)

# .... existing code ....
```

Running this Crew with an agent that has internet access produces a more relevant result. Weave automatically traces the tool usage, as shown in the following image.

[<img alt="crew_with_tool_trace.png" />](https://wandb.ai/ayut/crewai_demo/weave/traces?filter=%7B%22opVersionRefs%22%3A%5B%22weave%3A%2F%2F%2Fayut%2Fcrewai_demo%2Fop%2Fcrewai.Crew.kickoff%3A*%22%5D%7D\&peekPath=%2Fayut%2Fcrewai_demo%2Fcalls%2F0195c7c7-0213-7f42-b130-caa93a79316c%3FdescendentCallId%3D0195c7c7-0a16-7f11-8cfd-9dedf1d03b3b\&cols=%7B%22wb_run_id%22%3Afalse%2C%22attributes.weave.client_version%22%3Afalse%2C%22attributes.weave.os_name%22%3Afalse%2C%22attributes.weave.os_release%22%3Afalse%2C%22attributes.weave.os_version%22%3Afalse%2C%22attributes.weave.source%22%3Afalse%2C%22attributes.weave.sys_version%22%3Afalse%7D)

<Note>
  The integration automatically patches all the tools available in the [`crewAI-tools`](https://github.com/crewAIInc/crewAI-tools) repository.
</Note>

## Get started with Flow

The following example defines a CrewAI Flow and traces it with Weave. As with Crews, call `weave.init()` before defining the Flow so that Weave automatically captures the `Flow.kickoff` entry point and the `@start`, `@listen`, `@router`, `@or_`, and `@and_` decorators.

```python lines {3} theme={null}
import weave
# Initialize Weave with your project name
weave.init("crewai_demo")

from crewai.flow.flow import Flow, listen, router, start
from litellm import completion


class CustomerFeedbackFlow(Flow):
    model = "gpt-4o-mini"

    @start()
    def fetch_feedback(self):
        print("Fetching customer feedback")
        # In a real-world scenario, this could be replaced by an API call.
        # For this example, we simulate customer feedback.
        feedback = (
            "I had a terrible experience with the product. "
            "It broke after one use and customer service was unhelpful."
        )
        self.state["feedback"] = feedback
        return feedback

    @router(fetch_feedback)
    def analyze_feedback(self, feedback):
        # Use the language model to analyze sentiment
        prompt = (
            f"Analyze the sentiment of this customer feedback and "
            "return only 'positive' or 'negative':\n\n"
            f"Feedback: {feedback}"
        )
        response = completion(
            model=self.model,
            messages=[{"role": "user", "content": prompt}],
        )
        sentiment = response["choices"][0]["message"]["content"].strip().lower()
        # If the response is ambiguous, default to negative
        if sentiment not in ["positive", "negative"]:
            sentiment = "negative"
        return sentiment

    @listen("positive")
    def handle_positive_feedback(self):
        # Generate a thank you message for positive feedback
        prompt = "Generate a thank you message for a customer who provided positive feedback."
        response = completion(
            model=self.model,
            messages=[{"role": "user", "content": prompt}],
        )
        thank_you_message = response["choices"][0]["message"]["content"].strip()
        self.state["response"] = thank_you_message
        return thank_you_message

    @listen("negative")
    def handle_negative_feedback(self):
        # Generate an apology message with a promise to improve service for negative feedback
        prompt = (
            "Generate an apology message to a customer who provided negative feedback and offer assistance or a solution."
        )
        response = completion(
            model=self.model,
            messages=[{"role": "user", "content": prompt}],
        )
        apology_message = response["choices"][0]["message"]["content"].strip()
        self.state["response"] = apology_message
        return apology_message

# Instantiate and kickoff the flow
flow = CustomerFeedbackFlow()
result = flow.kickoff()
print(result)
```

[<img alt="flow.png" />](https://wandb.ai/ayut/crewai_demo/weave/traces?filter=%7B%22opVersionRefs%22%3A%5B%22weave%3A%2F%2F%2Fayut%2Fcrewai_demo%2Fop%2Fcrewai.Flow.kickoff%3A*%22%5D%7D\&peekPath=%2Fayut%2Fcrewai_demo%2Fcalls%2F0195c7e3-7a63-7283-bef4-9e0eb2f0eab1\&cols=%7B%22wb_run_id%22%3Afalse%2C%22attributes.weave.client_version%22%3Afalse%2C%22attributes.weave.os_name%22%3Afalse%2C%22attributes.weave.os_release%22%3Afalse%2C%22attributes.weave.os_version%22%3Afalse%2C%22attributes.weave.source%22%3Afalse%2C%22attributes.weave.sys_version%22%3Afalse%7D)

<Note>
  The integration automatically patches the `Flow.kickoff` entry point and all the available decorators (`@start`, `@listen`, `@router`, `@or_`, and `@and_`).
</Note>

## Crew guardrail: track your own ops

Task guardrails let you validate and transform task outputs before CrewAI passes them to the next task. You can use a Python function to validate the agent's execution on the fly.

Wrapping the guardrail function with `@weave.op` captures its inputs, outputs, and app logic so you can debug how data is validated through your agents. It also automatically versions the code as you experiment, capturing ad-hoc details that haven't been committed to git.

The following example extends the research analyst and writer Crew by adding a guardrail that validates the length of the generated report.

```python lines {4,36} theme={null}
# .... existing imports and weave initialization ....

# Decorate your guardrail function with `@weave.op()`
@weave.op(name="guardrail-validate_blog_content")
def validate_blog_content(result: TaskOutput) -> Tuple[bool, Any]:
    # Get raw string result
    result = result.raw

    """Validate blog content meets requirements."""
    try:
        # Check word count
        word_count = len(result.split())

        if word_count > 200:
            return (False, {
                "error": "Blog content exceeds 200 words",
                "code": "WORD_COUNT_ERROR",
                "context": {"word_count": word_count}
            })

        # Additional validation logic here
        return (True, result.strip())
    except Exception as e:
        return (False, {
            "error": "Unexpected error during validation",
            "code": "SYSTEM_ERROR"
        })


# .... existing agents and research analyst task ....

writing_task = Task(
    description='Write a detailed report based on the research under 200 words',
    expected_output='The report should be easy to read and understand. Use bullet points where applicable.',
    agent=writer,
    guardrail=validate_blog_content,
)

# .... existing code to run crew ....
```

By decorating the guardrail function with `@weave.op`, you can track the input and output to this function along with execution time, token information if the function uses an LLM, code version, and more.

[<img alt="guardrail.png" />](https://wandb.ai/ayut/crewai_demo/weave/traces?filter=%7B%22opVersionRefs%22%3A%5B%22weave%3A%2F%2F%2Fayut%2Fcrewai_demo%2Fop%2Fcrewai.Crew.kickoff%3A*%22%5D%7D\&peekPath=%2Fayut%2Fcrewai_demo%2Fcalls%2F0195c838-38cb-71a2-8a15-651ecddf9d89%3FdescendentCallId%3D0195c838-8632-7173-846d-f230e7272c20\&cols=%7B%22wb_run_id%22%3Afalse%2C%22attributes.weave.client_version%22%3Afalse%2C%22attributes.weave.os_name%22%3Afalse%2C%22attributes.weave.os_release%22%3Afalse%2C%22attributes.weave.os_version%22%3Afalse%2C%22attributes.weave.source%22%3Afalse%2C%22attributes.weave.sys_version%22%3Afalse%7D)

## Conclusion

You now have a Weave-traced CrewAI application that captures agent interactions, tool usage, Flow execution, and guardrail validations. To suggest improvements or report problems with this integration, [open an issue on GitHub](https://github.com/wandb/weave/issues/new/choose).

To learn more about building multi-agent systems with CrewAI, see the CrewAI [examples](https://github.com/crewAIInc/crewAI-examples) and [documentation](https://docs.crewai.com/introduction).


# DSPy
Source: https://docs.wandb.ai/weave/guides/integrations/dspy

Use Weave to automatically track and log calls made using DSPy modules and functions

<div>
  <ColabLink />

  <GitHubLink />
</div>

[DSPy](https://dspy.ai) is a framework for algorithmically optimizing language model (LM) prompts and weights, especially when you use LMs one or more times within a pipeline. W\&B Weave automatically tracks and logs calls made with DSPy modules and functions. This guide shows you how to enable Weave tracing for DSPy programs, track custom modules and signatures, and capture traces from DSPy optimizers and evaluations. With these traces, you can debug, analyze, and improve your DSPy applications.

## Traces

This section shows you how to enable automatic tracing of DSPy calls in Weave. Store traces of language model applications in a central location, both during development and in production. These traces are useful for debugging and as a dataset that helps you improve your application.

Weave automatically captures traces for [DSPy](https://dspy.ai). To start tracking, call `weave.init(project_name="[YOUR-WANDB-PROJECT-NAME]")` and use the library as normal. Replace `[YOUR-OPENAI-API-KEY]` with your OpenAI API key and `[YOUR-WANDB-PROJECT-NAME]` with your W\&B project name.

```python lines theme={null}
import os
import dspy
import weave

os.environ["OPENAI_API_KEY"] = "[YOUR-OPENAI-API-KEY]"

weave.init(project_name="[YOUR-WANDB-PROJECT-NAME]")

lm = dspy.LM('openai/gpt-4o-mini')
dspy.configure(lm=lm)
classify = dspy.Predict("sentence -> sentiment")
classify(sentence="it's a charming and often affecting journey.")
```

[<img alt="DSPy trace in Weave showing LM call inputs, outputs, and metadata" />](https://wandb.ai/geekyrakshit/dspy-project/weave/calls)

With tracing enabled, Weave logs every LM call your DSPy program makes to your Weave project, where you can inspect inputs, outputs, and metadata.

## Track your own DSPy modules and signatures

Beyond built-in calls, Weave also traces the custom modules and signatures you define. A `Module` is the building block with learnable parameters for DSPy programs that abstracts a prompting technique. A `Signature` is a declarative specification of input/output behavior of a DSPy `Module`. Weave automatically tracks all built-in and custom `Signature` and `Module` objects in your DSPy programs. Replace `[YOUR-OPENAI-API-KEY]` with your OpenAI API key and `[YOUR-WANDB-PROJECT-NAME]` with your W\&B project name.

```python lines theme={null}
import os
import dspy
import weave

os.environ["OPENAI_API_KEY"] = "[YOUR-OPENAI-API-KEY]"

weave.init(project_name="[YOUR-WANDB-PROJECT-NAME]")

class Outline(dspy.Signature):
    """Outline a thorough overview of a topic."""

    topic: str = dspy.InputField()
    title: str = dspy.OutputField()
    sections: list[str] = dspy.OutputField()
    section_subheadings: dict[str, list[str]] = dspy.OutputField(
        desc="mapping from section headings to subheadings"
    )


class DraftSection(dspy.Signature):
    """Draft a top-level section of an article."""

    topic: str = dspy.InputField()
    section_heading: str = dspy.InputField()
    section_subheadings: list[str] = dspy.InputField()
    content: str = dspy.OutputField(desc="markdown-formatted section")


class DraftArticle(dspy.Module):
    def __init__(self):
        self.build_outline = dspy.ChainOfThought(Outline)
        self.draft_section = dspy.ChainOfThought(DraftSection)

    def forward(self, topic):
        outline = self.build_outline(topic=topic)
        sections = []
        for heading, subheadings in outline.section_subheadings.items():
            section, subheadings = (
                f"## {heading}",
                [f"### {subheading}" for subheading in subheadings],
            )
            section = self.draft_section(
                topic=outline.title,
                section_heading=section,
                section_subheadings=subheadings,
            )
            sections.append(section.content)
        return dspy.Prediction(title=outline.title, sections=sections)


draft_article = DraftArticle()
article = draft_article(topic="World Cup 2002")
```

[<img alt="DSPy custom module trace in Weave with module execution flow and trace details" />](https://wandb.ai/geekyrakshit/dspy-project/weave/calls)

## Optimize and evaluate your DSPy program

Weave also captures traces for DSPy optimizers and evaluation calls, which you can use to improve and evaluate your DSPy program's performance on a development set. Replace `[YOUR-OPENAI-API-KEY]` with your OpenAI API key and `[YOUR-WANDB-PROJECT-NAME]` with your W\&B project name.

```python lines theme={null}
import os
import dspy
import weave

os.environ["OPENAI_API_KEY"] = "[YOUR-OPENAI-API-KEY]"
weave.init(project_name="[YOUR-WANDB-PROJECT-NAME]")

def accuracy_metric(answer, output, trace=None):
    predicted_answer = output["answer"].lower()
    return answer["answer"].lower() == predicted_answer

module = dspy.ChainOfThought("question -> answer: str, explanation: str")
optimizer = dspy.BootstrapFewShot(metric=accuracy_metric)
optimized_module = optimizer.compile(
    module, trainset=SAMPLE_EVAL_DATASET, valset=SAMPLE_EVAL_DATASET
)
```

[<img alt="DSPy optimizer trace in Weave with optimization process and performance improvements" />](https://wandb.ai/geekyrakshit/dspy-project/weave/calls)


# Google Agent Development Kit (ADK)
Source: https://docs.wandb.ai/weave/guides/integrations/google_adk

Trace Google Agent Development Kit (ADK) agent and tool calls in Weave using OpenTelemetry

You can trace [Google Agent Development Kit (ADK)](https://google.github.io/adk-docs/) agent and tool calls in Weave using [OpenTelemetry (OTEL)](https://opentelemetry.io/). ADK is a flexible and modular framework for developing and deploying AI agents. Although optimized for Gemini and the Google ecosystem, ADK is model independent and deployment independent. It provides tools for creating, deploying, and orchestrating agentic architectures that range from simple tasks to complex workflows.

This guide is for developers building agents with ADK who want end-to-end observability of agent reasoning, tool calls, and multi-agent workflows. It explains how to trace ADK agent and tool calls using OTEL, and visualize those traces in Weave. You'll learn how to install the required dependencies, configure an OTEL tracer to send data to Weave, and instrument your ADK agents and tools. With traces in Weave, you can debug agent behavior, monitor performance, and inspect how data flows through your agents and tools.

<Tip>
  For more information about OTEL tracing in Weave, see [Send OTEL Traces to Weave](../tracking/otel).
</Tip>

## Prerequisites

1. Install the required dependencies:

   ```bash theme={null}
   pip install google-adk opentelemetry-sdk opentelemetry-exporter-otlp-proto-http
   ```

2. Set your [Google API key](https://cloud.google.com/docs/authentication/api-keys) as an environment variable:

   ```bash theme={null}
   export GOOGLE_API_KEY=[YOUR-API-KEY]
   ```

3. [Configure OTEL tracing in Weave](#configure-otel-tracing-in-weave).

### Configure OTEL tracing in Weave

To send traces from ADK to Weave, configure OTEL with a `TracerProvider` and an `OTLPSpanExporter`. Set the exporter to the [correct endpoint and HTTP headers for authentication and project identification](#required-configuration).

<Warning>
  Store sensitive environment variables like your API key and project information in an environment file (for example, `.env`), and load them using `os.environ`. This keeps your credentials secure and out of your codebase.
</Warning>

#### Required configuration

* **Endpoint:** `https://trace.wandb.ai/otel/v1/traces`. If you're using a dedicated Weave instance, the URL follows this pattern instead: `[YOUR-WEAVE-HOST]/traces/otel/v1/traces`.
* **Headers:**
  * `Authorization`: Basic auth using your W\&B API key.
  * `project_id`: Your W\&B entity/project name (for example, `myteam/myproject`).

## Send OTEL traces from ADK to Weave

With the prerequisites in place, you can set up the OTEL exporter and tracer provider that forward span data to Weave. The following code snippet demonstrates how to configure an OTLP span exporter and tracer provider to send OTEL traces from an ADK application to Weave.

<Warning>
  To ensure that Weave traces ADK properly, set the global tracer provider *before* using ADK components in your code.
</Warning>

```python lines theme={null}
import base64
import os
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk import trace as trace_sdk
from opentelemetry.sdk.trace.export import SimpleSpanProcessor
from opentelemetry import trace

# Load sensitive values from environment variables
WANDB_BASE_URL = "https://trace.wandb.ai"
# Your W&B entity/project name e.g. "myteam/myproject"
PROJECT_ID = os.environ.get("WANDB_PROJECT_ID")  
# Create a W&B API key at https://wandb.ai/settings
WANDB_API_KEY = os.environ.get("WANDB_API_KEY")  

OTEL_EXPORTER_OTLP_ENDPOINT = f"{WANDB_BASE_URL}/otel/v1/traces"
AUTH = base64.b64encode(f"api:{WANDB_API_KEY}".encode()).decode()

OTEL_EXPORTER_OTLP_HEADERS = {
    "Authorization": f"Basic {AUTH}",
    "project_id": PROJECT_ID,
}

# Create the OTLP span exporter with endpoint and headers
exporter = OTLPSpanExporter(
    endpoint=OTEL_EXPORTER_OTLP_ENDPOINT,
    headers=OTEL_EXPORTER_OTLP_HEADERS,
)

# Create a tracer provider and add the exporter
tracer_provider = trace_sdk.TracerProvider()
tracer_provider.add_span_processor(SimpleSpanProcessor(exporter))

# Set the global tracer provider BEFORE importing/using ADK
trace.set_tracer_provider(tracer_provider)
```

## Trace ADK agents with OTEL

After you set up the tracer provider, you can create and run ADK agents with automatic tracing. The following example demonstrates how to create an LLM agent with a tool, and run it with an in-memory runner:

```python lines theme={null}
from google.adk.agents import LlmAgent
from google.adk.runners import InMemoryRunner
from google.adk.tools import FunctionTool
from google.genai import types
import asyncio

# Define a simple tool for demonstration
def calculator(a: float, b: float) -> str:
    """Add two numbers and return the result.

    Args:
        a: First number
        b: Second number

    Returns:
        The sum of a and b
    """
    return str(a + b)

calculator_tool = FunctionTool(func=calculator)

async def run_agent():
    # Create an LLM agent
    agent = LlmAgent(
        name="MathAgent",
        model="gemini-2.0-flash",  # You can change this to another model if needed
        instruction=(
            "You are a helpful assistant that can do math. "
            "When asked a math problem, use the calculator tool to solve it."
        ),
        tools=[calculator_tool],
    )

    # Set up runner
    runner = InMemoryRunner(agent=agent, app_name="math_assistant")
    session_service = runner.session_service

    # Create a session
    user_id = "example_user"
    session_id = "example_session"
    await session_service.create_session(
        app_name="math_assistant",
        user_id=user_id,
        session_id=session_id,
    )

    # Run the agent with a message that should trigger tool use
    async for event in runner.run_async(
        user_id=user_id,
        session_id=session_id,
        new_message=types.Content(
            role="user", parts=[types.Part(text="What is 5 + 7?")]
        ),
    ):
        if event.is_final_response() and event.content:
            print(f"Final response: {event.content.parts[0].text.strip()}")

# Run the async function
asyncio.run(run_agent())
```

All agent operations are automatically traced and sent to Weave, so you can visualize the execution flow. You can view model calls, reasoning steps, and tool invocations.

<Frame>
  <img alt="A trace visualization of an ADK agent" />
</Frame>

## Trace ADK tools with OTEL

This section shows how tool invocations appear in your traces when an agent uses more than one tool. When you define and use tools with ADK, the trace also captures these tool calls. The OTEL integration automatically instruments both the agent's reasoning process and the individual tool executions, so you can see how your agent behaves end to end.

Here's an example with multiple tools:

```python lines theme={null}
from google.adk.agents import LlmAgent
from google.adk.runners import InMemoryRunner
from google.adk.tools import FunctionTool
from google.genai import types
import asyncio

# Define multiple tools
def add(a: float, b: float) -> str:
    """Add two numbers.
    
    Args:
        a: First number
        b: Second number
        
    Returns:
        The sum of a and b
    """
    return str(a + b)

def multiply(a: float, b: float) -> str:
    """Multiply two numbers.
    
    Args:
        a: First number
        b: Second number
        
    Returns:
        The product of a and b
    """
    return str(a * b)

# Create function tools
add_tool = FunctionTool(func=add)
multiply_tool = FunctionTool(func=multiply)

async def run_agent():
    # Create an LLM agent with multiple tools
    agent = LlmAgent(
        name="MathAgent",
        model="gemini-2.0-flash",
        instruction=(
            "You are a helpful assistant that can do math operations. "
            "When asked to add numbers, use the add tool. "
            "When asked to multiply numbers, use the multiply tool."
        ),
        tools=[add_tool, multiply_tool],
    )

    # Set up runner
    runner = InMemoryRunner(agent=agent, app_name="math_assistant")
    session_service = runner.session_service

    # Create a session
    user_id = "example_user"
    session_id = "example_session"
    await session_service.create_session(
        app_name="math_assistant",
        user_id=user_id,
        session_id=session_id,
    )

    # Run the agent with a message that should trigger tool use
    async for event in runner.run_async(
        user_id=user_id,
        session_id=session_id,
        new_message=types.Content(
            role="user", parts=[types.Part(text="First add 5 and 7, then multiply the result by 2.")]
        ),
    ):
        if event.is_final_response() and event.content:
            print(f"Final response: {event.content.parts[0].text.strip()}")

# Run the async function
asyncio.run(run_agent())
```

<Frame>
  <img alt="A trace visualization of ADK tool calls" />
</Frame>

## Work with workflow agents

Beyond single-agent setups, ADK supports composing multiple agents into workflows. The same OTEL configuration traces those flows without additional setup. ADK provides [*workflow agents*](https://google.github.io/adk-docs/agents/workflow-agents/) for more complex scenarios. You can trace workflow agents just like regular LLM agents. The following example uses a [`SequentialAgent`](https://google.github.io/adk-docs/agents/workflow-agents/sequential-agents/):

```python lines theme={null}
from google.adk.agents import LlmAgent, SequentialAgent
from google.adk.runners import InMemoryRunner
from google.genai import types
import asyncio

async def run_workflow():
    # Create two LLM agents
    summarizer = LlmAgent(
        name="Summarizer",
        model="gemini-2.0-flash",
        instruction="Summarize the given text in one sentence.",
        description="Summarizes text in one sentence",
        output_key="summary"  # Store output in state['summary']
    )
    
    analyzer = LlmAgent(
        name="Analyzer",
        model="gemini-2.0-flash",
        instruction="Analyze the sentiment of the given text as positive, negative, or neutral. The text to analyze: {summary}",
        description="Analyzes sentiment of text",
        output_key="sentiment"  # Store output in state['sentiment']
    )
    
    # Create a sequential workflow
    workflow = SequentialAgent(
        name="TextProcessor",
        sub_agents=[summarizer, analyzer],
        description="Executes a sequence of summarization followed by sentiment analysis.",
    )
    
    # Set up runner
    runner = InMemoryRunner(agent=workflow, app_name="text_processor")
    session_service = runner.session_service
    
    # Create a session
    user_id = "example_user"
    session_id = "example_session"
    await session_service.create_session(
        app_name="text_processor",
        user_id=user_id,
        session_id=session_id,
    )
    
    # Run the workflow
    async for event in runner.run_async(
        user_id=user_id,
        session_id=session_id,
        new_message=types.Content(
            role="user", 
            parts=[types.Part(text="The product exceeded my expectations. It worked perfectly right out of the box, and the customer service was excellent when I had questions about setup.")]
        ),
    ):
        if event.is_final_response() and event.content:
            print(f"Final response: {event.content.parts[0].text.strip()}")

# Run the async function
asyncio.run(run_workflow())
```

This workflow agent trace shows the sequential execution of both agents in Weave, which provides visibility into how data flows through your multi-agent system.

<Frame>
  <img alt="A trace visualization of a Sequential workflow agent" />
</Frame>

## Learn more

* [Weave documentation: Send OTEL traces to Weave](../tracking/otel)
* [Official ADK documentation](https://google.github.io/adk-docs/)
* [Official OTEL documentation](https://opentelemetry.io/)
* [ADK GitHub repository](https://github.com/google/adk-python)


# Haystack
Source: https://docs.wandb.ai/weave/guides/integrations/haystack

Trace Deepset Haystack pipelines with W&B Weave using the WeaveConnector integration.

[Haystack](https://haystack.deepset.ai/) is an open-source framework for building search and LLM applications. Deepset maintains a WeaveConnector component that forwards Haystack pipeline traces to W\&B Weave so you can inspect component runs, prompts, and outputs in the Weave UI.

This guide is for Haystack developers who want to add observability to their pipelines.

For full API details and additional examples, see these Deepset resources:

* Haystack's [WeaveConnector](https://docs.haystack.deepset.ai/docs/weaveconnector).
* Haystack's [Weave integration API reference](https://docs.haystack.deepset.ai/reference/integrations-weave).
* Haystack's [Trace with W\&B Weave](https://docs.cloud.deepset.ai/docs/use-weights-and-biases) example using a RAG pipeline.

## Prerequisites

Before you begin, you must complete the following:

* Set `WANDB_API_KEY` in your environment using your W\&B [API key](https://wandb.ai/settings). This authenticates the connector with W\&B Weave.
* Set `HAYSTACK_CONTENT_TRACING_ENABLED` to `true` before you run a pipeline so Haystack emits tracing data the connector can forward.

## Install

Install the required dependencies using `pip`:

```bash lines theme={null}
pip install weave-haystack
```

The package declares compatible versions of `haystack-ai` and `weave` as dependencies.

## Trace a Haystack pipeline with Weave

The following example adds Haystack's `WeaveConnector` to a Haystack [`Pipeline`](https://docs.haystack.deepset.ai/docs/pipelines) and integrates with W\&B Weave to trace and monitor your pipeline components. The `pipeline_name` you pass is used as the Weave project name for traces from that pipeline.

In your Haystack pipeline, don't connect `WeaveConnector` to other components.

```python lines theme={null}
import os

os.environ["HAYSTACK_CONTENT_TRACING_ENABLED"] = "true"

from haystack import Pipeline
from haystack.components.builders import ChatPromptBuilder
from haystack.components.generators.chat import OpenAIChatGenerator
from haystack.dataclasses import ChatMessage
from haystack_integrations.components.connectors.weave import WeaveConnector

pipe = Pipeline()
pipe.add_component("prompt_builder", ChatPromptBuilder())
pipe.add_component("llm", OpenAIChatGenerator(model="gpt-3.5-turbo"))
pipe.connect("prompt_builder.prompt", "llm.messages")

# pipeline_name becomes your W&B project name.
connector = WeaveConnector(pipeline_name="haystack_demo")
# Add connector to pipeline but don't connect it.
pipe.add_component("weave", connector)

messages = [
    ChatMessage.from_system(
        "Always respond in German even if some input data is in other languages.",
    ),
    ChatMessage.from_user("Tell me about {{location}}"),
]

response = pipe.run(
    data={
        "prompt_builder": {
            "template_variables": {"location": "Berlin"},
            "template": messages,
        },
    },
)

print(response["llm"]["replies"][0])
```

After the pipeline runs, open your W\&B workspace, select the project named with `pipeline_name`, and go to **Traces** to review the completed trace.


# Instructor
Source: https://docs.wandb.ai/weave/guides/integrations/instructor

Trace structured data extraction from LLMs with Weave's Instructor integration, capturing Pydantic validation and retry logic.

<a href="https://colab.research.google.com/github/wandb/examples/blob/master/weave/docs/quickstart_instructor.ipynb" aria-label="Open in Google Colab">
  <img alt="Open In Colab" />
</a>

[Instructor](https://python.useinstructor.com/) is a lightweight library for getting structured data like JSON from LLMs. This guide shows you how to trace Instructor calls with Weave so you can debug structured extraction, capture Pydantic validation, and inspect retry logic.

## Tracing

It's important to store traces of language model applications in a central location, both during development and in production. These traces can be useful for debugging, and as a dataset that will help you improve your application.

Weave automatically captures traces for [Instructor](https://python.useinstructor.com/). To start tracking, call `weave.init(project_name="[YOUR-WANDB-PROJECT-NAME]")` and use the library as normal.

```python lines theme={null}
import instructor
import weave
from pydantic import BaseModel
from openai import OpenAI


# Define your desired output structure
class UserInfo(BaseModel):
    user_name: str
    age: int

# Initialize Weave
weave.init(project_name="instructor-test")

# Patch the OpenAI client
client = instructor.from_openai(OpenAI())

# Extract structured data from natural language
user_info = client.chat.completions.create(
    model="gpt-3.5-turbo",
    response_model=UserInfo,
    messages=[{"role": "user", "content": "John Doe is 30 years old."}],
)
```

| <img alt="Instructor LM trace in Weave with structured output extraction workflow" />                          |
| -------------------------------------------------------------------------------------------------------------- |
| Weave tracks and logs all LLM calls made using Instructor. You can view the traces in the Weave web interface. |

## Track your own ops

Wrapping a function with `@weave.op` starts capturing inputs, outputs, and app logic so you can debug how data flows through your app. You can deeply nest ops and build a tree of functions that you want to track. This also starts automatically versioning code as you experiment, capturing ad hoc details that haven't been committed to Git.

Create a function decorated with [`@weave.op`](/weave/guides/tracking/ops).

In the following example, the `extract_person` function is the metric function wrapped with `@weave.op`. This lets you see intermediate steps, such as the OpenAI chat completion call.

```python lines theme={null}
import instructor
import weave
from openai import OpenAI
from pydantic import BaseModel


# Define your desired output structure
class Person(BaseModel):
    person_name: str
    age: int


# Initialize Weave
weave.init(project_name="instructor-test")

# Patch the OpenAI client
lm_client = instructor.from_openai(OpenAI())


# Extract structured data from natural language
@weave.op()
def extract_person(text: str) -> Person:
    return lm_client.chat.completions.create(
        model="gpt-3.5-turbo",
        messages=[
            {"role": "user", "content": text},
        ],
        response_model=Person,
    )


person = extract_person("My name is John and I am 20 years old")
```

| <img alt="Instructor op trace with structured objects, function inputs, outputs, and Pydantic model validation" />                                                                                                                     |
| -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Decorating the `extract_person` function with `@weave.op` traces its inputs, outputs, and all internal LM calls made inside the function. Weave also automatically tracks and versions the structured objects generated by Instructor. |

## Create a `Model` for easier experimentation

Organizing experimentation is difficult when there are many moving pieces. By using the [`Model`](../core-types/models) class, you can capture and organize the experimental details of your app like your system prompt or the model you're using. This helps organize and compare different iterations of your app.

In addition to versioning code and capturing inputs and outputs, [`Model`](../core-types/models)s capture structured parameters that control your application's behavior, so you can find which parameters worked best. You can also use Weave Models with [`serve`](#serve-a-weave-model) and [`Evaluation`](../core-types/evaluations)s.

In the following example, you can experiment with `PersonExtractor`. Every time you change one of these, you'll get a new *version* of `PersonExtractor`.

```python lines theme={null}
import asyncio
from typing import List, Iterable

import instructor
import weave
from openai import AsyncOpenAI
from pydantic import BaseModel


# Define your desired output structure
class Person(BaseModel):
    person_name: str
    age: int


# Initialize Weave
weave.init(project_name="instructor-test")

# Patch the OpenAI client
lm_client = instructor.from_openai(AsyncOpenAI())


class PersonExtractor(weave.Model):
    openai_model: str
    max_retries: int

    @weave.op()
    async def predict(self, text: str) -> List[Person]:
        model = await lm_client.chat.completions.create(
            model=self.openai_model,
            response_model=Iterable[Person],
            max_retries=self.max_retries,
            stream=True,
            messages=[
                {
                    "role": "system",
                    "content": "You are a perfect entity extraction system",
                },
                {
                    "role": "user",
                    "content": f"Extract `{text}`",
                },
            ],
        )
        return [m async for m in model]


model = PersonExtractor(openai_model="gpt-4", max_retries=2)
asyncio.run(model.predict("John is 30 years old"))
```

| <img alt="Instructor Weave Model tracing and versioning interface with model versions and trace history" /> |
| ----------------------------------------------------------------------------------------------------------- |
| Tracing and versioning your calls using a [`Model`](../core-types/models)                                   |

## Serve a Weave Model

After you save a `weave.Model`, you can serve it as a FastAPI endpoint to test it from outside the notebook or integrate it with other applications. Given a Weave reference to a `weave.Model` object, you can start a FastAPI server and [`serve`](https://docs.wandb.ai/weave/guides/tools/serve) it.

| [<img alt="Instructor serve interface with FastAPI server configuration and model serving options" />](https://wandb.ai/geekyrakshit/instructor-test/weave/objects/PersonExtractor/versions/xXpMsJvaiTOjKafz1TnHC8wMgH5ZAAwYOaBMvHuLArI) |
| ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| You can find the weave reference of any `weave.Model` by navigating to the model and copying it from the UI.                                                                                                                             |

To serve your model, run the following command in your terminal:

```bash theme={null}
weave serve weave://your_entity/project-name/YourModel:[HASH]
```


# Koog
Source: https://docs.wandb.ai/weave/guides/integrations/koog

Trace Kotlin-based Koog agents with Weave using OpenTelemetry to capture tool calls, prompts, and completions for AI applications.

[Koog](https://docs.koog.ai/) is a Kotlin-based framework for building single-run and complex workflow agents. Koog includes built-in OpenTelemetry (OTEL) support and can export traces directly to Weave, giving you visibility into prompts, completions, tool calls, and end-to-end agent execution.

This guide shows you how to configure Koog's Weave exporter so your Kotlin agents forward OpenTelemetry spans to a Weave project. With the exporter enabled, you can debug agent runs, analyze performance, and iterate faster. This guide is for Kotlin developers who are already building agents with Koog and want to add Weave tracing.

<Frame>
  <img alt="Koog" />
</Frame>

## Prerequisites

You must set the following environment variables before running your agent so the Weave exporter can authenticate and route traces to the correct project:

Replace `[YOUR-API-KEY]` with your Weave API key and `[YOUR-ENTITY]` with your W\&B team or entity name. The `WEAVE_PROJECT_NAME` value can be any project name. Weave creates the project on first use.

```bash theme={null}
export WEAVE_API_KEY="[YOUR-API-KEY]"
export WEAVE_ENTITY="[YOUR-ENTITY]"
export WEAVE_PROJECT_NAME="koog-tracing"
```

## Install Koog (Gradle)

With your environment variables in place, add Koog to your Kotlin project so you can configure the agent and its exporter (Kotlin DSL shown):

```kotlin theme={null}
dependencies {
    implementation("ai.koog:koog-agents:LATEST_VERSION")
}
```

See Koog's documentation for [additional installation information](https://docs.koog.ai/).

## Enable Weave export (OpenTelemetry)

After you install Koog, install Koog's OpenTelemetry feature and add the Weave exporter. This maps Koog spans to Weave traces using Weave's OpenTelemetry endpoint.

The following example shows how to use the `addWeaveExporter`:

```kotlin theme={null}
fun main() = runBlocking {
    val apiKey = "api-key"
    val entity = System.getenv()["WEAVE_ENTITY"] ?: throw IllegalArgumentException("WEAVE_ENTITY is not set")
    val projectName = System.getenv()["WEAVE_PROJECT_NAME"] ?: "koog-tracing"

    val agent = AIAgent(
        executor = simpleOpenAIExecutor(apiKey),
        llmModel = OpenAIModels.CostOptimized.GPT4oMini,
        systemPrompt = "You are a code assistant. Provide concise code examples."
    ) {
        install(OpenTelemetry) {
            addWeaveExporter()
        }
    }

    println("Running agent with Weave tracing")

    val result = agent.run("""
        Create a Python function to calculate fibonacci numbers efficiently,
        include error handling, type hints, and unit tests.
        Verify the implementation works for n=50.
    """)

    println("Result: $result\nSee traces on https://wandb.ai/$entity/$projectName/weave/traces")
}
```

The function automatically reads your Weave environment variables, but you can also configure [specific parameters](https://api.koog.ai/agents/agents-features/agents-features-opentelemetry/ai.koog.agents.features.opentelemetry.integration.weave/add-weave-exporter.html?query=fun%20OpenTelemetryConfig.addWeaveExporter\(weaveOtelBaseUrl:%20String?%20=%20null,%20weaveEntity:%20String?%20=%20null,%20weaveProjectName:%20String?%20=%20null,%20weaveApiKey:%20String?%20=%20null,%20timeout:%20Duration%20=%2010.seconds\)) for the exporter, as in the following example:

```kotlin theme={null}
install(OpenTelemetry) {
    addWeaveExporter(
        weaveOtelBaseUrl = "https://trace.wandb.ai",
        weaveEntity = System.getenv()["WEAVE_ENTITY"],
        weaveProjectName = System.getenv()["WEAVE_PROJECT_NAME"],
        weaveApiKey = System.getenv()["WEAVE_API_KEY"],
        timeout = 10.seconds
    )
}
```

The preceding example:

* Uses `weaveEntity` and `weaveProjectName` to route traces to a specific team and project.
* Sets `weaveOtelBaseUrl` to your trace endpoint (for example, `https://[YOUR-SUBDOMAIN].wandb.io/[PATH]`). Use the parameter for dedicated Weave instances.

If you're new to using Koog with Weave, review the following documentation:

* [Koog's Weave exporter guide](https://docs.koog.ai/opentelemetry-weave-exporter/) for additional information about the exporter
* [Koog's OpenTelemetry support guide](https://docs.koog.ai/opentelemetry-support/) for core concepts about how OpenTelemetry works with Koog
* [Weave OTEL docs](../tracking/otel) for information about how Weave ingests OTLP data

## What gets traced

After you run an agent with the exporter enabled, Koog forwards traces to your Weave project. Koog's Weave exporter captures the same spans as Koog's general OTEL integration, including:

* Agent lifecycle events (start, stop, errors)
* LLM interactions (prompts, completions, token usage, latency)
* Tool and API calls (function calls and external requests)
* System context (model name, Koog version, environment metadata)

You can visualize these traces in the Weave UI to understand performance and quality. See [Weave's tracing overview](../tracking/tracing) for an introduction to capturing traces with Weave.

## Example notebook

See Koog's documentation for a [runnable notebook](https://docs.koog.ai/examples/Weave/) that streams traces to Weave.

## Troubleshooting

If your Koog agent isn't exporting traces as expected, check the following:

* If traces are missing, first verify that `WEAVE_API_KEY`, `WEAVE_ENTITY`, and `WEAVE_PROJECT_NAME` are set correctly.
* Confirm your environment can reach `https://trace.wandb.ai` and that the exporter is configured as shown in the preceding examples.
* For additional troubleshooting and sampling guidance, see Koog's [OpenTelemetry support](https://docs.koog.ai/opentelemetry-support/).


# LangChain
Source: https://docs.wandb.ai/weave/guides/integrations/langchain

Use Weave to track and log all calls made through the LangChain Python library

<a href="https://colab.research.google.com/github/wandb/examples/blob/master/weave/docs/quickstart_langchain.ipynb" aria-label="Open in Google Colab">
  <img alt="Open In Colab" />
</a>

Weave tracks and logs calls made through the [LangChain Python library](https://github.com/langchain-ai/langchain).

When working with LLMs, debugging is part of the job. Whether a model call fails, an output is misformatted, or nested model calls create confusion, pinpointing issues can be hard. LangChain applications often consist of multiple steps and LLM call invocations, so it helps to understand the inner workings of your chains and agents.

Weave captures traces for your [LangChain](https://docs.langchain.com/oss/python/langchain/overview) applications automatically. This lets you monitor and analyze your application's performance, which makes it easier to debug and optimize your LLM workflows.

This guide is for developers building LangChain applications who want to add tracing, evaluation, and observability with Weave. It walks through enabling automatic tracing, attaching metadata, controlling tracing manually, and wrapping LangChain chains as Weave models for evaluation.

## Getting started

To get started, call `weave.init()` at the beginning of your script. The argument to `weave.init()` is a project name that Weave uses to organize your traces.

```python lines {6} theme={null}
import weave
from langchain_core.prompts import PromptTemplate
from langchain_openai import ChatOpenAI

# Initialize Weave with your project name
weave.init("langchain_demo")

llm = ChatOpenAI()
prompt = PromptTemplate.from_template("1 + {number} = ")

llm_chain = prompt | llm

output = llm_chain.invoke({"number": 2})

print(output)
```

## Track call metadata

Custom metadata helps you filter and analyze traces in the Weave UI. To track metadata from your LangChain calls, use the [`weave.attributes`](/weave/reference/python-sdk#function-attributes) context manager. This context manager lets you set custom metadata for a specific block of code, such as a chain or a single request.

```python lines {6,13} theme={null}
import weave
from langchain_core.prompts import PromptTemplate
from langchain_openai import ChatOpenAI

# Initialize Weave with your project name
weave.init("langchain_demo")

llm = ChatOpenAI()
prompt = PromptTemplate.from_template("1 + {number} = ")

llm_chain = prompt | llm

with weave.attributes({"my_awesome_attribute": "value"}):
    output = llm_chain.invoke()

print(output)
```

Weave automatically tracks the metadata against the trace of the LangChain call. You can view the metadata in the Weave web interface.

[<img alt="Weave UI showing LangChain trace metadata attributes" />](https://wandb.ai/parambharat/langchain_demo/weave/traces?cols=%7B%22attributes.weave.client_version%22%3Afalse%2C%22attributes.weave.os_name%22%3Afalse%2C%22attributes.weave.os_release%22%3Afalse%2C%22attributes.weave.os_version%22%3Afalse%2C%22attributes.weave.source%22%3Afalse%2C%22attributes.weave.sys_version%22%3Afalse%7D)

## Traces

Storing traces of LLM applications in a central database helps during both development and production. These traces give you a dataset you can use to debug and improve your application.

Weave captures traces for your LangChain applications automatically. Weave tracks and logs calls made through the LangChain library, including prompt templates, chains, LLM calls, tools, and agent steps. You can view the traces in the Weave web interface.

[<img alt="Weave UI showing a LangChain trace" />](https://wandb.ai/parambharat/langchain_demo/weave/calls)

## Trace calls manually

Besides automatic tracing, you can manually trace calls using the `WeaveTracer` callback or the `weave_tracing_enabled` context manager. These methods resemble using request callbacks in individual parts of a LangChain application. Use them when you want to trace specific chains or invocations rather than your whole application.

The following sections describe each approach.

Note: Weave traces LangChain `Runnables` by default, and this is enabled when you call `weave.init()`. You can disable this behavior by setting the environment variable `WEAVE_TRACE_LANGCHAIN` to `"false"` before calling `weave.init()`. This lets you control the tracing behavior of specific chains or even individual requests in your application.

### Use `WeaveTracer`

You can pass the `WeaveTracer` callback to individual LangChain components to trace specific requests.

```python lines {11,13,15,22} theme={null}
import os

os.environ["WEAVE_TRACE_LANGCHAIN"] = "false" # <- explicitly disable global tracing.

from weave.integrations.langchain import WeaveTracer
from langchain_core.prompts import PromptTemplate
from langchain_openai import ChatOpenAI
import weave

# Initialize Weave with your project name
weave.init("langchain_demo")  # <-- we don't enable tracing here because the env var is explicitly set to `false`

weave_tracer = WeaveTracer()

config = {"callbacks": [weave_tracer]}

llm = ChatOpenAI()
prompt = PromptTemplate.from_template("1 + {number} = ")

llm_chain = prompt | llm

output = llm_chain.invoke({"number": 2}, config=config) # <-- this enables tracing only for this chain invoke.

llm_chain.invoke({"number": 4})  # <-- this will not have tracing enabled for langchain calls but openai calls will still be traced
```

### Use the `weave_tracing_enabled` context manager

Alternatively, you can use the `weave_tracing_enabled` context manager to enable tracing for specific blocks of code.

```python lines {11,18} theme={null}
import os

os.environ["WEAVE_TRACE_LANGCHAIN"] = "false" # <- explicitly disable global tracing.

from weave.integrations.langchain import weave_tracing_enabled
from langchain_core.prompts import PromptTemplate
from langchain_openai import ChatOpenAI
import weave

# Initialize Weave with your project name
weave.init("langchain_demo")  # <-- we don't enable tracing here because the env var is explicitly set to `false`

llm = ChatOpenAI()
prompt = PromptTemplate.from_template("1 + {number} = ")

llm_chain = prompt | llm

with weave_tracing_enabled():  # <-- this enables tracing only for this chain invoke.
    output = llm_chain.invoke({"number": 2})


llm_chain.invoke({"number": 4})  # <-- this will not have tracing enabled for langchain calls but openai calls will still be traced
```

## Configuration

When you call `weave.init()`, Weave enables tracing by setting the environment variable `WEAVE_TRACE_LANGCHAIN` to `"true"`. This lets Weave automatically capture traces for your LangChain applications. To disable this behavior, set the environment variable to `"false"`.

## Relation to LangChain callbacks

This section explains how Weave's tracing integrates with LangChain's callback system, so you can choose the approach that best fits your application.

### Auto logging

The automatic logging provided by `weave.init()` is similar to passing a constructor callback to every component in a LangChain application. This means Weave tracks all interactions globally across your entire application, including prompt templates, chains, LLM calls, tools, and agent steps.

### Manual logging

The manual logging methods (`WeaveTracer` and `weave_tracing_enabled`) are similar to using request callbacks in individual parts of a LangChain application. These methods give you finer control over which parts of your application Weave traces:

* **Constructor callbacks:** Apply to the entire chain or component and log all interactions consistently.
* **Request callbacks:** Apply to specific requests and provide detailed tracing of particular invocations.

When you integrate Weave with LangChain, you get logging and monitoring of your LLM applications, which makes debugging and performance tuning easier.

For more information, see the [LangChain documentation](https://python.langchain.com/v0.2/docs/how_to/debugging/#tracing).

## Models and evaluations

Organizing and evaluating LLMs across different use cases gets harder as you add components like prompts, model configurations, and inference parameters. With [`weave.Model`](/weave/guides/core-types/models), you can capture and organize experimental details like system prompts or the models you use, which makes it easier to compare iterations.

The following sections show how to wrap a LangChain chain as a `weave.Model` and then evaluate it.

The following example demonstrates wrapping a LangChain chain in a `WeaveModel`:

```python lines {10,12,16} theme={null}
import json
import asyncio

import weave

from langchain_core.prompts import PromptTemplate
from langchain_openai import ChatOpenAI

# Initialize Weave with your project name
weave.init("langchain_demo")

class ExtractFruitsModel(weave.Model):
    model_name: str
    prompt_template: str

    @weave.op()
    async def predict(self, sentence: str) -> dict:
        llm = ChatOpenAI(model=self.model_name, temperature=0.0)
        prompt = PromptTemplate.from_template(self.prompt_template)

        llm_chain = prompt | llm
        response = llm_chain.invoke({"sentence": sentence})
        result = response.content

        if result is None:
            raise ValueError("No response from model")
        parsed = json.loads(result)
        return parsed

model = ExtractFruitsModel(
    model_name="gpt-3.5-turbo-1106",
    prompt_template='Extract fields ("fruit": <str>, "color": <str>, "flavor": <str>) from the following text, as json: {sentence}',
)
sentence = "There are many fruits that were found on the recently discovered planet Goocrux. There are neoskizzles that grow there, which are purple and taste like candy."

prediction = asyncio.run(model.predict(sentence))

# if you're in a Jupyter Notebook, run:
# prediction = await model.predict(sentence)

print(prediction)
```

This code creates a model that you can visualize in the Weave UI:

[<img alt="Weave UI showing a LangChain chain wrapped as a Weave Model" />](https://wandb.ai/parambharat/langchain_demo/weave/object-versions?filter=%7B%22baseObjectClass%22%3A%22Model%22%7D\&peekPath=%2Fparambharat%2Flangchain_demo%2Fobjects%2FExtractFruitsModel%2Fversions%2FBeoL6WuCH8wgjy6HfmuBMyKzArETg1oAFpYaXZSq1hw%3F%26)

You can also use Weave Models with `serve`, and [`Evaluations`](/weave/guides/core-types/evaluations).

### Evaluations

Evaluations help you measure the performance of your models. The [`weave.Evaluation`](/weave/guides/core-types/evaluations) class captures how well your model performs on specific tasks or datasets. This makes it easier to compare different models and iterations of your application. The following example demonstrates how to evaluate the preceding model:

```python lines theme={null}

from weave.scorers import MultiTaskBinaryClassificationF1

sentences = [
    "There are many fruits that were found on the recently discovered planet Goocrux. There are neoskizzles that grow there, which are purple and taste like candy.",
    "Pounits are a bright green color and are more savory than sweet.",
    "Finally, there are fruits called glowls, which have a very sour and bitter taste which is acidic and caustic, and a pale orange tinge to them.",
]
labels = [
    {"fruit": "neoskizzles", "color": "purple", "flavor": "candy"},
    {"fruit": "pounits", "color": "bright green", "flavor": "savory"},
    {"fruit": "glowls", "color": "pale orange", "flavor": "sour and bitter"},
]
examples = [
    {"id": "0", "sentence": sentences[0], "target": labels[0]},
    {"id": "1", "sentence": sentences[1], "target": labels[1]},
    {"id": "2", "sentence": sentences[2], "target": labels[2]},
]

@weave.op()
def fruit_name_score(target: dict, output: dict) -> dict:
    return {"correct": target["fruit"] == output["fruit"]}


evaluation = weave.Evaluation(
    dataset=examples,
    scorers=[
        MultiTaskBinaryClassificationF1(class_names=["fruit", "color", "flavor"]),
        fruit_name_score,
    ],
)
scores = asyncio.run(evaluation.evaluate(model)))
# if you're in a Jupyter Notebook, run:
# scores = await evaluation.evaluate(model)

print(scores)
```

This code generates an evaluation trace that you can visualize in the Weave UI:

[<img alt="Weave UI showing a LangChain evaluation trace" />](https://wandb.ai/parambharat/langchain_demo/weave/calls?filter=%7B%22traceRootsOnly%22%3Atrue%7D\&peekPath=%2Fparambharat%2Flangchain_demo%2Fcalls%2F44c3f26c-d9d3-423e-b434-651ea5174be3)

## Known issues

**Tracing async calls**: A bug in the implementation of the `AsyncCallbackManager` in LangChain causes async calls to not be traced in the correct order. Weave has filed a [PR](https://github.com/langchain-ai/langchain/pull/23909) to fix this. As a result, the order of calls in the trace may not be accurate when you use the `ainvoke`, `astream`, and `abatch` methods in LangChain `Runnables`.


# LlamaIndex
Source: https://docs.wandb.ai/weave/guides/integrations/llamaindex

Trace and debug LlamaIndex applications with Weave, capturing LLM calls, RAG pipelines, agent steps, and evaluations automatically.

This guide shows you how to use Weave to trace, debug, and evaluate [LlamaIndex](https://docs.llamaindex.ai/en/stable/) applications. By following along, you learn how Weave automatically captures calls made through the [LlamaIndex Python library](https://github.com/run-llama/llama_index) so you can monitor RAG pipelines, agent steps, and LLM calls without writing custom logging code. This guide is for developers building LLM applications with LlamaIndex who want visibility into their workflows for debugging, performance analysis, and evaluation.

When you work with LLMs, debugging is inevitable. Whether a model call fails, an output is misformatted, or nested model calls create confusion, pinpointing issues can be challenging. LlamaIndex applications often consist of multiple steps and LLM call invocations, making it crucial to understand the inner workings of your chains and agents.

Weave streamlines this process by automatically capturing traces for your LlamaIndex applications. This lets you monitor and analyze your application's performance, helping you debug and optimize your LLM workflows. Weave also helps with your evaluation workflows.

## Get started

To get started, call `weave.init()` at the beginning of your script. This initializes Weave and starts capturing traces for any LlamaIndex calls that follow. The argument in `weave.init()` is a project name that helps you organize your traces.

```python lines {5} theme={null}
import weave
from llama_index.core.chat_engine import SimpleChatEngine

# Initialize Weave with your project name
weave.init("llamaindex_demo")

chat_engine = SimpleChatEngine.from_defaults()
response = chat_engine.chat(
    "Say something profound and romantic about fourth of July"
)
print(response)
```

The previous example creates a basic LlamaIndex chat engine which, internally, makes an OpenAI call. After you run this code, Weave captures a trace of the chat engine's execution that you can inspect in the Weave web interface. See the following trace:

[<img alt="simple_llamaindex.png" />](https://wandb.ai/wandbot/test-llamaindex-weave/weave/calls/b6b5d898-2df8-4e14-b553-66ce84661e74)

## Traces

This section explains how Weave captures multi-step LlamaIndex workflows like RAG pipelines.

LlamaIndex is known for its ease of connecting data with LLMs. A basic RAG application requires an embedding step, a retrieval step, and a response synthesis step. As complexity grows, it becomes important to store traces of individual steps in a central database during both development and production.

These traces are essential for debugging and improving your application. Weave automatically tracks all calls made through the LlamaIndex library, including prompt templates, LLM calls, tools, and agent steps. You can view the traces in the Weave web interface.

The following example shows a basic RAG pipeline from LlamaIndex's [Starter Tutorial (OpenAI)](https://docs.llamaindex.ai/en/stable/getting_started/starter_example/):

```python lines {5} theme={null}
import weave
from llama_index.core import VectorStoreIndex, SimpleDirectoryReader

# Initialize Weave with your project name
weave.init("llamaindex_demo")

# Assuming you have a `.txt` file in the `data` directory
documents = SimpleDirectoryReader("data").load_data()
index = VectorStoreIndex.from_documents(documents)

query_engine = index.as_query_engine()
response = query_engine.query("What did the author do growing up?")
print(response)
```

The trace timeline captures not only the "events" but also the execution time, cost, and token counts where applicable. Drill down into the trace to see the inputs and outputs of each step.

[<img alt="llamaindex_rag.png" />](https://wandb.ai/wandbot/test-llamaindex-weave/weave/calls?filter=%7B%22traceRootsOnly%22%3Atrue%7D\&peekPath=%2Fwandbot%2Ftest-llamaindex-weave%2Fcalls%2F6ac53407-1bb7-4c38-b5a3-c302bd877a11%3Ftracetree%3D1)

## One-click observability

This section explains how the Weave integration plugs into LlamaIndex's built-in observability system, so you don't have to configure handlers manually.

LlamaIndex provides [one-click observability](https://docs.llamaindex.ai/en/stable/module_guides/observability/) to let you build principled LLM applications in a production setting.

The Weave integration uses this capability of LlamaIndex and automatically sets [`WeaveCallbackHandler()`](https://github.com/wandb/weave/blob/master/weave/integrations/llamaindex/llamaindex.py) to `llama_index.core.global_handler`. As a user of LlamaIndex and Weave, you only need to initialize a Weave run with `weave.init([NAME_OF_PROJECT])`.

## Create a `Model` for easier experimentation

Organizing and evaluating LLMs in applications for various use cases is challenging when you have multiple components such as prompts, model configurations, and inference parameters. With [`weave.Model`](/weave/guides/core-types/models), you can capture and organize experimental details like system prompts or the models you use, which helps you compare different iterations.

The following example shows how to build a LlamaIndex query engine in a `WeaveModel`, using data that you can find in the [weave/data](https://github.com/wandb/weave/tree/master/data) folder:

```python lines {16,52,61} theme={null}
import weave

from llama_index.core import VectorStoreIndex, SimpleDirectoryReader
from llama_index.core.node_parser import SentenceSplitter
from llama_index.llms.openai import OpenAI
from llama_index.core import PromptTemplate


PROMPT_TEMPLATE = """
You are given with relevant information about Paul Graham. Answer the user query only based on the information provided. Don't make up stuff.

User Query: {query_str}
Context: {context_str}
Answer:
"""

class SimpleRAGPipeline(weave.Model):
    chat_llm: str = "gpt-4"
    temperature: float = 0.1
    similarity_top_k: int = 2
    chunk_size: int = 256
    chunk_overlap: int = 20
    prompt_template: str = PROMPT_TEMPLATE

    def get_llm(self):
        return OpenAI(temperature=self.temperature, model=self.chat_llm)

    def get_template(self):
        return PromptTemplate(self.prompt_template)

    def load_documents_and_chunk(self, data):
        documents = SimpleDirectoryReader(data).load_data()
        splitter = SentenceSplitter(
            chunk_size=self.chunk_size,
            chunk_overlap=self.chunk_overlap,
        )
        nodes = splitter.get_nodes_from_documents(documents)
        return nodes

    def get_query_engine(self, data):
        nodes = self.load_documents_and_chunk(data)
        index = VectorStoreIndex(nodes)

        llm = self.get_llm()
        prompt_template = self.get_template()

        return index.as_query_engine(
            similarity_top_k=self.similarity_top_k,
            llm=llm,
            text_qa_template=prompt_template,
        )

    @weave.op()
    def predict(self, query: str):
        query_engine = self.get_query_engine(
            # This data can be found in the weave repo under data/paul_graham
            "data/paul_graham",
        )
        response = query_engine.query(query)
        return {"response": response.response}

weave.init("test-llamaindex-weave")

rag_pipeline = SimpleRAGPipeline()
response = rag_pipeline.predict("What did the author do growing up?")
print(response)
```

The `SimpleRAGPipeline` class subclassed from `weave.Model` organizes the important parameters for this RAG pipeline. Decorating the `query` method with `weave.op()` enables tracing. With this structure in place, you can now version, compare, and evaluate different configurations of your RAG pipeline in Weave.

[<img alt="llamaindex_model.png" />](https://wandb.ai/wandbot/test-llamaindex-weave/weave/calls?filter=%7B%22traceRootsOnly%22%3Atrue%7D\&peekPath=%2Fwandbot%2Ftest-llamaindex-weave%2Fcalls%2Fa82afbf4-29a5-43cd-8c51-603350abeafd%3Ftracetree%3D1)

## Evaluate with `weave.Evaluation`

This section shows how to measure your model's performance on a fixed dataset so you can compare iterations quantitatively.

Evaluations help you measure the performance of your applications. With the [`weave.Evaluation`](/weave/guides/core-types/evaluations) class, you can capture how well your model performs on specific tasks or datasets, which helps you compare different models and iterations of your application. The following example shows how to evaluate the model created in the previous section:

```python lines {25,32,36} theme={null}
import asyncio
from llama_index.core.evaluation import CorrectnessEvaluator

eval_examples = [
    {
        "id": "0",
        "query": "What programming language did Paul Graham learn to teach himself AI when he was in college?",
        "ground_truth": "Paul Graham learned Lisp to teach himself AI when he was in college.",
    },
    {
        "id": "1",
        "query": "What was the name of the startup Paul Graham co-founded that was eventually acquired by Yahoo?",
        "ground_truth": "The startup Paul Graham co-founded that was eventually acquired by Yahoo was called Viaweb.",
    },
    {
        "id": "2",
        "query": "What is the capital city of France?",
        "ground_truth": "I cannot answer this question because no information was provided in the text.",
    },
]

llm_judge = OpenAI(model="gpt-4", temperature=0.0)
evaluator = CorrectnessEvaluator(llm=llm_judge)

@weave.op()
def correctness_evaluator(query: str, ground_truth: str, output: dict):
    result = evaluator.evaluate(
        query=query, reference=ground_truth, response=output["response"]
    )
    return {"correctness": float(result.score)}

evaluation = weave.Evaluation(dataset=eval_examples, scorers=[correctness_evaluator])

rag_pipeline = SimpleRAGPipeline()

asyncio.run(evaluation.evaluate(rag_pipeline))
```

This evaluation builds on the example in the previous section. Evaluating with `weave.Evaluation` requires an evaluation dataset, a scorer function, and a `weave.Model`. These requirements apply to the three key components:

* The keys of the evaluation sample dicts must match the arguments of the scorer function and of the `weave.Model`'s `predict` method.
* The `weave.Model` must have a method named `predict`, `infer`, or `forward`. You must decorate this method with `weave.op()` for tracing.
* The scorer function must be decorated with `weave.op()` and must have `output` as a named argument.

[<img alt="llamaindex_evaluation.png" />](https://wandb.ai/wandbot/llamaindex-weave/weave/calls?filter=%7B%22opVersionRefs%22%3A%5B%22weave%3A%2F%2F%2Fwandbot%2Fllamaindex-weave%2Fop%2FEvaluation.predict_and_score%3ANmwfShfFmgAhDGLXrF6Xn02T9MIAsCXBUcifCjyKpOM%22%5D%2C%22parentId%22%3A%2233491e66-b580-47fa-9d43-0cd6f1dc572a%22%7D\&peekPath=%2Fwandbot%2Fllamaindex-weave%2Fcalls%2F33491e66-b580-47fa-9d43-0cd6f1dc572a%3Ftracetree%3D1)

By integrating Weave with LlamaIndex, you can ensure comprehensive logging and monitoring of your LLM applications, which streamlines debugging and performance optimization through evaluation.


# Local Models
Source: https://docs.wandb.ai/weave/guides/integrations/local_models

Run open source models locally with Ollama, LMStudio, or GPT4All and trace them with Weave using OpenAI SDK compatibility.

Many developers run open source models like Llama 3, Mixtral, Gemma, and Phi on their own hardware. Weave supports several local model runners by default, as long as they offer OpenAI SDK compatibility.

This guide shows you how to trace calls to locally hosted models with Weave so you can capture inputs, outputs, and metadata in the same way you would for hosted LLM providers. It's intended for developers who run open source models on their own machine and want observability for those calls.

## Wrap local model functions with `@weave.op()`

If your local model isn't accessed through an OpenAI-compatible runner, you can still capture traces by wrapping your own model-calling functions. Integrate Weave with any LLM by initializing Weave with `weave.init('<your-project-name>')` and then wrapping the calls to your LLMs with `weave.op()`. For more details, see the [tracing guide](/weave/guides/tracking/tracing).

## Update your OpenAI SDK code to use local models

If your local model runner supports the OpenAI SDK, you can point the existing OpenAI client at it with two changes.

The main change is the `base_url` parameter during the `openai.OpenAI()` initialization. This tells the OpenAI SDK to send requests to your local server instead of the OpenAI hosted API.

```python lines theme={null}
client = openai.OpenAI(
    base_url="http://localhost:1234",
)
```

For local models, the `api_key` can be any string, but you must override it. Otherwise, the OpenAI SDK reads it from environment variables and shows an error.

## Supported local model runners

The following runners let you download and run models from Hugging Face on your computer. Each runner exposes an OpenAI-compatible endpoint that you can point the OpenAI SDK at using the preceding changes.

* Nomic [GPT4All](https://www.nomic.ai/gpt4all) - support through Local Server in settings ([FAQ](https://docs.gpt4all.io/gpt4all_help/faq.html))
* [LMStudio](https://lmstudio.ai/) - Local Server OpenAI SDK support [docs](https://lmstudio.ai/docs/developer/core/server)
* [Ollama](https://ollama.com/) - [OpenAI compatibility](https://ollama.readthedocs.io/en/openai/) for OpenAI SDK
* [llama-cpp-python](https://llama-cpp-python.readthedocs.io/en/latest/server/) - Python package for running `llama.cpp` with OpenAI SDK support
* [llamafile](https://github.com/Mozilla-Ocho/llamafile#other-example-llamafiles) - `http://localhost:8080/v1` automatically supports the OpenAI SDK when you run Llamafile


# Model Context Protocol (MCP) and Weave
Source: https://docs.wandb.ai/weave/guides/integrations/mcp

Trace activity between your MCP client and MCP server with Weave

<a href="https://colab.research.google.com/drive/174VzXlU5Qcgvjt4OoIWN-guTxJcOefAh?usp=sharing" aria-label="Open in Google Colab">
  <img alt="Open In Colab" />
</a>

The Model Context Protocol (MCP) is a standardized communication protocol that enables AI applications to exchange information with large language models (LLMs). MCP provides an interface for LLMs to access data sources and interact with external tools, without requiring custom integrations for each new service.

The Weave integration lets you trace activity between your MCP client and MCP server. It gives you detailed visibility into tool calls, resource access, and prompt generation across MCP-based systems, so you can debug, audit, and optimize your MCP applications.

This guide explains how the integration works, how to enable tracing on the server and client sides, and walks through a full example you can run yourself.

## How it works

<Warning>
  The integration captures client-side and server-side operations separately, but doesn't provide end-to-end visibility into their interaction. An ongoing proposal aims to add OpenTelemetry trace support to MCP to enable end-to-end observability. For more information, see [GitHub discussion #269](https://github.com/modelcontextprotocol/modelcontextprotocol/discussions/269).
</Warning>

The Weave integration automatically traces key components of the Model Context Protocol (MCP) by patching core methods with the [`weave.op()`](../tracking/ops) decorator. Specifically, it patches methods in the [`mcp.server.fastmcp.FastMCP`](https://github.com/modelcontextprotocol/python-sdk/blob/b4c7db6a50a5c88bae1db5c1f7fba44d16eebc6e/src/mcp/server/fastmcp/server.py#L109) and [`mcp.ClientSession`](https://github.com/modelcontextprotocol/python-sdk/blob/b4c7db6a50a5c88bae1db5c1f7fba44d16eebc6e/src/mcp/client/session.py#L84) classes.

Through this integration, Weave traces the following MCP components:

* [Tools](https://modelcontextprotocol.io/specification/2025-06-18/server/tools)
* [Resources](https://modelcontextprotocol.io/specification/2025-06-18/server/resources)
* [Prompts](https://modelcontextprotocol.io/specification/2025-06-18/server/prompts)

[<img alt="mcp_trace_timeline.png" />](https://wandb.ai/ayut/mcp_example/weave/traces?filter=%7B%22opVersionRefs%22%3A%5B%22weave%3A%2F%2F%2Fayut%2Fmcp_example%2Fop%2Frun_client%3A*%22%5D%7D\&peekPath=%2Fayut%2Fmcp_example%2Fcalls%2F01966bbe-cc5e-7012-b45f-bf10617d8c1e%3FhideTraceTree%3D0)

## Use the integration

The Weave integration works with both the MCP server and client. Once installed, you can enable tracing with two additional lines of code: one to import `weave`, and another to initialize it.

### Prerequisites

Before you begin, install the required packages:

```bash theme={null}
pip install -qq "mcp[cli]" weave
```

### Configuration

Configure the MCP integration through the `MCP_TRACE_LIST_OPERATIONS` environment variable. Set it to `true` to trace list operations (`list_tools`, `list_resources`, and `list_prompts`) on both server and client sides.

### Server-side integration

Use this section if you're building or instrumenting an MCP server. To trace an MCP server, add two lines to your existing `FastMCP` setup: one to import Weave and one to initialize the client. Once added, Weave automatically traces tool, resource, and prompt operations.

```python lines theme={null}
# Import Weave (required for tracing)
import weave
from mcp.server.fastmcp import FastMCP

# Initialize Weave with your project name
weave_client = weave.init("my-project")

# Set up the MCP server
mcp = FastMCP("Demo")

# Define a tool (this call will be traced)
@mcp.tool()
def add(a: int, b: int) -> int:
    """Add two numbers."""
    return a + b

# Define a resource (this call will be traced)
@mcp.resource("greeting://{name}")
def get_greeting(name: str) -> str:
    """Return a personalized greeting."""
    return f"Hello, {name}!"

# Define a prompt (this call will be traced)
@mcp.prompt()
def review_code(code: str) -> str:
    """Return a prompt for reviewing code."""
    return f"Please review this code:\n\n{code}"

# Start the server
mcp.run(transport="stdio")
```

### Client-side integration

Use this section if you're building or instrumenting an MCP client. On the client side, tracing also requires two changes: import Weave and initialize it. Weave automatically traces all tool calls, resource accesses, and prompt requests.

```python lines theme={null}
# Import Weave (required for tracing)
import weave
from mcp import ClientSession, StdioServerParameters
from mcp.client.stdio import stdio_client

# Initialize Weave with your project name
weave_client = weave.init("my-project")

# Set up and run the MCP client
async with stdio_client(server_params) as (read, write):
    async with ClientSession(read, write) as session:
        # Initialize the session
        await session.initialize()
        
        # Call a tool (this will be traced)
        result = await session.call_tool("add", arguments={"a": 1, "b": 2})
        
        # Read a resource (this will be traced)
        resource = await session.read_resource("greeting://user")
        
        # Get a prompt (this will be traced)
        prompt = await session.get_prompt("review_code", arguments={"code": "print('Hello')"})
```

## Tutorial: `mcp_demo` example

The `mcp_demo` example demonstrates an integration between MCP and Weave for tracing. It showcases how to instrument both the client and server components to capture detailed traces of their interactions. By running through this code, you can see traces from both sides of an MCP application in the Weave UI and gain a concrete reference you can adapt to your own projects.

### Run the example

1. Clone the docs repository and navigate to the `mcp_demo` example:

   ```bash theme={null}
   git clone https://github.com/wandb/docs
   cd docs/weave/examples/mcp_demo
   ```

   The example includes two main files:

   * `example_server.py`: A demo MCP server built with `FastMCP`. It defines tools, resources, and prompts.
   * `example_client.py`: A client that connects to the server and interacts with its components.

2. Install the required dependencies manually:

   ```bash theme={null}
   pip install mcp[cli] weave
   ```

3. Run the demo:

   ```bash theme={null}
   python example_client.py example_server.py
   ```

   This command launches both the client and server. The client starts an interactive CLI where you can test various features.

### Client CLI commands

The client interface supports the following commands:

| Command                 | Description                                                                                                                                  |
| ----------------------- | -------------------------------------------------------------------------------------------------------------------------------------------- |
| `tools`                 | List available tools                                                                                                                         |
| `resources`             | List available resources                                                                                                                     |
| `prompts`               | List available prompts                                                                                                                       |
| `add <a> <b>`           | Add two numbers                                                                                                                              |
| `bmi <weight> <height>` | Calculate Body Mass Index                                                                                                                    |
| `weather <city>`        | Get weather data for a city                                                                                                                  |
| `greeting <name>`       | Get a personalized greeting                                                                                                                  |
| `user <id>`             | Retrieve a user profile                                                                                                                      |
| `config`                | Fetch app configuration                                                                                                                      |
| `code-review <code>`    | Generate a code review prompt                                                                                                                |
| `debug <error>`         | Generate a debugging prompt                                                                                                                  |
| `demo`                  | Run a full demo of all available features. Runs each feature in sequence and produces a full trace timeline of interactions in the Weave UI. |
| `q`                     | Quit the session                                                                                                                             |

### Example overview

The `example_server.py` server defines the following:

* *Tools*: Functions such as `add()`, `calculate_bmi()`, `fetch_weather()`
* *Resources*: Endpoints like `greeting://{name}`, `config://app`, `users://{id}/profile`
* *Prompts*: Templates like `review_code()` and `debug_error()`

Weave automatically traces all server-side operations when you initialize the client with `weave.init()`.

The `example_client.py` client demonstrates how to:

* Connect to an MCP server.
* Discover available tools, resources, and prompts.
* Call tools with parameters.
* Read from resource URIs.
* Generate prompts with arguments.
* Show usage of [`weave.op()`](../tracking/ops) with custom methods and functions.

Weave traces all client-side calls to provide a complete view of interactions between the client and server.

## FAQ

The following section answers common questions about why and how to use Weave's MCP tracing.

### Why MCP tracing is needed

As an LLM application developer, you fall into one of three categories:

* *MCP server-side developer*: You want to expose multiple tools, resources, and prompts to the MCP client. You expose your existing application's tools and resources, or you've built agents or have multiple agents orchestrated by an orchestrator agent.

* *MCP client-side developer*: You want to plug your client-side application into multiple MCP servers. A core part of your client-side logic is making LLM calls to decide which tool to call or which resource to fetch.

* *MCP server and client developer*: You're developing both the server and the client.

If you fall into either of the first two categories, you want to know when each tool is called, what the execution flow looks like, the token count, and latency of different components in your server or client-side logic.

If you're developing both the server and client, a unified trace timeline helps you iterate through both server and client-side logic.

In any case, an observability layer lets you:

* Iterate through your application.
* Audit the workflow or execution logic.
* Identify bottlenecks.


# PydanticAI
Source: https://docs.wandb.ai/weave/guides/integrations/pydantic_ai

Trace PydanticAI agent and tool calls in Weave using OpenTelemetry

You can trace [PydanticAI](https://ai.pydantic.dev/) agent and tool calls in Weave using [OpenTelemetry (OTEL)](https://opentelemetry.io/). PydanticAI is a Python agent framework built by the Pydantic team to make it straightforward and type-safe to build production-grade applications with Generative AI. It uses OTEL for tracing all agent and tool calls.

<Tip>
  For more information on OTEL tracing in Weave, see [Send OTEL Traces to Weave](../tracking/otel).
</Tip>

This guide is for developers building PydanticAI agents who want visibility into agent and tool execution in Weave. It shows you how to trace PydanticAI agent and tool calls using OTEL and visualize those traces in Weave. You learn how to install the required dependencies, configure an OTEL tracer to send data to Weave, and instrument your PydanticAI agents and tools. You also see how to enable tracing by default across all agents in your application.

## Prerequisites

Before you can send traces to Weave, you must install the OTEL dependencies and configure an OTEL exporter. Install the required OTEL dependencies:

```bash theme={null}
pip install opentelemetry-sdk opentelemetry-exporter-otlp-proto-http
```

Then, [configure OTEL tracing in Weave](#configure-otel-tracing-in-weave).

### Configure OTEL tracing in Weave

To send traces from PydanticAI to Weave, configure OTEL with a `TracerProvider` and an `OTLPSpanExporter`. Set the exporter to the [correct endpoint and HTTP headers for authentication and project identification](#required-configuration).

Store sensitive environment variables like your API key and project info in an environment file (for example, `.env`), and load them using `os.environ`. This keeps your credentials secure and out of your codebase.

### Required configuration

* **Endpoint:** `https://trace.wandb.ai/otel/v1/traces`
* **Headers:**
  * `Authorization`: Basic auth using your W\&B API key.
  * `project_id`: Your W\&B entity and project name (for example, `myteam/myproject`).

### Example setup

The following code snippet demonstrates how to configure an OTLP span exporter and tracer provider to send OTEL traces from a PydanticAI application to Weave.

```python lines theme={null}
import base64
import os
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk import trace as trace_sdk
from opentelemetry.sdk.trace.export import SimpleSpanProcessor

# Load sensitive values from environment variables
WANDB_BASE_URL = "https://trace.wandb.ai"
PROJECT_ID = os.environ.get("WANDB_PROJECT_ID")  # Your W&B entity/project name e.g. "myteam/myproject"
WANDB_API_KEY = os.environ.get("WANDB_API_KEY")  # Your W&B API key

OTEL_EXPORTER_OTLP_ENDPOINT = f"{WANDB_BASE_URL}/otel/v1/traces"
AUTH = base64.b64encode(f"api:{WANDB_API_KEY}".encode()).decode()

OTEL_EXPORTER_OTLP_HEADERS = {
    "Authorization": f"Basic {AUTH}",
    "project_id": PROJECT_ID,
}

# Create the OTLP span exporter with endpoint and headers
exporter = OTLPSpanExporter(
    endpoint=OTEL_EXPORTER_OTLP_ENDPOINT,
    headers=OTEL_EXPORTER_OTLP_HEADERS,
)

# Create a tracer provider and add the exporter
tracer_provider = trace_sdk.TracerProvider()
tracer_provider.add_span_processor(SimpleSpanProcessor(exporter))
```

You now have a `tracer_provider` that can be passed to PydanticAI agents to route their OTEL traces to Weave.

## Trace PydanticAI agents with OTEL

To trace your PydanticAI agents and send trace data to Weave, pass an `InstrumentationSettings` object configured with your tracer provider to the `Agent` constructor. This ensures that all agent and tool calls are traced according to your OTEL configuration.

The following example shows how to create a basic agent with tracing enabled. The key step is setting the `instrument` argument when initializing the agent:

```python lines theme={null}
from pydantic_ai import Agent
from pydantic_ai.models.instrumented import InstrumentationSettings

# Create a PydanticAI agent with OTEL tracing
agent = Agent(
    "openai:gpt-4o",
    instrument=InstrumentationSettings(tracer_provider=tracer_provider),
)

result = agent.run_sync("What is the capital of France?")
print(result.output)
```

All calls to the agent are traced and sent to Weave.

<Frame>
  <img alt="A trace visualization of a basic PydanticAI agent" />
</Frame>

## Trace PydanticAI tools with OTEL

Weave can trace any PydanticAI operations that are instrumented with OTEL, including both agent and tool calls. This means that when your agent invokes a tool (for example, a function decorated with `@agent.tool_plain`), the entire interaction is captured and visualized in Weave, including tool inputs, outputs, and the model's reasoning.

The following example shows how to create an agent with a system prompt and a tool. Tracing is enabled automatically for both the agent and the tool:

```python lines theme={null}
from pydantic_ai import Agent
from pydantic_ai.models.instrumented import InstrumentationSettings

# Create a PydanticAI agent with a system prompt and OTEL tracing
agent = Agent(
    "openai:gpt-4o",
    system_prompt=(
        "You are a helpful assistant that can multiply numbers. "
        "When asked to multiply numbers, use the multiply tool."
    ),
    instrument=InstrumentationSettings(tracer_provider=tracer_provider),
)

# Define a tool
@agent.tool_plain
def multiply(a: int, b: int) -> int:
    """Multiply two numbers."""
    return a * b

# Ask the agent to use the tool
result = agent.run_sync("What is 7 multiplied by 8?")
print(result.output)
```

<Frame>
  <img alt="A trace visualization of a tool call" />
</Frame>

Both the agent call and the tool call are traced in Weave, allowing you to inspect the full reasoning and execution path of your application.

## Instrument all agents by default

To apply OTEL tracing to all PydanticAI agents in your application, use the `Agent.instrument_all()` method. This sets a default `InstrumentationSettings` instance for any agent that doesn't explicitly specify the `instrument` parameter.

```python lines theme={null}
from pydantic_ai import Agent
from pydantic_ai.models.instrumented import InstrumentationSettings

# Set up default instrumentation for all agents
Agent.instrument_all(InstrumentationSettings(tracer_provider=tracer_provider))

# Now, any new agent uses this instrumentation by default
agent1 = Agent("openai:gpt-4o")
agent2 = Agent("openai:gpt-4o", system_prompt="Be helpful.")

result = agent1.run_sync("What is the capital of France?")
print(result.output)
```

This is useful for larger applications where you want consistent tracing across all agents without repeating configuration. For more details, see the [PydanticAI OTEL docs](https://ai.pydantic.dev/logfire/#using-logfire).

## Learn more

* [Weave documentation: Send OTEL traces to Weave](../tracking/otel)
* [Official OTEL documentation](https://opentelemetry.io/)
* [Official PydanticAI documentation](https://ai.pydantic.dev/)
* [PydanticAI GitHub repository](https://github.com/pydantic/pydantic-ai)


# Smolagents
Source: https://docs.wandb.ai/weave/guides/integrations/smolagents

Track Smolagents applications with Weave's automatic tracing, capturing tool calls, model inferences, and multi-step agent workflows.

<Warning>
  All code samples shown on this page are in Python.
</Warning>

This page explains how to integrate [Smolagents](https://huggingface.co/docs/smolagents/en/index) with W\&B Weave to track and analyze your agentic applications. It's intended for developers building agentic workflows who want visibility into their model and tool calls. You'll learn how to log model inferences, monitor function calls, and organize experiments using Weave's tracing and versioning capabilities. By following the examples provided, you can capture valuable insights, debug your applications efficiently, and compare different model configurations (all within the Weave web interface).

## Overview

Smolagents is a framework that offers minimal abstractions for building agentic applications. It supports multiple LLM providers, such as OpenAI, Hugging Face Transformers, and Anthropic.

Weave automatically captures traces for [Smolagents](https://huggingface.co/docs/smolagents/en/index). To start tracking, call `weave.init()` and use the library as usual.

## Prerequisites

Before you start, you must install the required libraries and configure access to your chosen LLM provider.

1. Before you can use Smolagents with Weave, you must install the required libraries or upgrade to the latest versions. The following command installs or upgrades `smolagents`, `openai`, and `weave`, and suppresses output:

   ```bash theme={null}
   pip install -U smolagents openai weave -qqq
   ```

2. Smolagents supports multiple LLM providers, such as OpenAI, Hugging Face Transformers, and Anthropic. Set the API key for your chosen provider so that Smolagents can authenticate when invoking the model. Set the corresponding environment variable:

   ```python lines theme={null}
   import os
   import getpass

   os.environ["OPENAI_API_KEY"] = getpass.getpass("Enter your OpenAI API key: ")
   ```

## Basic tracing

This section shows how Weave automatically captures traces for a basic Smolagents workflow.

Storing traces of language model applications in a central location is essential during development and production. These traces help with debugging and serve as datasets for improving your application.

Weave automatically captures traces for [Smolagents](https://huggingface.co/docs/smolagents/en/index). To start tracking, initialize Weave by calling `weave.init()`, then use the library as usual.

The following example demonstrates how to log inference calls to a tool-using LLM agent with Weave. In this scenario:

* You define a language model (OpenAI's `gpt-4o`) using Smolagents' `OpenAIServerModel`.
* You configure a search tool (`DuckDuckGoSearchTool`) that the agent can invoke when needed.
* You construct a `ToolCallingAgent`, passing in the tool and model.
* You run a query through the agent that triggers the search tool.
* Weave logs each function and model invocation, making them available for inspection in its web interface.

```python lines theme={null}
import weave
from smolagents import DuckDuckGoSearchTool, OpenAIServerModel, ToolCallingAgent

# Initialize Weave
weave.init(project_name="smolagents")

# Define your LLM provider supported by Smolagents
model = OpenAIServerModel(model_id="gpt-4o")

# Define a DuckDuckGo web search tool based on your query
search_tool = DuckDuckGoSearchTool()

# Define a tool-calling agent
agent = ToolCallingAgent(tools=[search_tool], model=model)
answer = agent.run(
    "Get me just the title of the page at url 'https://wandb.ai/geekyrakshit/story-illustration/reports/Building-a-GenAI-assisted-automatic-story-illustrator--Vmlldzo5MTYxNTkw'?"
)
```

Once you run the code sample, navigate to your Weave project dashboard to view the traces.

<Frame>
  <img alt="Weave logs each inference call, providing details about inputs, outputs, and metadata." />
</Frame>

## Tracing custom tools

In addition to built-in tools, you can extend your agents with custom tools and have Weave trace those calls as well.

You can declare custom tools for your agentic workflows by decorating a function with `@tool` from `smolagents` or by inheriting from the `smolagents.Tool` class.

Weave automatically tracks custom tool calls for your Smolagents workflows. The following example shows how to log a custom Smolagents tool call with Weave:

* You define a custom `get_weather` function and decorate it with `@tool` from Smolagents, enabling the agent to invoke it as part of its reasoning process.
* The function accepts a location and an optional flag for Celsius output.
* You instantiate a language model using `OpenAIServerModel`.
* You create a `ToolCallingAgent` with the custom tool and model.
* When the agent runs the query, it selects and invokes the `get_weather` tool.
* Weave logs both the model inference and the custom tool invocation, including arguments and return values.

```python lines theme={null}
from typing import Optional

import weave
from smolagents import OpenAIServerModel, ToolCallingAgent, tool

weave.init(project_name="smolagents")

@tool
def get_weather(location: str, celsius: Optional[bool] = False) -> str:
    """
    Get the weather in the next few days for a given location.
    Args:
        location: The location.
        celsius: Whether to use Celsius for temperature.
    """
    return f"The weather in {location} is sunny with temperatures around 7°C."

model = OpenAIServerModel(model_id="gpt-4o")
agent = ToolCallingAgent(tools=[get_weather], model=model)
answer = agent.run("What is the weather in Tokyo?")
```

Once you run the code sample, navigate to your Weave project dashboard to view the traces.

<Frame>
  <img alt="Weave logs each custom tool call." />
</Frame>


# Vercel AI SDK
Source: https://docs.wandb.ai/weave/guides/integrations/vercel_ai_sdk

Trace Vercel AI SDK calls in Weave using OpenTelemetry

You can trace [Vercel AI SDK](https://ai-sdk.dev/) calls in Weave using [OpenTelemetry (OTel)](https://opentelemetry.io/). The Vercel AI SDK is a TypeScript toolkit for building AI-powered applications, with framework support for Next.js, Nuxt, SvelteKit, and others. It has built-in OpenTelemetry support through its `experimental_telemetry` option.

This guide shows you how to configure OTel to send traces from the Vercel AI SDK to Weave so that you can observe and debug your AI SDK calls in the Weave UI. You can use the AI SDK with Next.js or as a standalone Node.js application. This guide is intended for developers building AI-powered applications with the Vercel AI SDK who want to add tracing to an existing app.

For more information about OTel tracing in Weave, see [Send OTel traces to Weave](../tracking/otel).

## Prerequisites

Before you begin, complete the following setup. Both the Next.js and Node.js examples in this guide require the same dependencies:

1. Install the following Vercel and OTel libraries:

   ```bash theme={null}
   npm install ai @ai-sdk/openai @opentelemetry/api @opentelemetry/sdk-trace-node @opentelemetry/sdk-trace-base @opentelemetry/exporter-trace-otlp-proto @opentelemetry/resources zod
   ```

2. Set the following environment variables:

   ```bash theme={null}
   export WANDB_API_KEY="your-wandb-api-key"
   export OPENAI_API_KEY="your-openai-api-key"
   ```

   You can get your W\&B API key from [User Settings](https://wandb.ai/settings).

## Configure OTel tracing for Next.js

This section demonstrates how to configure Weave in a Next.js app. The example doesn't contain a full app, only the instrumentation configuration and how to invoke the instrumentation on a Vercel AI SDK function (in this case, a call to OpenAI).

### Configure instrumentation

Next.js applications use an `instrumentation.ts` file to set up OTel. This file runs once when the server starts and configures the tracer provider that the AI SDK uses.

To integrate Weave with Vercel's OTel functionality, create an `instrumentation.ts` file in your project root and add the following code to it, updating the `resourceFromAttributes()` function with your team and project names:

```typescript instrumentation.ts lines {17-19} theme={null}
import { NodeTracerProvider } from "@opentelemetry/sdk-trace-node";
import { BatchSpanProcessor } from "@opentelemetry/sdk-trace-base";
import { OTLPTraceExporter } from "@opentelemetry/exporter-trace-otlp-proto";
import { resourceFromAttributes } from "@opentelemetry/resources";

export function register() {
  const WANDB_API_KEY = process.env.WANDB_API_KEY!;

// Configure OTel exporter to use W&B Weave endpoint.
  const exporter = new OTLPTraceExporter({
    url: "https://trace.wandb.ai/otel/v1/traces",
    headers: { "wandb-api-key": WANDB_API_KEY },
  });

// Configure your W&B credentials.
  const provider = new NodeTracerProvider({
    resource: resourceFromAttributes({
      "wandb.entity": "[YOUR-TEAM-NAME]",
      "wandb.project": "[YOUR-PROJECT-NAME]",
    }),
    spanProcessors: [new BatchSpanProcessor(exporter)],
  });

  provider.register();
}
```

This creates an OTLP exporter configured to send trace data to Weave's OTel endpoint, authenticating with your W\&B API key.

### Configure telemetry on a function

After you've added the instrumentation, use Vercel's `experimental_telemetry` option on any AI SDK function call to emit OTel spans:

```typescript route.ts lines {10} theme={null}
import { openai } from "@ai-sdk/openai";
import { generateText } from "ai";

export async function POST(req: Request) {
  const { prompt } = await req.json();

// Add experiment_telemetry field to function.
  const result = await generateText({
    model: openai("gpt-4o-mini"),
    prompt,
    experimental_telemetry: { isEnabled: true },
  });

  return Response.json({ text: result.text });
}
```

All `generateText` calls with telemetry enabled produce OTel spans that are exported to Weave. Your Next.js app is now configured to send Vercel AI SDK traces to Weave.

## Configure OTel tracing for Node.js

If you're not using Next.js, you can configure OTel directly in a standalone Node.js application. For standalone Node.js applications, configure the tracer provider at the top of your entry file before any AI SDK calls so that subsequent AI SDK calls are captured.

After meeting the prerequisites, you can run the following example and generate spans without any additional configuration.

```typescript lines {11-14,18-19,31} title="test-app.ts" theme={null}
import { NodeTracerProvider } from "@opentelemetry/sdk-trace-node";
import { BatchSpanProcessor } from "@opentelemetry/sdk-trace-base";
import { OTLPTraceExporter } from "@opentelemetry/exporter-trace-otlp-proto";
import { resourceFromAttributes } from "@opentelemetry/resources";
import { openai } from "@ai-sdk/openai";
import { generateText } from "ai";

const WANDB_API_KEY = process.env.WANDB_API_KEY!;

// Configure OTel exporter to use W&B Weave endpoint.
const exporter = new OTLPTraceExporter({
  url: "https://trace.wandb.ai/otel/v1/traces",
  headers: { "wandb-api-key": WANDB_API_KEY },
});

// Configure your W&B credentials.
const provider = new NodeTracerProvider({
  resource: resourceFromAttributes({
    "wandb.entity": "[YOUR-TEAM-NAME]",
    "wandb.project": "[YOUR-PROJECT-NAME]",
  }),
  spanProcessors: [new BatchSpanProcessor(exporter)],
});

provider.register();

// Add experiment_telemetry field to function.
async function main() {
  const result = await generateText({
    model: openai("gpt-4o-mini"),
    prompt: "Explain OpenTelemetry in one sentence.",
    experimental_telemetry: { isEnabled: true },
  });

  console.log(result.text);

  await provider.shutdown();
}

main();
```

`BatchSpanProcessor` flushes spans asynchronously. In short-lived processes like standalone scripts, serverless functions, or CLI tools, call `provider.shutdown()` before the process exits to ensure all spans are sent to Weave. For long-running servers (like a Next.js dev server started through `instrumentation.ts`), this isn't necessary.


# Verdict
Source: https://docs.wandb.ai/weave/guides/integrations/verdict

Use Verdict evaluation framework with Weave to trace and monitor your LLM evaluation pipelines

<a href="https://github.com/wandb/examples/blob/master/weave/docs/quickstart_verdict.ipynb">
  <img alt="Open In Colab" />
</a>

Weave is designed to track and log all calls made through the [Verdict Python library](https://verdict.haizelabs.com/docs/) automatically.

When working with AI evaluation pipelines, debugging matters. Whether a pipeline step fails, outputs are unexpected, or nested operations create confusion, pinpointing issues can be hard. Verdict applications often consist of multiple pipeline steps, judges, and transformations, so it helps to understand the inner workings of your evaluation workflows.

Weave streamlines this process by automatically capturing traces for your [Verdict](https://verdict.readthedocs.io/) applications. This lets you monitor and analyze your pipeline's performance to debug and optimize your AI evaluation workflows.

## Getting started

To enable Weave tracing for your Verdict pipelines, call `weave.init(project=...)` at the beginning of your script. Use the `project` argument to log to a specific W\&B Team name with `team-name/project-name`, or pass `project-name` to log to your default team or entity.

```python lines {7} theme={null}
import weave
from verdict import Pipeline
from verdict.common.judge import JudgeUnit
from verdict.schema import Schema

# Initialize Weave with your project name
weave.init("verdict_demo")

# Create a simple evaluation pipeline
pipeline = Pipeline()
pipeline = pipeline >> JudgeUnit().prompt("Rate the quality of this text: {source.text}")

# Create sample data
data = Schema.of(text="This is a sample text for evaluation.")

# Run the pipeline - Weave traces this automatically
output = pipeline.run(data)

print(output)
```

## Tracking call metadata

To attach custom metadata to your Verdict pipeline calls, use the [`weave.attributes`](/weave/reference/python-sdk#function-attributes) context manager. This context manager lets you tag a specific block of code, such as a pipeline run or evaluation batch, so that you can filter and group related traces later in the Weave UI.

```python lines {7,14} theme={null}
import weave
from verdict import Pipeline
from verdict.common.judge import JudgeUnit
from verdict.schema import Schema

# Initialize Weave with your project name
weave.init("verdict_demo")

pipeline = Pipeline()
pipeline = pipeline >> JudgeUnit().prompt("Evaluate sentiment: {source.text}")

data = Schema.of(text="I love this product!")

with weave.attributes({"evaluation_type": "sentiment", "batch_id": "batch_001"}):
    output = pipeline.run(data)

print(output)
```

Weave automatically tracks the metadata against the trace of the Verdict pipeline call. You can view the metadata in the Weave web interface.

## Traces

Storing traces of AI evaluation pipelines in a central database helps during both development and production. These traces support debugging and improving your evaluation workflows, and they provide a useful dataset.

Weave automatically captures traces for your Verdict applications. It tracks and logs all calls made through the Verdict library, including:

* `Pipeline` execution steps.
* `JudgeUnit` evaluations.
* `Layer` transformations.
* Pooling operations.
* Custom units and transformations.

You can view the traces in the Weave web interface, which shows the hierarchical structure of your pipeline execution.

## Pipeline tracing example

The following example shows how Weave traces nested pipeline operations, so you can see how each step in a multi-stage Verdict pipeline is captured:

```python lines {8} theme={null}
import weave
from verdict import Pipeline, Layer
from verdict.common.judge import JudgeUnit
from verdict.transform import MeanPoolUnit
from verdict.schema import Schema

# Initialize Weave with your project name
weave.init("verdict_demo")

# Create a pipeline with multiple steps
pipeline = Pipeline()
pipeline = pipeline >> Layer([
    JudgeUnit().prompt("Rate coherence: {source.text}"),
    JudgeUnit().prompt("Rate relevance: {source.text}"),
    JudgeUnit().prompt("Rate accuracy: {source.text}")
], 3)
pipeline = pipeline >> MeanPoolUnit()

# Sample data
data = Schema.of(text="This is an evaluation of text quality across multiple dimensions.")

# Run the pipeline - Weave traces all operations
result = pipeline.run(data)

print(f"Average score: {result}")
```

This creates a detailed trace showing:

* The main `Pipeline` execution.
* Each `JudgeUnit` evaluation within the `Layer`.
* The `MeanPoolUnit` aggregation step.
* Timing information for each operation.

## Configuration

When you call `weave.init()`, Weave automatically enables tracing for Verdict pipelines. The integration works by patching the `Pipeline.__init__()` method to inject a `VerdictTracer` that forwards all trace data to Weave.

You don't need any additional configuration. Weave automatically performs the following:

* Captures all pipeline operations.
* Tracks execution timing.
* Logs inputs and outputs.
* Maintains trace hierarchy.
* Handles concurrent pipeline execution.

## Custom tracers and Weave

If you already use custom Verdict tracers in your application, Weave's `VerdictTracer` can run alongside them so that you don't have to choose between integrations:

```python lines {8} theme={null}
import weave
from verdict import Pipeline
from verdict.common.judge import JudgeUnit
from verdict.util.tracing import ConsoleTracer
from verdict.schema import Schema

# Initialize Weave with your project name
weave.init("verdict_demo")

# You can still use Verdict's built-in tracers
console_tracer = ConsoleTracer()

# Create pipeline with both Weave (automatic) and Console tracing
pipeline = Pipeline(tracer=[console_tracer])  # Weave tracer is added automatically
pipeline = pipeline >> JudgeUnit().prompt("Evaluate: {source.text}")

data = Schema.of(text="Sample evaluation text")

# This traces to both Weave and console
result = pipeline.run(data)
```

## Models and evaluations

Organizing and evaluating AI systems with multiple pipeline components can be challenging. Using the [`weave.Model`](/weave/guides/core-types/models), you can capture and organize experimental details like prompts, pipeline configurations, and evaluation parameters, making it easier to compare different iterations.

The following example demonstrates wrapping a Verdict pipeline in a `weave.Model`:

```python lines {8,10,14} theme={null}
import asyncio
import weave
from verdict import Pipeline
from verdict.common.judge import JudgeUnit
from verdict.schema import Schema

# Initialize Weave with your project name
weave.init("verdict_demo")

class TextQualityEvaluator(weave.Model):
    judge_prompt: str
    pipeline_name: str

    @weave.op()
    async def predict(self, text: str) -> dict:
        pipeline = Pipeline(name=self.pipeline_name)
        pipeline = pipeline >> JudgeUnit().prompt(self.judge_prompt)
        
        data = Schema.of(text=text)
        result = pipeline.run(data)
        
        return {
            "text": text,
            "quality_score": result.score if hasattr(result, 'score') else result,
            "evaluation_prompt": self.judge_prompt
        }

model = TextQualityEvaluator(
    judge_prompt="Rate the quality of this text on a scale of 1-10: {source.text}",
    pipeline_name="text_quality_evaluator"
)

text = "This is a well-written and informative piece of content that provides clear value to readers."

prediction = asyncio.run(model.predict(text))

# if you're in a Jupyter Notebook, run:
# prediction = await model.predict(text)

print(prediction)
```

This code creates a model that you can visualize in the Weave UI, showing both the pipeline structure and the evaluation results.

### Evaluations

Evaluations help you measure the performance of your evaluation pipelines themselves. By using the [`weave.Evaluation`](/weave/guides/core-types/evaluations) class, you can capture how well your Verdict pipelines perform on specific tasks or datasets:

```python lines {8} theme={null}
import asyncio
import weave
from verdict import Pipeline
from verdict.common.judge import JudgeUnit
from verdict.schema import Schema

# Initialize Weave
weave.init("verdict_demo")

# Create evaluation model
class SentimentEvaluator(weave.Model):
    @weave.op()
    async def predict(self, text: str) -> dict:
        pipeline = Pipeline()
        pipeline = pipeline >> JudgeUnit().prompt(
            "Classify sentiment as positive, negative, or neutral: {source.text}"
        )
        
        data = Schema.of(text=text)
        result = pipeline.run(data)
        
        return {"sentiment": result}

# Test data
texts = [
    "I love this product, it's amazing!",
    "This is terrible, worst purchase ever.",
    "The weather is okay today."
]
labels = ["positive", "negative", "neutral"]

examples = [
    {"id": str(i), "text": texts[i], "target": labels[i]}
    for i in range(len(texts))
]

# Scoring function
@weave.op()
def sentiment_accuracy(target: str, output: dict) -> dict:
    predicted = output.get("sentiment", "").lower()
    return {"correct": target.lower() in predicted}

model = SentimentEvaluator()

evaluation = weave.Evaluation(
    dataset=examples,
    scorers=[sentiment_accuracy],
)

scores = asyncio.run(evaluation.evaluate(model))
# if you're in a Jupyter Notebook, run:
# scores = await evaluation.evaluate(model)

print(scores)
```

This creates an evaluation trace that shows how your Verdict pipeline performs across different test cases.

## Best practices

The following sections describe best practices for monitoring performance and handling errors when using Weave with Verdict pipelines.

### Performance monitoring

Weave automatically captures timing information for all pipeline operations, which you can use to identify performance bottlenecks across runs:

```python lines {6} theme={null}
import weave
from verdict import Pipeline, Layer
from verdict.common.judge import JudgeUnit
from verdict.schema import Schema

weave.init("verdict_demo")

# Create a pipeline that might have performance variations
pipeline = Pipeline()
pipeline = pipeline >> Layer([
    JudgeUnit().prompt("Quick evaluation: {source.text}"),
    JudgeUnit().prompt("Detailed analysis: {source.text}"),  # This might be slower
], 2)

data = Schema.of(text="Sample text for performance testing")

# Run multiple times to see timing patterns
for i in range(3):
    with weave.attributes({"run_number": i}):
        result = pipeline.run(data)
```

### Error handling

Weave automatically captures exceptions that occur during pipeline execution, so Weave records failures in the trace even when your application handles the exception:

```python lines {6} theme={null}
import weave
from verdict import Pipeline
from verdict.common.judge import JudgeUnit
from verdict.schema import Schema

weave.init("verdict_demo")

pipeline = Pipeline()
pipeline = pipeline >> JudgeUnit().prompt("Process: {source.invalid_field}")  # This causes an error

data = Schema.of(text="Sample text")

try:
    result = pipeline.run(data)
except Exception as e:
    print(f"Pipeline failed: {e}")
    # Error details are captured in Weave trace
```

Integrating Weave with Verdict gives you visibility into your AI evaluation pipelines, making it easier to debug, optimize, and understand your evaluation workflows.


# Verifiers
Source: https://docs.wandb.ai/weave/guides/integrations/verifiers

Track Verifiers RL environments and LLM agent training with Weave, capturing multi-round conversations and evaluation rollouts.

[Verifiers](https://github.com/willccbb/verifiers) is a library of modular components for creating Reinforcement Learning (RL) environments and training LLM agents. Environments built with Verifiers can serve as LLM evaluations, synthetic data pipelines, agent harnesses for any OpenAI-compatible endpoint, and RL training.

Along with using W\&B to record your training metrics, you can integrate Weave with your Verifiers RL workflows to gain observability into how your model performs during training. Weave records inputs, outputs, and timestamps for each step so you can inspect how data transforms at every turn, debug complex multi-round conversations, and optimize training results.

You can also use Weave and Verifiers together to perform evaluations.

This guide is for ML engineers and researchers who want to add observability to their Verifiers-based RL workflows. It shows you how to install Verifiers, W\&B, and Weave, and provides two examples of how to use Verifiers with Weave and W\&B: tracing rollouts during evaluation, and fine-tuning a model with experiment tracking and tracing enabled.

<img alt="verifiers wandb run page" />

## Get started

To integrate Verifiers with Weave, start by installing the Verifiers library using `uv` ([recommended by the library's authors](https://github.com/willccbb/verifiers?tab=readme-ov-file#setup)). Use one of the following commands to install the library:

```bash theme={null}
# Installs core library for local development and API-based models
uv add verifiers

# Installs full version of library with all optional dependencies, including PyTorch and GPU support
uv add 'verifiers[all]' && uv pip install flash-attn --no-build-isolation

# Installs latest version of library directly from GitHub, including latest unreleased features and fixes
uv add verifiers @ git+https://github.com/willccbb/verifiers.git
```

Then install Weave and W\&B:

```bash theme={null}
uv pip install weave wandb
```

Weave enables [implicit patching](/weave/guides/integrations#automatic-implicit-patching) for the library by default. This lets you use Weave with Verifiers without requiring explicit call patch functions.

With the libraries installed, you're ready to run the examples in the following sections.

### Trace rollouts and evaluate

Once you've installed the necessary libraries, you can use Weave and Verifiers together to trace calls and run evaluations.

The following example script demonstrates how to run an evaluation with Verifiers and log the results to Weave. The script tests the LLM's ability to solve math problems using the [GSM8K dataset](https://huggingface.co/datasets/openai/gsm8k). It asks GPT-4 to solve two math problems, extracts the numerical value from each response, and then grades the attempt using Verifiers as an evaluation framework.

Run the example and inspect the results in Weave:

```python lines theme={null}
import os
from openai import OpenAI
import verifiers as vf
import weave

os.environ["OPENAI_API_KEY"] = "[YOUR-OPENAI-API-KEY]"

# Initialize Weave
weave.init("verifiers_demo")

# Minimal single-turn environment
dataset = vf.load_example_dataset("gsm8k", split="train").select(range(2))
parser = vf.ThinkParser()

def correct_answer(parser, completion, answer):
    parsed = parser.parse_answer(completion) or ""
    return 1.0 if parsed.strip() == answer.strip() else 0.0

rubric = vf.Rubric(funcs=[correct_answer, parser.get_format_reward_func()], weights=[1.0, 0.2])

env = vf.SingleTurnEnv(
    dataset=dataset,
    system_prompt="Think step-by-step, then answer.",
    parser=parser,
    rubric=rubric,
)

client = OpenAI()
results = env.evaluate(
    client, "gpt-4.1-mini", num_examples=2, rollouts_per_example=2, max_concurrent=8
)
```

### Fine-tune a model with experiment tracking and tracing

Weave provides insight into how your models perform during RL fine-tuning. When you use it alongside W\&B, you get observability across the workflow. W\&B tracks training metrics and performance charts, while Weave captures detailed traces of each interaction during the training process.

The `verifiers` repository includes ready-to-run [examples](https://github.com/PrimeIntellect-ai/verifiers) to help you get started.

The following example RL training pipeline runs a local inference server and trains a model using the GSM8K dataset. The model responds with answers to the math problems and the training loop scores the output and updates the model accordingly. W\&B logs the training metrics, like loss, reward, and accuracy, while Weave captures the input, output, reasoning, and scoring.

To use this pipeline:

1. Install the framework from the source. The following commands install the Verifiers library from GitHub and the necessary dependencies:

```bash theme={null}
git clone https://github.com/willccbb/verifiers
cd verifiers
uv sync --all-extras && uv pip install flash-attn --no-build-isolation
```

2. Install an off-the-shelf environment. The following command installs the pre-configured GSM8K training environment:

```bash theme={null}
vf-install gsm8k --from-repo
```

3. Train your model. The following commands launch the inference server and training loop, respectively. This example workflow sets `report_to=wandb` by default, so you don't need to call `wandb.init()` separately. You'll be prompted to authenticate this machine to log metrics to W\&B.

```bash theme={null}
# Runs inference server
CUDA_VISIBLE_DEVICES=0 vf-vllm --model willcb/Qwen3-0.6B --enforce-eager --disable-log-requests

# Runs training loop
CUDA_VISIBLE_DEVICES=1 accelerate launch --num-processes 1 --config-file configs/zero3.yaml examples/grpo/train_gsm8k.py
```

<Note>
  This example was tested on 2xH100s with the following environment variables set for increased stability:

  ```bash theme={null}
  # In BOTH shells (server and trainer) before launch
  export NCCL_CUMEM_ENABLE=0
  export NCCL_CUMEM_HOST_ENABLE=0
  ```

  These variables disable CUDA Unified Memory (CuMem) for device memory allocations.
</Note>

Once training begins, you can [view the traces](/weave/guides/tools/weave-in-workspaces) logged during your run in the UI. W\&B captures the training metrics while Weave records the per-interaction traces, giving you both high-level performance trends and detailed step-by-step visibility into your training run.

Traces omit `logprobs` for the `Environment.a_generate` and `Rubric.score_rollouts` methods. This keeps payloads small while leaving the originals intact for training.

## See also

Verifiers has first-class integration with W\&B Models. For more information, see the [Verifiers repository](https://github.com/willccbb/verifiers).


# Deployment options and security features
Source: https://docs.wandb.ai/weave/guides/platform

Learn about Weave's deployment options, IAM management, and data security features.

This page describes how to deploy Weave, manage access, and protect data, so you can choose the deployment option and security controls that fit your organization's needs.

Weave is available on the following deployment options:

* **[W\&B Multi-tenant Cloud](https://docs.wandb.ai/platform/hosting/hosting-options/multi_tenant_cloud):** A multi-tenant, fully managed platform deployed in W\&B's Google Cloud Platform (Google Cloud) account in a North America region.
* **[W\&B Dedicated Cloud](https://docs.wandb.ai/platform/hosting/hosting-options/dedicated-cloud):** Generally available on AWS, Google Cloud, and Azure.
* **[Self-Managed instances](/weave/guides/platform/weave-self-managed):** For teams that prefer to host Weave independently, your W\&B team provides guidance to evaluate deployment options.

## Identity and access management

Use the identity and access management capabilities for secure authentication and effective authorization in your [W\&B Organization](/platform/hosting/iam/org_team_struct#organization). The following capabilities are available for Weave users depending on your deployment option and [pricing plan](https://wandb.ai/site/pricing/):

* **Authenticate using single sign-on (SSO):** Options include public identity providers like Google and GitHub, and enterprise providers such as Okta and Azure Active Directory, [using OIDC](/platform/hosting/iam/sso).
* **[Team-based logical separation](/platform/app/settings-page/teams):** Each team may correspond to a business unit, department, or project team within your organization.
* **Use W\&B projects to organize initiatives:** Organize initiatives within teams and configure the required [visibility scope](/platform/hosting/iam/access-management/restricted-projects), including the `restricted` scope for sensitive collaborations.
* **Role-based access control:** Configure access at the [team](/platform/hosting/iam/access-management/restricted-projects#assign-project-level-role-to-a-user) or [project](/platform/hosting/iam/access-management/restricted-projects#assign-project-level-role-to-a-user) level to ensure users access data on a need-to-know basis.
* **Scoped service accounts:** Automate generative AI workflows using service accounts scoped to your organization or team.
* **[SCIM API and Python SDK](/platform/hosting/iam/automate_iam):** Manage users and teams with the SCIM API and the Python SDK.

## Data security

The data security protections available to you depend on your deployment option.

* **Multi-tenant Cloud:** W\&B stores data for all Weave users in a shared ClickHouse Cloud cluster, encrypted with cloud-native encryption. Shared compute services process the data and ensure isolation through a security context that comprises your W\&B organization, team, and project.

* **Dedicated Cloud:** W\&B stores data in a unique ClickHouse Cloud cluster in the cloud and region of your choice. A unique compute environment processes the data, with the following additional protections:
  * **[IP allowlisting](/platform/hosting/data-security/ip-allowlisting):** Authorize access to your instance from specific IP addresses. This is an optional capability.
  * **[Private connectivity](/platform/hosting/data-security/private-connectivity):** Route data securely through the cloud provider's private network. This is an optional capability.
  * **[Data encryption](/platform/hosting/data-security/data-encryption):** W\&B encrypts data at rest using a unique W\&B-managed encryption key.
  * **ClickHouse cluster security:** W\&B connects to the unique ClickHouse Cloud cluster for your Dedicated Cloud instance over the cloud provider's private network. W\&B also encrypts the cluster with a unique W\&B-managed encryption key and uses ClickHouse's file-level encryption.

<Warning>
  [The W\&B Platform secure storage connector or BYOB](/platform/hosting/data-security/secure-storage-connector) is not available for Weave.
</Warning>

## Maintenance

If you're using Weave on Multi-tenant Cloud or Dedicated Cloud, you avoid the overhead and costs of provisioning, operating, and maintaining the W\&B platform, because W\&B fully manages it for you.

## Compliance

<Tip>
  To request SOC 2 reports and other security and compliance documents, refer to the [W\&B Security Portal](https://security.wandb.ai/) or contact your W\&B team for more information.
</Tip>

Security controls for both Multi-tenant Cloud and Dedicated Cloud are periodically audited internally and externally. Both platforms are SOC 2 Type II compliant. Dedicated Cloud is also HIPAA-compliant for organizations managing PHI data while building generative AI applications.


# Manage Weave projects
Source: https://docs.wandb.ai/weave/guides/platform/weave-projects

Use Weave projects to organize related assets like traces, prompts, evaluations, models, and dashboards.

Weave projects organize related assets, including traces, evaluations, models, datasets, and other Weave assets. Projects keep your team organized, streamline collaboration and sharing, and help you track the evolution of your LLM applications over time.

This page explains what resources a Weave project contains, how project paths work, and how to create, view, and delete projects using either the W\&B UI or the Weave SDK.

## Project resources

Projects store the following resources:

| Resource         | Description                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
| ---------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| **Traces**       | Individual LLM calls, function executions, and application flows that capture inputs, outputs, latency, and token usage for debugging and analysis. See [Traces](/weave/guides/tracking/tracing).                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              |
| **Evaluations**  | Test results that compare model performance against datasets using scoring functions to measure quality, accuracy, and other metrics. See [Evaluations](/weave/guides/core-types/evaluations).                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |
| **Monitors**     | Background processes that automatically score a subset of function calls using LLM-as-a-judge scorers to track quality metrics over time in production. See [Monitors](/weave/guides/evaluation/monitors).                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
| **Leaderboards** | Ranked comparisons of model performance across different evaluation runs that help you identify the best-performing configurations. See [Leaderboards](/weave/guides/core-types/leaderboards).                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |
| **Threads**      | Multi-turn conversations and session-based workflows that group related calls under a shared thread ID to analyze complete user interactions. See [Threads](/weave/guides/tracking/threads).                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
| **Assets**       | The **Assets** tab stores and organizes versioned objects in your project: <ul><li>**Models**: Versioned model definitions that capture prompts, parameters, and configuration, and automatically track changes for reproducibility and comparison. See [Models](/weave/guides/core-types/models).</li><li>**Datasets**: Collections of test examples stored as Weave objects that you can download, browse, and use to run evaluations. See [Datasets](/weave/guides/core-types/datasets).</li><li>**Prompts**: Versioned system prompts that you can test and reuse in your model configurations. See [Prompts](/weave/guides/core-types/prompts).</li><li>**Scorers**: Functions that evaluate model inputs and outputs to measure quality, safety, or custom metrics. See [Scorers](/weave/guides/evaluation/scorers).</li><li>**Ops**: Automatically versioned and tracked functions decorated with `@weave.op()` that produce calls and maintain a record of function modifications. See [Ops](/weave/guides/tracking/ops).</li><li>**Other objects and metadata**: Additional Weave objects, custom types, and associated metadata stored within the project.</li></ul> |

## Project paths

[Teams](/platform/hosting/iam/org_team_struct#team) (also called entities) own and manage projects in your W\&B organization. Your team name and project name make up the path to your project in the W\&B UI. You also use this path in your code to associate your Weave resources with the project.

For example, if your team's name is `my-team` and your project's name is `my-project`, the path to your project is:

```text theme={null}
my-team/my-project
```

Use this path when you initialize Weave in your code to associate your app's data with the project:

```python theme={null}
weave.init('my-team/my-project')
```

## Create and find projects

You can create a Weave project using the W\&B UI or by setting a new project path in your code.

<Tabs>
  <Tab title="W&B UI">
    To create a new project using the W\&B UI:

    1. Navigate to [wandb.ai](https://wandb.ai).
    2. Select **New project**. This opens the **Create a new project** modal dialog.
    3. In the modal, configure the following fields:
       * **Team**: The existing team to associate the project with.
       * **Name**: A name for your project.
       * **Project visibility**: The scope of who can view and edit the project.
       * **Description**: (Optional) A short description of the project to help you track and organize its purpose.
    4. After you configure the necessary fields, select **Create project**. W\&B creates the project and opens its workspace.

    To locate your newly created project, select the **Weights & Biases logo** in the upper-left corner to open the **Global navigation** menu. The menu lists your most recent projects and the teams you belong to, so you can switch context. Select your project to open the project's workspace.

    Alternatively, open your **Profile** from the menu under your name in the upper-right corner. The **Projects** tab lists all of your projects.

    When you open the new project's workspace, the UI offers quickstarts for both W\&B Models and W\&B Weave. The project's toolbar also contains tools for each product. After you associate Weave data with your project through your code, the sidebar shows only Weave tools and the Weave dashboard becomes available.
  </Tab>

  <Tab title="Weave SDK">
    To create a Weave project directly from your code, provide a new project path as an argument to `weave.init()`:

    <CodeGroup>
      ```python theme={null}
      import weave

      weave.init('my-team/my-llm-app')
      ```

      ```typescript theme={null}
      import * as weave from 'weave';

      await weave.init('my-team/my-llm-app');
      ```
    </CodeGroup>

    When you run the code, Weave creates the new project in your W\&B account and uploads any applicable data from the run to the project. Weave also configures the project as a Weave project (not a W\&B Models project), and the UI sidebar shows only Weave tools.
  </Tab>
</Tabs>

## View projects in the UI

To access a project in the Weave UI:

1. Navigate to [wandb.ai](https://wandb.ai).
2. Select the **Weights & Biases logo** in the upper-left corner to open the **Global navigation menu**, then select your project from the **Projects** sidebar. This opens the project's overview page, or the **Traces** page if you've already logged traces to the project.
3. From the overview page, you can:
   * Review your project's statistics, such as the number of traces logged to the project and how much storage the project uses.
   * Navigate to traces, evaluations, and other resources using the sidebar.
   * Navigate to recently logged traces, evaluations, models, and datasets.

<Frame>
  <img alt="Weave project" />
</Frame>

## Delete a project

<Warning>
  Deleting a project is permanent and removes all associated traces, evaluations, models, datasets, and other resources from your W\&B account.
</Warning>

To delete a project from your W\&B account:

1. Navigate to [wandb.ai](https://wandb.ai).
2. Select your project from the **Projects** sidebar. This opens the project's overview page, or the **Traces** page if you've already logged traces to the project.
3. Select the **Project** (<Icon icon="circle-info" />) button in the sidebar to open the project's overview page if it isn't already open.
4. Select the **action (<Icon icon="ellipsis" />)** menu and choose **Delete Project** from the list. This opens the **Delete project** modal dialog.
5. In the modal, enter the project name in the confirmation field and select **Delete**. W\&B permanently deletes the project from your account.


# Set up a self-managed W&B Weave instance
Source: https://docs.wandb.ai/weave/guides/platform/weave-self-managed

Deploy and manage Weave on your own infrastructure

Self-hosting W\&B Weave gives you more control over its environment and configuration. This can help you create a more isolated environment and meet additional security compliance. This document guides you through how to deploy the components required to run W\&B Weave in a self-managed environment using the Altinity ClickHouse Operator. By the end, you'll have a production-grade Weave instance running on your own Kubernetes cluster, backed by a replicated ClickHouse database and S3-compatible object storage. This guide is for Kubernetes administrators and platform engineers who are responsible for deploying and operating W\&B in their organization.

Self-managed Weave deployments rely on [ClickHouseDB](https://clickhouse.com/) to manage its backend. This deployment uses:

* **Altinity ClickHouse Operator**: Enterprise-grade ClickHouse management for Kubernetes.
* **ClickHouse Keeper**: Distributed coordination service (replaces ZooKeeper).
* **ClickHouse Cluster**: High-availability database cluster for trace storage.
* **S3-compatible storage**: Object storage for ClickHouse data persistence.

<Tip>
  For a detailed reference architecture, see [W\&B Self-Managed Reference Architecture](https://docs.wandb.ai/guides/hosting/self-managed/ref-arch/#models-and-weave).
</Tip>

## Important setup notes

The configuration examples in this guide are for reference only. Because each organization's Kubernetes environment is unique, your self-hosted instance likely requires you to adjust:

* **Security and compliance**: Security contexts, `runAsUser` or `fsGroup` values, and other security settings according to your organization's security policies and Kubernetes or OpenShift requirements.
* **Resource sizing**: The resource allocations shown are starting points. Consult with your W\&B Solutions Architect team for proper sizing based on your expected trace volume and performance requirements.
* **Infrastructure specifics**: Update storage classes, node selectors, and other infrastructure-specific settings to match your environment.

Treat these configurations as templates, not prescriptive solutions.

## Architecture

The following diagram shows how the W\&B Platform, the ClickHouse cluster, the ClickHouse Keeper coordination service, and S3 storage fit together in a self-managed Weave deployment.

```mermaid theme={null}
graph TD
    A["W&B Platform (wandb)<br/>weave-trace · app/API · console/parquet"] --> B["ClickHouse Cluster"]
    B --> C["ch-server-0"]
    B --> D["ch-server-1"]
    B --> E["ch-server-2"]
    C --> F["ClickHouse Keeper Cluster<br/>keeper-0 <br/>keeper-1 <br/>keeper-2"]
    D --> F
    E --> F
    C --> G["S3 Storage<br/>(AWS/MinIO)"]
    D --> G
    E --> G
```

## Prerequisites

Before you begin, ensure your environment meets the following requirements. Self-managed Weave instances require the following resources:

* **Kubernetes cluster**: Version 1.29 or later.
* **Kubernetes nodes**: Multi-node cluster (minimum 3 nodes recommended for high availability).
* **Storage class**: A working StorageClass for persistent volumes (for example, `gp3`, `standard`, or `nfs-csi`).
* **S3 bucket**: Pre-configured S3 or S3-compatible bucket with appropriate access permissions.
* **W\&B Platform**: Already installed and running. See the [W\&B Self-Managed Deployment Guide](https://docs.wandb.ai/guides/hosting/hosting-options/self-managed/).
* **W\&B license**: Weave-enabled license from W\&B Support.

<Warning>
  Don't make sizing decisions based on this prerequisites list alone. Resource needs vary based on trace volume and usage patterns. For more information, see [Resource requirements](#resource-requirements).
</Warning>

### Required tools

To set up your instance, you need the following tools:

* `kubectl` configured with cluster access.
* `helm` version 3.0 or later.
* AWS credentials (if using S3) or access to S3-compatible storage.

### Network requirements

Your Kubernetes cluster requires the following network setup:

* Pods in the `clickhouse` namespace must communicate with pods in the `wandb` namespace.
* ClickHouse nodes must communicate with each other on ports `8123`, `9000`, `9009`, and `2181`.

## Deploy your self-managed Weave instance

The following steps walk you through deploying the operator, preparing storage, deploying ClickHouse Keeper and the ClickHouse cluster, and enabling Weave in the W\&B Platform. Complete the steps in order, since each one builds on the resources created in the previous step.

### Deploy the Altinity ClickHouse Operator

The Altinity ClickHouse Operator manages ClickHouse installations in Kubernetes. Installing the operator first lets the later steps declare ClickHouse Keeper and ClickHouse cluster resources that the operator reconciles for you.

#### Add the Altinity Helm repository

```bash theme={null}
helm repo add altinity https://helm.altinity.com
helm repo update
```

#### Create the operator configuration

Create a file named `ch-operator.yaml`. This file defines the security context and metadata for the operator deployment:

```yaml theme={null}
operator:
  image:
    repository: altinity/clickhouse-operator

  # Security context - adjust according to your cluster's requirements
  containerSecurityContext:
    runAsGroup: 0
    runAsNonRoot: true
    runAsUser: 10001 # Update based on your OpenShift/Kubernetes security policies
    allowPrivilegeEscalation: false
    capabilities:
      drop:
        - ALL
    privileged: false
    readOnlyRootFilesystem: false

metrics:
  enabled: false

# Name override - customize if needed
nameOverride: "wandb"
```

The `containerSecurityContext` values shown here work for most Kubernetes distributions. For OpenShift, you may need to adjust `runAsUser` and `fsGroup` to match your project's assigned UID range.

#### Install the operator

```bash theme={null}
helm upgrade --install ch-operator altinity/altinity-clickhouse-operator \
  --namespace clickhouse \
  --create-namespace \
  -f ch-operator.yaml
```

#### Verify the operator installation

```bash theme={null}
# Check operator pod is running
kubectl get pods -n clickhouse

# Expected output:
# NAME                                 READY   STATUS    RESTARTS   AGE
# ch-operator-wandb-xxxxx              1/1     Running   0          30s

# Verify operator image version
kubectl get pods -n clickhouse -o jsonpath="{.items[*].spec.containers[*].image}" | \
  tr ' ' '\n' | grep -v 'metrics-exporter' | sort -u

# Expected output:
# altinity/clickhouse-operator:0.25.4
```

With the operator running, you can now provision the persistent storage and coordination services that the ClickHouse cluster depends on.

### Prepare S3 storage

ClickHouse requires S3 or S3-compatible storage for data persistence. In this step you create the bucket and configure how ClickHouse authenticates to it.

#### Create an S3 bucket

Create an S3 bucket in your AWS account or S3-compatible storage provider. Replace `[BUCKET-NAME]` with your bucket name and `[REGION]` with your AWS region:

```bash theme={null}
# Example for AWS
aws s3 mb s3://[BUCKET-NAME] --region [REGION]
```

#### Configure S3 credentials

ClickHouse requires credentials to read from and write to the bucket. You have two options for providing S3 access credentials. W\&B recommends Option A (IRSA) on AWS because it avoids storing long-lived secrets in the cluster.

##### Option A: Use AWS IAM roles (IRSA, recommended for AWS)

If your Kubernetes nodes have an IAM role with S3 access, ClickHouse can use the EC2 instance metadata:

```yaml theme={null}
# In ch-server.yaml, set:
<use_environment_credentials>true</use_environment_credentials>
```

Required IAM policy (attached to your node IAM role):

```json theme={null}
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:PutObject",
        "s3:DeleteObject",
        "s3:ListBucket"
      ],
      "Resource": [
        "arn:aws:s3:::[BUCKET-NAME]",
        "arn:aws:s3:::[BUCKET-NAME]/*"
      ]
    }
  ]
}
```

##### Option B: Use access keys

If you prefer using static credentials, create a Kubernetes secret:

Replace `[ACCESS-KEY]` with your AWS access key and `[SECRET-KEY]` with your AWS secret key:

```bash theme={null}
kubectl create secret generic aws-creds \
  --namespace clickhouse \
  --from-literal aws_access_key=[ACCESS-KEY] \
  --from-literal aws_secret_key=[SECRET-KEY]
```

Then configure ClickHouse to use the secret (see the ch-server.yaml configuration in Step 4).

### Deploy ClickHouse Keeper

[ClickHouse Keeper](https://clickhouse.com/docs/guides/sre/keeper/clickhouse-keeper) provides the coordination system for data replication and distributed DDL queries execution. You must deploy Keeper before the ClickHouse cluster, since the ClickHouse servers in Step 4 connect to Keeper at startup.

#### Create the Keeper configuration

Create a file named `ch-keeper.yaml`. This manifest defines a three-replica Keeper cluster with anti-affinity, persistent storage, and the settings used by the Altinity operator to provision Keeper pods:

```yaml theme={null}
apiVersion: "clickhouse-keeper.altinity.com/v1"
kind: "ClickHouseKeeperInstallation"
metadata:
  name: wandb
  namespace: clickhouse
  annotations: {}
spec:
  defaults:
    templates:
      podTemplate: default
      dataVolumeClaimTemplate: default

  templates:
    podTemplates:
      - name: keeper
        metadata:
          labels:
            app: clickhouse-keeper
        spec:
          # Pod security context - adjust according to your environment
          securityContext:
            fsGroup: 10001 # Update based on your cluster's security requirements
            fsGroupChangePolicy: Always
            runAsGroup: 0
            runAsNonRoot: true
            runAsUser: 10001 # For OpenShift, use your project's assigned UID range
            seccompProfile:
              type: RuntimeDefault

          # Anti-affinity to spread keepers across nodes (recommended for HA)
          # Customize or remove based on your cluster size and availability requirements
          affinity:
            podAntiAffinity:
              requiredDuringSchedulingIgnoredDuringExecution:
                - labelSelector:
                    matchExpressions:
                      - key: "app"
                        operator: In
                        values:
                          - clickhouse-keeper
                  topologyKey: "kubernetes.io/hostname"

          containers:
            - name: clickhouse-keeper
              imagePullPolicy: IfNotPresent
              image: "clickhouse/clickhouse-keeper:25.10"
              # Resource requests - example values, adjust based on workload
              resources:
                requests:
                  memory: "256Mi"
                  cpu: "0.5"
                limits:
                  memory: "2Gi"
                  cpu: "1"

              securityContext:
                allowPrivilegeEscalation: false
                capabilities:
                  drop:
                    - ALL
                privileged: false
                readOnlyRootFilesystem: false

    volumeClaimTemplates:
      - name: data
        metadata:
          labels:
            app: clickhouse-keeper
        spec:
          storageClassName: gp3 # Change to your StorageClass
          accessModes:
            - ReadWriteOnce
          resources:
            requests:
              storage: 10Gi

  configuration:
    clusters:
      - name: keeper # Keeper cluster name - used in service DNS naming
        layout:
          replicasCount: 3
        templates:
          podTemplate: keeper
          dataVolumeClaimTemplate: data

    settings:
      logger/level: "information"
      logger/console: "true"
      listen_host: "0.0.0.0"
      keeper_server/four_letter_word_white_list: "*"
      keeper_server/coordination_settings/raft_logs_level: "information"
      keeper_server/enable_ipv6: "false"
      keeper_server/coordination_settings/async_replication: "true"
```

Important configuration updates:

* **StorageClass**: Update `storageClassName: gp3` to match your cluster's available StorageClass.
* **Security context**: Adjust `runAsUser` and `fsGroup` values to comply with your organization's security policies.
* **Anti-affinity**: Customize or remove the `affinity` section based on your cluster topology and HA requirements.
* **Resources**: The CPU and memory values are examples. Consult with W\&B Solutions Architects for proper sizing.
* **Naming**: If you change `metadata.name` or `configuration.clusters[0].name`, you must update the Keeper hostnames in `ch-server.yaml` (Step 4) to match.

#### Deploy ClickHouse Keeper resources

```bash theme={null}
kubectl apply -f ch-keeper.yaml
```

#### Verify the Keeper deployment

```bash theme={null}
# Check Keeper pods
kubectl get pods -n clickhouse -l app=clickhouse-keeper

# Expected output:
# NAME                     READY   STATUS    RESTARTS   AGE
# chk-wandb-keeper-0-0-0   1/1     Running   0          2m
# chk-wandb-keeper-0-1-0   1/1     Running   0          2m
# chk-wandb-keeper-0-2-0   1/1     Running   0          2m

# Check Keeper services
kubectl get svc -n clickhouse | grep keeper

# Expected to see keeper services on port 2181
```

With Keeper running, you can now deploy the ClickHouse cluster that uses it for coordination.

### Deploy the ClickHouse cluster

Now deploy the ClickHouse server cluster that stores Weave trace data. This is the largest step in the guide, since the cluster connects to both the Keeper service from Step 3 and the S3 bucket from Step 2.

#### Create the ClickHouse server configuration

Create a file named `ch-server.yaml`. This manifest declares the ClickHouse cluster, its connection to Keeper, the Weave user account, and the S3 storage policy used for trace data:

```yaml theme={null}
apiVersion: "clickhouse.altinity.com/v1"
kind: "ClickHouseInstallation"
metadata:
  name: wandb
  namespace: clickhouse
  annotations: {}
spec:
  defaults:
    templates:
      podTemplate: default
      dataVolumeClaimTemplate: default

  templates:
    podTemplates:
      - name: clickhouse
        metadata:
          labels:
            app: clickhouse-server
        spec:
          # Pod security context - customize for your environment
          securityContext:
            fsGroup: 10001 # Adjust based on your security policies
            fsGroupChangePolicy: Always
            runAsGroup: 0
            runAsNonRoot: true
            runAsUser: 10001 # For OpenShift, use assigned UID range
            seccompProfile:
              type: RuntimeDefault

          # Anti-affinity rule - ensures servers run on different nodes (optional but recommended)
          # Adjust or remove based on your cluster size and requirements
          affinity:
            podAntiAffinity:
              requiredDuringSchedulingIgnoredDuringExecution:
                - labelSelector:
                    matchExpressions:
                      - key: "app"
                        operator: In
                        values:
                          - clickhouse-server
                  topologyKey: "kubernetes.io/hostname"

          containers:
            - name: clickhouse
              image: clickhouse/clickhouse-server:25.10
              # Example resource allocation - adjust based on workload
              resources:
                requests:
                  memory: 1Gi
                  cpu: 1
                limits:
                  memory: 16Gi
                  cpu: 4

              # AWS credentials (remove this section if using IRSA)
              env:
                - name: AWS_ACCESS_KEY_ID
                  valueFrom:
                    secretKeyRef:
                      name: aws-creds
                      key: aws_access_key
                - name: AWS_SECRET_ACCESS_KEY
                  valueFrom:
                    secretKeyRef:
                      name: aws-creds
                      key: aws_secret_key

              securityContext:
                allowPrivilegeEscalation: false
                capabilities:
                  drop:
                    - ALL
                privileged: false
                readOnlyRootFilesystem: false

    volumeClaimTemplates:
      - name: data
        metadata:
          labels:
            app: clickhouse-server
        spec:
          accessModes:
            - ReadWriteOnce
          resources:
            requests:
              storage: 50Gi
          storageClassName: gp3 # Change to your StorageClass

  configuration:
    # Keeper (ZooKeeper) configuration
    # IMPORTANT: These hostnames MUST match your Keeper deployment from Step 3
    zookeeper:
      nodes:
        - host: chk-wandb-keeper-0-0.clickhouse.svc.cluster.local
          port: 2181
        - host: chk-wandb-keeper-0-1.clickhouse.svc.cluster.local
          port: 2181
        - host: chk-wandb-keeper-0-2.clickhouse.svc.cluster.local
          port: 2181
      # Optional: Uncomment to adjust timeouts if needed
      # session_timeout_ms: 30000
      # operation_timeout_ms: 10000

    # Users configuration: https://clickhouse.com/docs/operations/configuration-files#user-settings
    # For production, use a SHA-256 hashed password instead of plain text:
    # printf "your-password" | sha256sum
    # Then use: weave/password_sha256_hex: <hash> instead of weave/password
    users:
      weave/password: [WEAVE-PASSWORD]  # Replace with a strong password before deploying
      weave/access_management: 1
      weave/profile: default
      weave/networks/ip:
        - "0.0.0.0/0"
        - "::"

    # Server settings
    settings:
      disable_internal_dns_cache: 1

    # Cluster configuration
    clusters:
      - name: weavecluster # Cluster name - can be customized but must match wandb-cr.yaml
        layout:
          shardsCount: 1
          replicasCount: 3 # Number of replicas - adjust based on HA requirements
        templates:
          podTemplate: clickhouse
          dataVolumeClaimTemplate: data

    # Configuration files
    files:
      config.d/network_configuration.xml: |
        <clickhouse>
            <listen_host>0.0.0.0</listen_host>
            <listen_host>::</listen_host>
        </clickhouse>

      config.d/logger.xml: |
        <clickhouse>
            <logger>
                <level>information</level>
            </logger>
        </clickhouse>

      config.d/storage_configuration.xml: |
        <clickhouse>
            <storage_configuration>
                <disks>
                    <s3_disk>
                        <type>s3</type>
                        <!-- Update with your S3 bucket endpoint and region -->
                        <endpoint>https://[BUCKET-NAME].s3.[REGION].amazonaws.com/s3_disk/{replica}</endpoint>
                        <metadata_path>/var/lib/clickhouse/disks/s3_disk/</metadata_path>
                        <use_environment_credentials>true</use_environment_credentials>
                        <region>[REGION]</region>
                    </s3_disk>
                    <s3_disk_cache>
                        <type>cache</type>
                        <disk>s3_disk</disk>
                        <path>/var/lib/clickhouse/s3_disk_cache/cache/</path>
                        <!-- Cache size MUST be smaller than persistent volume -->
                        <max_size>40Gi</max_size>
                        <cache_on_write_operations>true</cache_on_write_operations>
                    </s3_disk_cache>
                </disks>
                <policies>
                    <s3_main>
                        <volumes>
                            <main>
                                <disk>s3_disk_cache</disk>
                            </main>
                        </volumes>
                    </s3_main>
                </policies>
            </storage_configuration>
            <merge_tree>
                <storage_policy>s3_main</storage_policy>
            </merge_tree>
        </clickhouse>
```

Critical configuration updates required:

1. **StorageClass**: Update `storageClassName: gp3` to match your cluster's StorageClass.
2. **S3 endpoint**: Replace `[BUCKET-NAME]` and `[REGION]` with your actual values.
3. **Cache size**: The `<max_size>40Gi</max_size>` must be smaller than the persistent volume size (50Gi).
4. **Security context**: Adjust `runAsUser`, `fsGroup`, and other security settings to match your organization's policies.
5. **Resource allocation**: The CPU and memory values are examples. Consult with your W\&B Solutions Architect for proper sizing based on your expected trace volume.
6. **Anti-affinity rules**: Customize or remove based on your cluster topology and high-availability needs.
7. **Keeper hostnames**: The Keeper node hostnames must match your Keeper deployment naming from Step 3 (see "Keeper naming").
8. **Cluster naming**: The cluster name `weavecluster` can be changed, but it must match the `WF_CLICKHOUSE_REPLICATED_CLUSTER` value in Step 5.
9. **Credentials**:
   * For IRSA: Keep `<use_environment_credentials>true</use_environment_credentials>` or access your secret keys mapped to environment variables.

#### Update the S3 configuration

Edit the `storage_configuration.xml` section in `ch-server.yaml`.

Example for AWS S3:

```xml theme={null}
<endpoint>https://my-wandb-clickhouse.s3.eu-central-1.amazonaws.com/s3_disk/{replica}</endpoint>
<region>eu-central-1</region>
```

Example for MinIO:

```xml theme={null}
<endpoint>https://minio.example.com:9000/my-bucket/s3_disk/{replica}</endpoint>
<region>us-east-1</region>
```

<Warning>
  Do not remove `{replica}`. This ensures each ClickHouse replica writes to its own folder in the bucket.
</Warning>

#### Configure credentials (Option B only)

If using Option B (access keys) from Step 2, ensure the `env` section in `ch-server.yaml` references the secret:

```yaml theme={null}
env:
  - name: AWS_ACCESS_KEY_ID
    valueFrom:
      secretKeyRef:
        name: aws-creds
        key: aws_access_key
  - name: AWS_SECRET_ACCESS_KEY
    valueFrom:
      secretKeyRef:
        name: aws-creds
        key: aws_secret_key
```

If using Option A (IRSA), remove the entire `env` section.

#### Keeper naming

Getting the Keeper hostnames right is critical. If they don't match the services created in Step 3, ClickHouse won't start. The Keeper node hostnames in the `zookeeper.nodes` section follow a specific pattern based on your Keeper deployment from Step 3.

Hostname pattern: `chk-[INSTALLATION-NAME]-[CLUSTER-NAME]-[CLUSTER-INDEX]-[REPLICA-INDEX].[NAMESPACE].svc.cluster.local`

Where:

* `chk` is the ClickHouseKeeperInstallation prefix (fixed).
* `[INSTALLATION-NAME]` is the `metadata.name` from `ch-keeper.yaml` (for example, `wandb`).
* `[CLUSTER-NAME]` is the `configuration.clusters[0].name` from `ch-keeper.yaml` (for example, `keeper`).
* `[CLUSTER-INDEX]` is the cluster index, typically `0` for a single cluster.
* `[REPLICA-INDEX]` is the replica number: `0`, `1`, or `2` for 3 replicas.
* `[NAMESPACE]` is the Kubernetes namespace (for example, `clickhouse`).

Example with default names:

```text theme={null}
chk-wandb-keeper-0-0.clickhouse.svc.cluster.local
chk-wandb-keeper-0-1.clickhouse.svc.cluster.local
chk-wandb-keeper-0-2.clickhouse.svc.cluster.local
```

If you customize the Keeper installation name (for example, `metadata.name: myweave`):

```text theme={null}
chk-myweave-keeper-0-0.clickhouse.svc.cluster.local
chk-myweave-keeper-0-1.clickhouse.svc.cluster.local
chk-myweave-keeper-0-2.clickhouse.svc.cluster.local
```

If you customize the Keeper cluster name (for example, `clusters[0].name: coordination`):

```text theme={null}
chk-wandb-coordination-0-0.clickhouse.svc.cluster.local
chk-wandb-coordination-0-1.clickhouse.svc.cluster.local
chk-wandb-coordination-0-2.clickhouse.svc.cluster.local
```

To verify your actual Keeper hostnames:

```bash theme={null}
# List Keeper services to see the actual names
kubectl get svc -n clickhouse | grep keeper

# List Keeper pods to confirm the naming pattern
kubectl get pods -n clickhouse -l app=clickhouse-keeper
```

<Note>
  The Keeper hostnames in `ch-server.yaml` must exactly match the actual service names created by the Keeper deployment, or ClickHouse servers fail to connect to the coordination service.
</Note>

#### Deploy the ClickHouse cluster resources

```bash theme={null}
kubectl apply -f ch-server.yaml
```

#### Verify the ClickHouse deployment

```bash theme={null}
# Check ClickHouse pods
kubectl get pods -n clickhouse -l app=clickhouse-server

# Expected output:
# NAME                           READY   STATUS    RESTARTS   AGE
# chi-wandb-weavecluster-0-0-0   1/1     Running   0          3m
# chi-wandb-weavecluster-0-1-0   1/1     Running   0          3m
# chi-wandb-weavecluster-0-2-0   1/1     Running   0          3m

# Test ClickHouse connectivity
kubectl exec -n clickhouse chi-wandb-weavecluster-0-0-0 -- \
  clickhouse-client --user weave --password [WEAVE-PASSWORD] --query "SELECT version()"

# Check cluster status
kubectl exec -n clickhouse chi-wandb-weavecluster-0-0-0 -- \
  clickhouse-client --user weave --password [WEAVE-PASSWORD] --query \
  "SELECT cluster, host_name, port FROM system.clusters WHERE cluster='weavecluster'"
```

At this point you have a running ClickHouse cluster backed by Keeper and S3. The remaining steps connect the W\&B Platform to that cluster and confirm that Weave traces flow end-to-end.

### Enable Weave in the W\&B Platform

Now configure the W\&B Platform to use the ClickHouse cluster for Weave traces. This step informs the W\&B operator where to find your externally managed ClickHouse and turns on the `weave-trace` service.

#### Gather ClickHouse connection information

You'll need:

* **Host**: `clickhouse-wandb.clickhouse.svc.cluster.local`
* **Port**: `8123`
* **User**: `weave` (as configured in `ch-server.yaml`)
* **Password**: The password you set in `ch-server.yaml`
* **Database**: `weave` (created automatically)
* **Cluster name**: `weavecluster` (as configured in `ch-server.yaml`)

The host name follows this pattern: `clickhouse-[INSTALLATION-NAME].[NAMESPACE].svc.cluster.local`

#### Update the W\&B Custom Resource

Edit your W\&B Platform Custom Resource (CR) to add Weave configuration:

```yaml theme={null}
apiVersion: apps.wandb.com/v1
kind: WeightsAndBiases
metadata:
  name: wandb
  namespace: wandb
spec:
  values:
    global:
      # ... existing configuration ...

      # Add ClickHouse configuration
      clickhouse:
        install: false # We deployed it separately
        host: clickhouse-wandb.clickhouse.svc.cluster.local
        port: 8123
        user: weave
        password: [WEAVE-PASSWORD]
        database: weave
        replicated: true # REQUIRED for multi-replica setup

      # Enable Weave Trace
      weave-trace:
        enabled: true

    # Weave Trace configuration
    weave-trace:
      install: true
      extraEnv:
        WF_CLICKHOUSE_REPLICATED: "true"
        WF_CLICKHOUSE_REPLICATED_CLUSTER: "weavecluster"
      image:
        repository: wandb/weave-trace
        tag: 0.74.1
      replicaCount: 1
      size: "default"
      sizing:
        default:
          autoscaling:
            horizontal:
              enabled: false
          # Example resource allocation - adjust based on workload
          resources:
            limits:
              cpu: 4
              memory: "8Gi"
            requests:
              cpu: 1
              memory: "4Gi"
      # Pod security context - customize for your environment
      podSecurityContext:
        fsGroup: 10001 # Adjust based on your security requirements
        fsGroupChangePolicy: Always
        runAsGroup: 0
        runAsNonRoot: true
        runAsUser: 10001 # For OpenShift, use assigned UID range
        seccompProfile:
          type: RuntimeDefault
      # Container security context
      securityContext:
        allowPrivilegeEscalation: false
        capabilities:
          drop:
            - ALL
        privileged: false
        readOnlyRootFilesystem: false
```

Critical settings:

* `clickhouse.replicated: true`: Required when using 3 replicas.
* `WF_CLICKHOUSE_REPLICATED: "true"`: Required for replicated setup.
* `WF_CLICKHOUSE_REPLICATED_CLUSTER: "weavecluster"`: Must match the cluster name in `ch-server.yaml`.

<Note>
  The security contexts, resource allocations, and other Kubernetes-specific configurations shown here are reference examples. Customize them according to your organization's requirements and consult with your W\&B Solutions Architect team for proper resource sizing.
</Note>

#### Apply the updated configuration

```bash theme={null}
kubectl apply -f wandb-cr.yaml
```

#### Verify the Weave Trace deployment

```bash theme={null}
# Check weave-trace pod status
kubectl get pods -n wandb | grep weave-trace

# Expected output:
# wandb-weave-trace-bc-xxxxx   1/1     Running   0          2m

# Check weave-trace logs for ClickHouse connection
kubectl logs -n wandb [WEAVE-TRACE-POD-NAME] --tail=50

# Look for successful ClickHouse connection messages
```

### Initialize the Weave database

The weave-trace service automatically creates the required database schema on first startup. In this step you confirm that the migration completed successfully before exposing Weave to end users.

#### Monitor the database migration

```bash theme={null}
# Watch weave-trace logs during startup
kubectl logs -n wandb [WEAVE-TRACE-POD-NAME] -f

# Look for migration messages indicating successful database initialization
```

#### Verify database creation

```bash theme={null}
# Connect to ClickHouse and check database
kubectl exec -n clickhouse chi-wandb-weavecluster-0-0-0 -- \
  clickhouse-client --user weave --password [WEAVE-PASSWORD] --query \
  "SHOW DATABASES"

# Expected to see 'weave' database listed

# Check tables in weave database
kubectl exec -n clickhouse chi-wandb-weavecluster-0-0-0 -- \
  clickhouse-client --user weave --password [WEAVE-PASSWORD] --query \
  "SHOW TABLES FROM weave"
```

### Verify that Weave is enabled

This final step confirms that Weave is licensed, reachable from the W\&B Console, and able to record traces from a client SDK.

#### Access the W\&B Console

Navigate to your W\&B instance URL in a web browser.

#### Check the Weave license status

In the W\&B Console:

1. Go to **Top Right Menu** > **Organization Dashboard**.
2. Verify that **Weave access** is enabled.

#### Test Weave functionality

Create a Python test to verify that Weave is working:

```python theme={null}
import os
import weave

# Point Weave at your self-managed W&B instance
os.environ["WANDB_BASE_URL"] = "https://[WANDB-HOST]"  # Replace with your W&B URL

weave.init('test-project')

# Create a simple traced function
@weave.op()
def hello_weave(name: str) -> str:
    return f"Hello, {name}!"

# Call the function
result = hello_weave("World")
print(result)
```

After running this, check your W\&B UI for traces at the traces page in your organization. When the trace appears, your self-managed Weave deployment is operational.

## Troubleshooting

The following sections describe common deployment problems and how to resolve them, grouped by the component where the symptom first appears.

### ClickHouse Keeper issues

**Problem**: Keeper pods stuck in `Pending` state

**Solution**: Check multiple possible causes:

1. **PVC and StorageClass issues**:

```bash theme={null}
kubectl get pvc -n clickhouse
kubectl describe pvc -n clickhouse
```

Ensure your StorageClass is configured correctly and has available capacity.

2. **Anti-affinity and node availability**:

```bash theme={null}
# Check if anti-affinity rules prevent scheduling
kubectl describe pod -n clickhouse [POD-NAME] | grep -A 10 "Events:"

# Check available nodes and their resources
kubectl get nodes
kubectl describe nodes | grep -A 5 "Allocated resources"
```

Common issues:

* Anti-affinity requires 3 separate nodes, but the cluster has fewer nodes.
* Nodes don't have sufficient CPU or memory to meet pod requests.
* Node taints prevent pod scheduling.

**Solutions**:

* Remove or adjust anti-affinity rules if you have fewer than 3 nodes.
* Use `preferredDuringSchedulingIgnoredDuringExecution` instead of `requiredDuringSchedulingIgnoredDuringExecution` for softer anti-affinity.
* Reduce resource requests if nodes are constrained.
* Add more nodes to your cluster.

***

**Problem**: Keeper pods in `CrashLoopBackOff`

**Solution**: Check logs and verify configuration:

```bash theme={null}
kubectl logs -n clickhouse [KEEPER-POD-NAME]
```

Common issues:

* Incorrect security context (check `runAsUser` and `fsGroup`).
* Volume permission issues.
* Port conflicts.
* Configuration errors in `ch-keeper.yaml`.

### ClickHouse server issues

**Problem**: ClickHouse can't connect to S3

**Solution**: Verify S3 credentials and permissions:

```bash theme={null}
# Check if secret exists (if using access keys)
kubectl get secret aws-creds -n clickhouse

# Check ClickHouse logs for S3 errors
kubectl logs -n clickhouse [CLICKHOUSE-POD-NAME] | grep -i s3

# Verify S3 endpoint in storage configuration
kubectl get chi wandb -n clickhouse -o yaml | grep -A 10 storage_configuration
```

***

**Problem**: ClickHouse can't connect to Keeper

**Solution**: Verify Keeper endpoints and naming:

```bash theme={null}
# Check Keeper services and their actual names
kubectl get svc -n clickhouse | grep keeper

# Check Keeper pods to confirm naming pattern
kubectl get pods -n clickhouse -l app=clickhouse-keeper

# Compare with zookeeper.nodes configuration in ch-server.yaml
# The hostnames MUST match the actual service names

# Check ClickHouse logs for connection errors
kubectl logs -n clickhouse chi-wandb-weavecluster-0-0-0 | grep -i keeper
```

If the connection fails, the Keeper hostnames in `ch-server.yaml` likely don't match your actual Keeper deployment. See "Keeper naming" in Step 4 for the naming pattern.

### Weave Trace issues

**Problem**: `weave-trace` pod fails to start

**Solution**: Check ClickHouse connectivity:

```bash theme={null}
# Get weave-trace pod name
kubectl get pods -n wandb | grep weave-trace

# Check weave-trace logs
kubectl logs -n wandb [WEAVE-TRACE-POD-NAME]

# Common error: "connection refused" or "authentication failed"
# Verify ClickHouse credentials in wandb-cr.yaml match ch-server.yaml
```

***

**Problem**: Weave not showing as enabled in Console

**Solution**: Verify configuration:

1. Check license includes Weave:

   ```bash theme={null}
   kubectl get secret license-key -n wandb -o jsonpath='{.data.value}' | base64 -d | jq
   ```

2. Ensure that `weave-trace.enabled: true` and `clickhouse.replicated: true` are set in `wandb-cr.yaml`.

3. Check W\&B operator logs:
   ```bash theme={null}
   kubectl logs -n wandb deployment/wandb-controller-manager
   ```

***

**Problem**: Database migration fails

**Solution**: Check cluster name matches:

The `WF_CLICKHOUSE_REPLICATED_CLUSTER` environment variable must match the cluster name in `ch-server.yaml`:

```yaml theme={null}
# In ch-server.yaml:
clusters:
  - name: weavecluster # <-- This name

# Must match in wandb-cr.yaml:
weave-trace:
  extraEnv:
    WF_CLICKHOUSE_REPLICATED_CLUSTER: "weavecluster" # <-- This value
```

## Resource requirements

This section provides example resource allocations for two common deployment profiles. Use them as starting points when planning your cluster, and refine the numbers based on your observed workload.

<Warning>
  The resource allocations in this section are example starting points. Actual requirements vary based on:

  * Trace import volume (traces per second)
  * Query patterns and concurrency
  * Data retention period
  * Number of concurrent users

  Always consult with your W\&B Solutions Architect team to determine appropriate sizing for your specific use case. Under-provisioned resources can lead to performance issues, while over-provisioning wastes infrastructure costs.
</Warning>

### Minimum production setup

| Component         | Replicas   | CPU (request, limit) | Memory (request, limit) | Storage   |
| ----------------- | ---------- | -------------------- | ----------------------- | --------- |
| ClickHouse Keeper | 3          | 0.5, 1               | 256Mi, 2Gi              | 10Gi each |
| ClickHouse Server | 3          | 1, 4                 | 1Gi, 16Gi               | 50Gi each |
| Weave Trace       | 1          | 1, 4                 | 4Gi, 8Gi                | -         |
| **Total**         | **7 pods** | **\~4.5, 15 CPU**    | **\~7.8Gi, 58Gi**       | **180Gi** |

Suitable for development, testing, or low-volume production environments.

### Recommended production setup

For production workloads with high trace volume:

| Component         | Replicas        | CPU (request, limit)       | Memory (request, limit)        | Storage    |
| ----------------- | --------------- | -------------------------- | ------------------------------ | ---------- |
| ClickHouse Keeper | 3               | 1, 2                       | 1Gi, 4Gi                       | 20Gi each  |
| ClickHouse Server | 3               | 1, 16                      | 8Gi, 64Gi                      | 200Gi each |
| Weave Trace       | 2 to 3          | 1, 4                       | 4Gi, 8Gi                       | -          |
| **Total**         | **8 to 9 pods** | **\~6 to 9, 52 to 64 CPU** | **\~27 to 33Gi, 204 to 216Gi** | **660Gi**  |

Suitable for high-volume production environments.

For ultra-high volume deployments, contact your W\&B Solutions Architect team for custom sizing recommendations based on your specific trace volume and performance requirements.

## Advanced configuration

This section covers customization options for self-managed Weave deployments, including scaling ClickHouse capacity through vertical scaling or horizontal scaling, updating ClickHouse versions by modifying image tags in both keeper and server configurations, and monitoring ClickHouse health.

W\&B recommends consulting with your W\&B Solutions Architect team when making advanced changes to your instance to ensure that they align with your performance and reliability requirements.

### Scale ClickHouse

To increase ClickHouse capacity, you can:

1. **Vertical scaling**: Increase resources per pod (straightforward approach).

   ```yaml theme={null}
   resources:
     requests:
       memory: 8Gi
       cpu: 1
     limits:
       memory: 64Gi
       cpu: 16
   ```

   Recommendation: monitor actual resource usage and scale accordingly. For ultra-high volume deployments, contact your W\&B Solutions Architect team.

2. **Horizontal scaling**: Add more replicas (requires careful planning).
   * Increasing replicas requires data rebalancing.
   * Consult ClickHouse's documentation for shard management.
   * Contact a W\&B Solutions Architect before implementing horizontal scaling in production.

### Use a different ClickHouse version

To use a different ClickHouse version, update the image tag in both `ch-keeper.yaml` and `ch-server.yaml`:

```yaml theme={null}
image: clickhouse/clickhouse-keeper:25.10   # Keeper version
image: clickhouse/clickhouse-server:25.10   # Server version
```

Keeper and server versions should match, or the Keeper version should be greater than or equal to the server version for compatibility.

### Monitor ClickHouse

Access ClickHouse system tables for monitoring:

```bash theme={null}
# Check disk usage
kubectl exec -n clickhouse chi-wandb-weavecluster-0-0-0 -- \
  clickhouse-client --user weave --password [WEAVE-PASSWORD] --query \
  "SELECT name, path, formatReadableSize(free_space) as free, formatReadableSize(total_space) as total FROM system.disks"

# Check replication status
kubectl exec -n clickhouse chi-wandb-weavecluster-0-0-0 -- \
  clickhouse-client --user weave --password [WEAVE-PASSWORD] --query \
  "SELECT database, table, is_leader, total_replicas, active_replicas FROM system.replicas WHERE database='weave'"

# Check ClickHouse server status
kubectl get pods -n clickhouse -l app=clickhouse-server
```

### Backup and recovery

ClickHouse stores data in S3, providing inherent backup capabilities through S3 versioning and bucket replication features. For backup strategies specific to your deployment, consult with your W\&B Solutions Architect team and refer to the [ClickHouse backup documentation](https://clickhouse.com/docs/en/operations/backup).

## Security considerations

Production deployments should harden the defaults shown in this guide. The following list highlights the most important areas to review with your security team.

1. **Credentials**: Store ClickHouse passwords in Kubernetes secrets, not plain text.
2. **Network policies**: Consider implementing NetworkPolicies to restrict ClickHouse access.
3. **RBAC**: Ensure service accounts have minimal required permissions.
4. **S3 bucket**: Enable encryption at rest and restrict bucket access to necessary IAM roles.
5. **TLS**: Optional. For production, enable TLS for ClickHouse client connections.

## Upgrade

The following procedures cover routine upgrades for the operator, ClickHouse server, and Weave Trace components. Upgrade one component at a time and confirm that the deployment is healthy before moving on.

### Upgrade the ClickHouse Operator

```bash theme={null}
helm upgrade ch-operator altinity/altinity-clickhouse-operator \
  --namespace clickhouse \
  -f ch-operator.yaml
```

### Upgrade ClickHouse Server

Update the image version in `ch-server.yaml` and apply:

```bash theme={null}
# Edit ch-server.yaml, change image tag
kubectl apply -f ch-server.yaml

# Monitor the pods
kubectl get pods -n clickhouse
```

### Upgrade Weave Trace

Update the image tag in `wandb-cr.yaml` and apply:

```bash theme={null}
kubectl apply -f wandb-cr.yaml

# Monitor weave-trace pod restart
kubectl get pods -n wandb | grep weave-trace
```

## Additional resources

* [Altinity ClickHouse Operator Documentation](https://docs.altinity.com/altinitykubernetesoperator/)
* [ClickHouse Documentation](https://clickhouse.com/docs)
* [W\&B Weave Documentation](https://docs.wandb.ai/weave)
* [ClickHouse S3 Storage Configuration](https://clickhouse.com/docs/en/engines/table-engines/mergetree-family/mergetree#s3-virtual-hosted-style)

## Support

For production deployments or issues:

* **W\&B Support**: `support@wandb.com`
* **Solutions Architects**: For ultra-high volume deployments, custom sizing, and deployment planning.
* **Include in support requests**:
  * Logs from `weave-trace`, ClickHouse pods, and the operator.
  * W\&B version, ClickHouse version, and Kubernetes version.
  * Cluster information and trace volume.

## FAQ

**Q: Can I use a single ClickHouse replica instead of 3?**

A: Yes, but it's not recommended for production. Update `replicasCount: 1` in `ch-server.yaml` and set `clickhouse.replicated: false` in `wandb-cr.yaml`.

**Q: Can I use another database instead of ClickHouse?**

A: No, Weave Trace requires ClickHouse for its high-performance columnar storage capabilities.

**Q: How much S3 storage do I need?**

A: S3 storage requirements depend on your trace volume, retention period, and data compression. Monitor your actual usage after deployment and adjust accordingly. ClickHouse's columnar format compresses trace data efficiently.

**Q: Do I need to configure the `database` name in ClickHouse?**

A: No, the weave-trace service creates the `weave` database automatically during initial startup.

**Q: What if my cluster name is not `weavecluster`?**

A: You must set the `WF_CLICKHOUSE_REPLICATED_CLUSTER` environment variable to match your cluster name, otherwise database migrations fail.

**Q: Should I use the exact security contexts shown in the examples?**

A: No. The security contexts such as `runAsUser` and `fsGroup` provided in this guide are reference examples. You must adjust them to comply with your organization's security policies, especially for OpenShift clusters, which have specific UID and GID range requirements.

**Q: How do I know if I've sized my ClickHouse cluster correctly?**

A: Contact your W\&B Solutions Architect team with your expected trace volume and usage patterns. They provide sizing recommendations. Monitor your deployment's resource usage and adjust as needed.

**Q: Can I customize the naming conventions used in the examples?**

A: Yes, but you must maintain consistency across all components:

1. **ClickHouse Keeper names**: Must match the Keeper node hostnames in the `zookeeper.nodes` section of `ch-server.yaml`.
2. **ClickHouse cluster name** (`weavecluster`): Must match `WF_CLICKHOUSE_REPLICATED_CLUSTER` in `wandb-cr.yaml`.
3. **ClickHouse installation name**: Affects the service hostname used by `weave-trace`.

See the "Keeper naming" section in Step 4 for details on the naming pattern and how to verify your actual names.

**Q: What if my cluster uses different anti-affinity requirements?**

A: The anti-affinity rules shown are recommendations for high availability. Adjust or remove them based on your cluster size, topology, and availability requirements. For small clusters or development environments, you might not need anti-affinity rules.


# Python SDK
Source: https://docs.wandb.ai/weave/ref-link-python





# Service API
Source: https://docs.wandb.ai/weave/ref-link-service-api





# TypeScript SDK
Source: https://docs.wandb.ai/weave/ref-link-typescript





# CLI Reference SDK 0.26.1
Source: https://docs.wandb.ai/models/ref/cli

Use the W&B Command Line Interface (CLI) to log in, run jobs, execute sweeps, and more using shell commands

## Usage

```bash theme={null}
wandb [OPTIONS] COMMAND [ARGS]...
```

## Options

| Option      | Description                                 |
| :---------- | :------------------------------------------ |
| `--version` | Show the version and exit. (default: False) |

## Commands

| Command                                            | Description                                                                |
| :------------------------------------------------- | :------------------------------------------------------------------------- |
| [agent](/models/ref/cli/wandb-agent)               | Start a sweep agent.                                                       |
| [artifact](/models/ref/cli/wandb-artifact)         | Upload, download, and manage W\&B artifacts.                               |
| [beta](/models/ref/cli/wandb-beta)                 | Beta versions of wandb CLI commands.                                       |
| [controller](/models/ref/cli/wandb-controller)     | Start a local sweep controller for a W\&B hyperparameter sweep.            |
| [disabled](/models/ref/cli/wandb-disabled)         | Disable W\&B functionality.                                                |
| [docker](/models/ref/cli/wandb-docker)             | Run code in a Docker container with W\&B configured.                       |
| [docker-run](/models/ref/cli/wandb-docker-run)     | Wrap `docker run` and inject W\&B environment variables automatically.     |
| [enabled](/models/ref/cli/wandb-enabled)           | Re-enable W\&B after it was deactivated with `wandb disabled`.             |
| [init](/models/ref/cli/wandb-init)                 | Initialize or update W\&B configuration for the current directory.         |
| [job](/models/ref/cli/wandb-job)                   | Commands for managing and viewing W\&B jobs.                               |
| [launch](/models/ref/cli/wandb-launch)             | Launch or queue a W\&B Job.                                                |
| [launch-agent](/models/ref/cli/wandb-launch-agent) | Run a W\&B launch agent.                                                   |
| [launch-sweep](/models/ref/cli/wandb-launch-sweep) | Run a W\&B launch sweep (Experimental).                                    |
| [login](/models/ref/cli/wandb-login)               | Authenticate your machine with W\&B.                                       |
| [offline](/models/ref/cli/wandb-offline)           | Save data logged to W\&B locally without uploading it to the cloud.        |
| [online](/models/ref/cli/wandb-online)             | Re-enable cloud syncing for W\&B runs.                                     |
| [pull](/models/ref/cli/wandb-pull)                 | Download files from a W\&B run.                                            |
| [purge-cache](/models/ref/cli/wandb-purge-cache)   | Purges cached logs, run history, and artifacts from the local W\&B cache.  |
| [restore](/models/ref/cli/wandb-restore)           | Restore the code, config, or Docker environement from a previous W\&B run. |
| [scheduler](/models/ref/cli/wandb-scheduler)       | Run a W\&B launch sweep scheduler (Experimental).                          |
| [server](/models/ref/cli/wandb-server)             | Commands for operating a local W\&B server                                 |
| [status](/models/ref/cli/wandb-status)             | Display the current W\&B configuration settings.                           |
| [sweep](/models/ref/cli/wandb-sweep)               | Create, update, or manage a hyperparameter sweep.                          |
| [sync](/models/ref/cli/wandb-sync)                 | Upload existing local W\&B run data to the cloud.                          |
| [verify](/models/ref/cli/wandb-verify)             | Run integration checks against a self-hosted W\&B instance.                |


# wandb agent
Source: https://docs.wandb.ai/models/ref/cli/wandb-agent



<GitHubLink />

## Usage

```bash theme={null}
agent [OPTIONS] SWEEP_ID
```

## Description

Start a sweep agent.

Poll the W\&B server for hyperparameter configurations from
the sweep and start a run for each configuration.

The agent exits when the sweep completes, the sweep
is stopped, cancelled, or the `--count` limit is reached.

Find the sweep ID within the Sweeps tab in the W\&B App
or in the output of the `wandb sweep` command when you create a new sweep.

The sweep ID can include the entity and project path
(entity/project/sweep\_id) or the eight character sweep ID.

For example, to start an agent for a sweep with a sweep ID of wbyz9876:

```bash theme={null}
wandb agent wbyz9876
```

To start an agent with a run limit of 10 runs for the sweep:

```bash theme={null}
wandb agent --count 10 wbyz9876
```

To start an agent for a sweep and save it to a project
called "sweeps-project" that belongs to the "team-awesome" entity:

```bash theme={null}
wandb agent -p sweeps-project -e team-awesome wbyz9876
```

To forward signals to child runs for clean shutdown:

```bash theme={null}
wandb agent --forward-signals wbyz9876
```

## Arguments

| Name       | Default | Type |
| ---------- | ------- | ---- |
| `sweep_id` | STR     | True |

## Options

| Flag                    | Type      | Description                                                                                                       |
| ----------------------- | --------- | ----------------------------------------------------------------------------------------------------------------- |
| `--project, -p`         | STR       | Set the project to upload runs to. **Default**: None                                                              |
| `--entity, -e`          | STR       | Set the entity to scope the project to. **Default**: None                                                         |
| `--count`               | INT       | Maximum number of runs this agent will execute. Continues until the sweep completes if not set. **Default**: None |
| `--forward-signals, -f` | BOOL Flag | Forward signals (e.g. SIGINT/SIGTERM) to child runs so they can shut down cleanly. **Default**: False             |


# wandb artifact
Source: https://docs.wandb.ai/models/ref/cli/wandb-artifact



<GitHubLink />

## Usage

```bash theme={null}
artifact [OPTIONS] COMMAND [ARGS]...
```

## Description

Upload, download, and manage W\&B artifacts.

## Subcommands

| Command                | Description                                 |
| ---------------------- | ------------------------------------------- |
| `wandb artifact cache` | Manage the local artifact cache.            |
| `wandb artifact get`   | Download an artifact from W\&B by its path. |
| `wandb artifact ls`    | List all artifacts in a W\&B project.       |
| `wandb artifact put`   | Upload an artifact to W\&B.                 |


# wandb artifact cache
Source: https://docs.wandb.ai/models/ref/cli/wandb-artifact/wandb-artifact-cache



<GitHubLink />

## Usage

```bash theme={null}
cache [OPTIONS] COMMAND [ARGS]...
```

## Description

Manage the local artifact cache.

Cache downloaded artifact files locally to avoid redundant downloads.

Use subcommands to inspect and reclaim disk space used by the cache.

## Subcommands

| Command                        | Description                           |
| ------------------------------ | ------------------------------------- |
| `wandb artifact cache cleanup` | Reduce the local artifact cache size. |


# wandb artifact cache cleanup
Source: https://docs.wandb.ai/models/ref/cli/wandb-artifact/wandb-artifact-cache/wandb-artifact-cache-cleanup



<GitHubLink />

## Usage

```bash theme={null}
cleanup [OPTIONS] TARGET_SIZE
```

## Description

Reduce the local artifact cache size.

Remove the least recently accessed files first until the cache is
at or below the TARGET\_SIZE. TARGET\_SIZE accepts human-readable
formats (for example, 10GB or 500MB).

For example, to reduce the artifact cache to 10 GB:

```bash theme={null}
wandb artifact cache cleanup 10GB
```

To remove temporary files and reduce the artifact cache to 5 GB:

```bash theme={null}
wandb artifact cache cleanup --remove-temp 5GB
```

## Arguments

| Name          | Default | Type |
| ------------- | ------- | ---- |
| `target_size` | STR     | True |

## Options

| Flag            | Type      | Description                                                    |
| --------------- | --------- | -------------------------------------------------------------- |
| `--remove-temp` | BOOL Flag | Also remove temporary files from the cache. **Default**: False |


# wandb artifact get
Source: https://docs.wandb.ai/models/ref/cli/wandb-artifact/wandb-artifact-get



<GitHubLink />

## Usage

```bash theme={null}
get [OPTIONS] PATH
```

## Description

Download an artifact from W\&B by its path.

The PATH format is entity/project/artifact\_name:version. If
the version is omitted, use the "latest" alias.

For example, to download the latest version of an artifact called "processed-training-set"
from the "foobar" project under the "team-awesome" entity:

```bash theme={null}
wandb artifact get team-awesome/foobar/processed-training-set:latest
```

To download a specific version (v2) of the "processed-training-set" artifact
to a local directory (./data):

```bash theme={null}
wandb artifact get --root ./data team-awesome/foobar/processed-training-set:v2
```

## Arguments

| Name   | Default | Type |
| ------ | ------- | ---- |
| `path` | STR     | True |

## Options

| Flag     | Type | Description                                                                                          |
| -------- | ---- | ---------------------------------------------------------------------------------------------------- |
| `--root` | STR  | Directory to download the artifact to. Uses the default artifact cache if not set. **Default**: None |
| `--type` | STR  | Expected artifact type. Fails if the artifact does not match. **Default**: None                      |


# wandb artifact ls
Source: https://docs.wandb.ai/models/ref/cli/wandb-artifact/wandb-artifact-ls



<GitHubLink />

## Usage

```bash theme={null}
ls [OPTIONS] PATH
```

## Description

List all artifacts in a W\&B project.

Display the latest version of each artifact collection in a
project. Show the type, last updated time, size, and name.

The PATH consists of the entity and project (entity/project) to
list artifacts from.

For example, to list all artifacts in a project called "foobar" under
the "team-awesome" entity:

```bash theme={null}
wandb artifact ls team-awesome/foobar
```

To list only artifacts of type "model" in the same project:

```bash theme={null}
wandb artifact ls --type model team-awesome/foobar
```

## Arguments

| Name   | Default | Type |
| ------ | ------- | ---- |
| `path` | STR     | True |

## Options

| Flag         | Type | Description                                 |
| ------------ | ---- | ------------------------------------------- |
| `--type, -t` | STR  | Filter artifacts by type. **Default**: None |


# wandb artifact put
Source: https://docs.wandb.ai/models/ref/cli/wandb-artifact/wandb-artifact-put



<GitHubLink />

## Usage

```bash theme={null}
put [OPTIONS] PATH
```

## Description

Upload an artifact to W\&B.

Upload a file, directory, or URL reference as a versioned artifact.

The PATH can be a local file, a local directory, or a URL
(containing `://`) to log as a reference artifact.

If `--name` is not specified, the artifact name defaults to the
basename of the path. If the project cannot be parsed from the
name, you are prompted to enter one.

For example, to upload all files in a local directory ./data/training as a
dataset artifact in W\&B:

```bash theme={null}
wandb artifact put --type dataset ./data/training
```

To upload model.pt to the "foobar" project and assign "trained-model" as
the artifact name:

```bash theme={null}
wandb artifact put --name foobar/trained-model --type model ./model.pt
```

To tag the artifact with both "latest" and "v2.0" so it can be
referenced by either alias:

```bash theme={null}
wandb artifact put --alias latest --alias v2.0 --type model ./model.pt
```

To record an Amazon S3 path as a reference without downloading or re-uploading the data:

```bash theme={null}
wandb artifact put --type dataset s3://my-bucket/datasets/training
```

To attach a human-readable description to a dataset artifact for documentation:

```bash theme={null}
wandb artifact put --type dataset --description "Training data, Jan 2025" ./data/training
```

## Arguments

| Name   | Default | Type |
| ------ | ------- | ---- |
| `path` | STR     | True |

## Options

| Flag                | Type      | Description                                                                                                         |
| ------------------- | --------- | ------------------------------------------------------------------------------------------------------------------- |
| `--name, -n`        | STR       | Artifact name in project/artifact\_name format. Defaults to the basename of the path. **Default**: None             |
| `--description, -d` | STR       | A description of this artifact. **Default**: None                                                                   |
| `--type, -t`        | STR       | The type of the artifact. Defaults to 'dataset'. **Default**: dataset                                               |
| `--alias, -a`       | STR       | An alias to apply to this artifact. Can be specified multiple times. Defaults to 'latest'. **Default**: \['latest'] |
| `--id`              | STR       | Upload to an existing run with this ID. **Default**: None                                                           |
| `--resume`          | BOOL Flag | Resume the last run from your current directory. **Default**: None                                                  |
| `--skip_cache`      | BOOL Flag | Skip caching while uploading artifact files. **Default**: False                                                     |
| `--policy`          | Choice    | Set the storage policy for artifact files. Either 'mutable' (default) or 'immutable'. **Default**: mutable          |


# wandb beta
Source: https://docs.wandb.ai/models/ref/cli/wandb-beta



<GitHubLink />

## Usage

```bash theme={null}
beta [OPTIONS] COMMAND [ARGS]...
```

## Description

Beta versions of wandb CLI commands.

These commands may change or even completely break in any release of wandb.

## Subcommands

| Command           | Description                                                           |
| ----------------- | --------------------------------------------------------------------- |
| `wandb beta core` | Manage a shared local wandb-core service for multi-process workloads. |
| `wandb beta leet` | W\&B LEET: the Lightweight Experiment Exploration Tool.               |
| `wandb beta sync` | Upload .wandb files specified by PATHS.                               |


# wandb beta core
Source: https://docs.wandb.ai/models/ref/cli/wandb-beta/wandb-beta-core



<GitHubLink />

## Usage

```bash theme={null}
core [OPTIONS] COMMAND [ARGS]...
```

## Description

Manage a shared local wandb-core service for multi-process workloads.

wandb-core is the local backend process that handles run data,
file uploads, and system metrics collection. By default, each
process that calls `wandb.init()` starts its own backend. On a
machine running many independent workers, that duplicates work
and wastes CPU and memory.

Use these commands to start one detached wandb-core instance and
point multiple workers on the same machine at it with the
WANDB\_SERVICE environment variable.

Typical workflow:

```bash theme={null}
wandb beta core start
```

```bash theme={null}
export WANDB_SERVICE=printed_value
```

```bash theme={null}
python -m your_launcher
```

```bash theme={null}
wandb beta core stop
```

For shell scripts, capture the raw WANDB\_SERVICE value from stdout:

```bash theme={null}
export WANDB_SERVICE="$(wandb beta core start)"
```

The shared service exits after 10 minutes of idleness by default.
Override this with --idle-timeout on the start command.

## Subcommands

| Command                 | Description                          |
| ----------------------- | ------------------------------------ |
| `wandb beta core start` | Start a detached wandb-core service. |
| `wandb beta core stop`  | Stop a detached wandb-core service.  |


# wandb beta core start
Source: https://docs.wandb.ai/models/ref/cli/wandb-beta/wandb-beta-core/wandb-beta-core-start



<GitHubLink />

## Usage

```bash theme={null}
start [OPTIONS]
```

## Description

Start a detached wandb-core service.

## Options

| Flag             | Type | Description                                                                                                                                                              |
| ---------------- | ---- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `--idle-timeout` | STR  | Shut down wandb-core after this much idle time with no connected clients. Uses Go duration syntax, for example 30s, 10m, or 0 to disable idle shutdown. **Default**: 10m |


# wandb beta core stop
Source: https://docs.wandb.ai/models/ref/cli/wandb-beta/wandb-beta-core/wandb-beta-core-stop



<GitHubLink />

## Usage

```bash theme={null}
stop [OPTIONS]
```

## Description

Stop a detached wandb-core service.

The service address is taken from the WANDB\_SERVICE environment variable.


# wandb beta leet
Source: https://docs.wandb.ai/models/ref/cli/wandb-beta/wandb-beta-leet



<GitHubLink />

## Usage

```bash theme={null}
leet [PATH] | COMMAND [ARGS]...
```

## Description

W\&B LEET: the Lightweight Experiment Exploration Tool.

A terminal UI for viewing your W\&B runs locally.

## Subcommands

| Command                  | Description                           |
| ------------------------ | ------------------------------------- |
| `wandb beta leet config` | Edit LEET configuration.              |
| `wandb beta leet run`    | Launch the LEET TUI.                  |
| `wandb beta leet symon`  | Launch the standalone system monitor. |


# wandb beta leet config
Source: https://docs.wandb.ai/models/ref/cli/wandb-beta/wandb-beta-leet/wandb-beta-leet-config



<GitHubLink />

## Usage

```bash theme={null}
config [OPTIONS]
```

## Description

Edit LEET configuration.


# wandb beta leet run
Source: https://docs.wandb.ai/models/ref/cli/wandb-beta/wandb-beta-leet/wandb-beta-leet-run



<GitHubLink />

## Usage

```bash theme={null}
run [OPTIONS] [PATH]
```

## Description

Launch the LEET TUI.

LEET is a terminal UI for viewing a W\&B run specified by an optional PATH.

PATH can include a .wandb file or a run directory containing a .wandb file.
If PATH is not provided, the command will look for the latest run.

## Arguments

| Name   | Default | Type  |
| ------ | ------- | ----- |
| `path` | STR     | False |

## Options

| Flag         | Type      | Description                                    |
| ------------ | --------- | ---------------------------------------------- |
| `-h, --help` | BOOL Flag | Show this message and exit. **Default**: False |


# wandb beta leet symon
Source: https://docs.wandb.ai/models/ref/cli/wandb-beta/wandb-beta-leet/wandb-beta-leet-symon



<GitHubLink />

## Usage

```bash theme={null}
symon [OPTIONS]
```

## Description

Launch the standalone system monitor.

## Options

| Flag         | Type      | Description                                                             |
| ------------ | --------- | ----------------------------------------------------------------------- |
| `--interval` | STR       | Sampling interval for system metrics (e.g. 500ms, 2s, 1m). **Default**: |
| `-h, --help` | BOOL Flag | Show this message and exit. **Default**: False                          |


# wandb beta sync
Source: https://docs.wandb.ai/models/ref/cli/wandb-beta/wandb-beta-sync



<GitHubLink />

## Usage

```bash theme={null}
sync [OPTIONS] [PATHS]...
```

## Description

Upload .wandb files specified by PATHS.

This is a beta re-implementation of `wandb sync`.
It is not feature complete, not guaranteed to work, and may change
in backward-incompatible ways in any release of wandb.

PATHS can include .wandb files, run directories containing .wandb files,
and "wandb" directories containing run directories.

For example, to sync all runs in the current .wandb directory:

```bash theme={null}
wandb beta sync ./wandb
```

To sync a specific run by specifying the run directory:

```bash theme={null}
wandb beta sync ./wandb/run-20250813_124246-n67z9ude
```

Or equivalently:

```bash theme={null}
wandb beta sync ./wandb/run-20250813_124246-n67z9ude/run-n67z9ude.wandb
```

## Arguments

| Name    | Default | Type  |
| ------- | ------- | ----- |
| `paths` | PATH    | False |

## Options

| Flag             | Type      | Description                                                                                                                                                                                                                                                                         |
| ---------------- | --------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `--live`         | BOOL Flag | Sync a run while it's still being logged.  This may hang if the process generating the run crashes uncleanly. **Default**: False                                                                                                                                                    |
| `-e, --entity`   | STR       | An entity override to use for all runs being synced. **Default**:                                                                                                                                                                                                                   |
| `-p, --project`  | STR       | A project override to use for all runs being synced. **Default**:                                                                                                                                                                                                                   |
| `--id`           | STR       | A run ID override to use for all runs being synced.  If setting this and syncing multiple files (with the same entity and project), the files will be synced in order of start time. This is intended to work with syncing multiple resumed fragments of the same run. **Default**: |
| `--job-type`     | STR       | A job type override for all runs being synced. **Default**:                                                                                                                                                                                                                         |
| `--replace-tags` | STR       | Rename tags using the format 'old1=new1,old2=new2'. **Default**:                                                                                                                                                                                                                    |
| `--skip-synced`  | BOOL Flag | Skip runs that have already been synced with this command. **Default**: True                                                                                                                                                                                                        |
| `--dry-run`      | BOOL Flag | Print what would happen without uploading anything. **Default**: False                                                                                                                                                                                                              |
| `-v, --verbose`  | BOOL Flag | Print more information. **Default**: False                                                                                                                                                                                                                                          |
| `-n`             | INT       | Max number of runs to sync at a time.  When syncing multiple files that are part of the same run, the files are synced sequentially in order of start time regardless of this setting. This happens for resumed runs or when using the --id parameter. **Default**: 5               |


# wandb controller
Source: https://docs.wandb.ai/models/ref/cli/wandb-controller



<GitHubLink />

## Usage

```bash theme={null}
controller [OPTIONS] SWEEP_ID
```

## Description

Start a local sweep controller for a W\&B hyperparameter sweep.

Start a local process that orchestrates the specified sweep. Read the
sweep configuration from W\&B, select hyperparameter combinations based
on the configured search strategy (grid, random, Bayesian, and so on),
and dispatch runs to sweep agents.

By default, W\&B runs sweep controllers on its managed infrastructure.
Use this command to run the controller locally instead. For example, you
can use this command to debug behavior or operate in environments with
limited connectivity.

sweep\_id is printed by `wandb sweep` when you create a sweep. It
consists of a unique identifier for the sweep and may include the
entity and project path (entity/project/sweep\_id).

For example, to start a local sweep controller for a sweep with sweep ID wbyz9876:

```bash theme={null}
wandb controller wbyz9876
```

## Arguments

| Name       | Default | Type |
| ---------- | ------- | ---- |
| `sweep_id` | STR     | True |

## Options

| Flag        | Type      | Description                                                |
| ----------- | --------- | ---------------------------------------------------------- |
| `--verbose` | BOOL Flag | Display verbose output from controller. **Default**: False |


# wandb disabled
Source: https://docs.wandb.ai/models/ref/cli/wandb-disabled



<GitHubLink />

## Usage

```bash theme={null}
disabled [OPTIONS]
```

## Description

Disable W\&B functionality.

While W\&B is in `disabled` mode, it does not log or sync data.

Use `wandb enable` to restore W\&B functionality.

Use `wandb offline` to stop cloud syncing while continuning to log data
locally.

For example, to turn off W\&B so that the train.py script executes without logging or
syncing data to W\&B:

```bash theme={null}
wandb disabled
```

Next, train the model without logging or syncing to W\&B:

```bash theme={null}
python train.py  # Does not log or sync data to W&B
```

Restore W\&B functionality when ready to log and sync again:

```bash theme={null}
wandb enabled
```

## Options

| Flag        | Type      | Description                                                        |
| ----------- | --------- | ------------------------------------------------------------------ |
| `--service` | BOOL Flag | No effect. Accepted for backwards compatibility. **Default**: True |


# Automations overview
Source: https://docs.wandb.ai/models/ref/python/automations

Use the W&B Automations API to create and manage automated workflows in your ML pipelines

The W\&B Automations API enables programmatic creation and management of automated workflows that respond to events in your ML pipeline. Configure actions to trigger when specific conditions are met, such as model performance thresholds or artifact creation.

### Core classes

| Class                                                            | Description                                                    |
| ---------------------------------------------------------------- | -------------------------------------------------------------- |
| [`Automation`](/models/ref/python/automations/automation/)       | Represents a saved automation instance with its configuration. |
| [`NewAutomation`](/models/ref/python/automations/newautomation/) | Builder class for creating new automations.                    |

### Events (Triggers)

| Event                                                                      | Description                                                                        |
| -------------------------------------------------------------------------- | ---------------------------------------------------------------------------------- |
| [`OnRunMetric`](/models/ref/python/automations/onrunmetric/)               | Trigger when a run metric satisfies a defined condition (threshold, change, etc.). |
| [`OnCreateArtifact`](/models/ref/python/automations/oncreateartifact/)     | Trigger when a new artifact is created in a collection.                            |
| [`OnLinkArtifact`](/models/ref/python/automations/onlinkartifact/)         | Trigger when an artifact is linked to a registry.                                  |
| [`OnAddArtifactAlias`](/models/ref/python/automations/onaddartifactalias/) | Trigger when an alias is added to an artifact.                                     |

### Actions

| Action                                                                 | Description                                                |
| ---------------------------------------------------------------------- | ---------------------------------------------------------- |
| [`SendNotification`](/models/ref/python/automations/sendnotification/) | Send notifications via Slack or other integrated channels. |
| [`SendWebhook`](/models/ref/python/automations/sendwebhook/)           | Send HTTP webhook requests to external services.           |
| [`DoNothing`](/models/ref/python/automations/donothing/)               | Placeholder action for testing automation configurations.  |

### Filters

| Filter                                                                           | Description                                                       |
| -------------------------------------------------------------------------------- | ----------------------------------------------------------------- |
| [`MetricThresholdFilter`](/models/ref/python/automations/metricthresholdfilter/) | Filter runs based on metric value comparisons against thresholds. |
| [`MetricChangeFilter`](/models/ref/python/automations/metricchangefilter/)       | Filter runs based on metric value changes over time.              |

## Common use cases

### Model performance monitoring

* Alert when model accuracy drops below a threshold
* Notify team when training loss plateaus
* Trigger retraining pipelines based on performance metrics

### Artifact Management

* Send notifications when new model versions are created
* Trigger deployment workflows when artifacts are tagged
* Automate downstream processing when datasets are updated

### Experiment tracking

* Alert on failed or crashed runs
* Notify when long-running experiments complete
* Send daily summaries of experiment metrics

### Integration workflows

* Update external tracking systems via webhooks
* Sync model registry with deployment platforms
* Trigger CI/CD pipelines based on W\&B events

## Example usage

The following example creates an automation that sends a Slack notification whenever a metric called `custom-metric` exceeds 10. `custom-metric` is expected to be logged during training using `wandb.Run.log({"custom-metric": value })`.

```python theme={null}
import wandb
from wandb.automations import OnRunMetric, RunEvent, SendNotification

api = wandb.Api()

project = api.project("<my-project>", entity="<my-team>")

# Use the first Slack integration for the team
slack_hook = next(api.slack_integrations(entity="<my-team>"))

# Create a trigger event
event = OnRunMetric(
     scope=project,
     filter=RunEvent.metric("custom-metric") > 10,
)

# Create an action that responds to the event
action = SendNotification.from_integration(slack_hook)

# Create the automation
automation = api.create_automation(
     event >> action,
     name="my-automation",
     description="Send a Slack message whenever 'custom-metric' exceeds 10.",
)
```

For more information about using the Automations API, see the [Automations Guide](/models/automations/).


# Automation
Source: https://docs.wandb.ai/models/ref/python/automations/automation



<GitHubLink />

## <kbd>class</kbd> `Automation`

A local instance of a saved W\&B automation that supports editing.

### <kbd>method</kbd> `Automation.__init__`

```python theme={null}
__init__(
    typename__: 'Literal['Trigger']' = 'Trigger',
    id: 'str',
    created_at: 'datetime',
    updated_at: 'datetime.datetime | None' = None,
    name: 'str',
    description: 'str | None',
    enabled: 'bool',
    scope: 'wandb.automations.scopes._ArtifactSequenceScope | wandb.automations.scopes._ArtifactPortfolioScope | wandb.automations.scopes.ProjectScope',
    event: 'SavedEvent',
    action: 'wandb.automations.actions.SavedLaunchJobAction | wandb.automations.actions.SavedNotificationAction | wandb.automations.actions.SavedWebhookAction | wandb.automations.actions.SavedNoOpAction'
) → None
```

**Args:**

* `typename__` (Literal\['Trigger']):
* `id` (str):
* `created_at` (datetime): The date and time when this automation was created.
* `updated_at` (datetime.datetime | None): The date and time when this automation was last updated, if applicable.
* `name` (str): The name of this automation.
* `description` (str | None): An optional description of this automation.
* `enabled` (bool): Whether this automation is enabled.  Only enabled automations will trigger.
* `scope` (wandb.automations.scopes.\_ArtifactSequenceScope | wandb.automations.scopes.\_ArtifactPortfolioScope | wandb.automations.scopes.ProjectScope): The scope in which the triggering event must occur.
* `event` (SavedEvent): The event that will trigger this automation.
* `action` (wandb.automations.actions.SavedLaunchJobAction | wandb.automations.actions.SavedNotificationAction | wandb.automations.actions.SavedWebhookAction | wandb.automations.actions.SavedNoOpAction): The action that will execute when this automation is triggered.

**Returns:**
An `Automation` object.


# DoNothing
Source: https://docs.wandb.ai/models/ref/python/automations/donothing



<GitHubLink />

## <kbd>class</kbd> `DoNothing`

Defines an automation action that intentionally does nothing.

### <kbd>method</kbd> `DoNothing.__init__`

```python theme={null}
__init__(
    no_op: 'bool' = True,
    action_type: 'Literal[NO_OP]' = NO_OP
) → None
```

**Args:**

* `no_op` (bool): Placeholder field which exists only to satisfy backend schema requirements.
  There should never be a need to set this field explicitly, as its value is ignored.
* `action_type` (Literal\[NO\_OP]):

**Returns:**
An `DoNothing` object.


# MetricChangeFilter
Source: https://docs.wandb.ai/models/ref/python/automations/metricchangefilter



<GitHubLink />

## <kbd>class</kbd> `MetricChangeFilter`

Filter that compares a **change** in a metric value to a user-defined threshold.

The change is calculated over "tumbling" windows, i.e. the difference
between the current window and the non-overlapping prior window.

### <kbd>method</kbd> `MetricChangeFilter.__init__`

```python theme={null}
__init__(
    name: 'str',
    agg: 'wandb.automations._filters.run_metrics.Agg | None' = None,
    window: 'int' = 1,
    cmp: 'None' = None,
    threshold: 'Annotated | Annotated',
    prior_window: 'int' = None,
    change_type: 'ChangeType',
    change_dir: 'ChangeDir'
) → None
```

**Args:**

* `name` (str):
* `agg` (wandb.automations.\_filters.run\_metrics.Agg | None):
* `window` (int):
* `cmp` (None): Ignored.
* `threshold` (Union\[Annotated, Annotated]):
* `prior_window` (int): Size of the "prior" metric aggregation window (ignored if `agg` is `None`).
  If omitted, defaults to the size of the current window.
* `change_type` (ChangeType):
* `change_dir` (ChangeDir):

**Returns:**
An `MetricChangeFilter` object.


# MetricThresholdFilter
Source: https://docs.wandb.ai/models/ref/python/automations/metricthresholdfilter



<GitHubLink />

## <kbd>class</kbd> `MetricThresholdFilter`

Filter that compares an **absolute** metric value against a user-defined threshold.

The value may be a single value or an aggregated result over a window of
multiple values.

### <kbd>method</kbd> `MetricThresholdFilter.__init__`

```python theme={null}
__init__(
    name: 'str',
    agg: 'wandb.automations._filters.run_metrics.Agg | None' = None,
    window: 'int' = 1,
    cmp: 'Literal['$gte', '$gt', '$lt', '$lte']',
    threshold: 'Annotated | Annotated'
) → None
```

**Args:**

* `name` (str):
* `agg` (wandb.automations.\_filters.run\_metrics.Agg | None):
* `window` (int):
* `cmp` (Literal\['$gte', '$gt', '$lt', '$lte']): Comparison operator between the metric value (left) vs. the threshold (right).
* `threshold` (Union\[Annotated, Annotated]):

**Returns:**
An `MetricThresholdFilter` object.


# MetricZScoreFilter
Source: https://docs.wandb.ai/models/ref/python/automations/metriczscorefilter



<GitHubLink />

## <kbd>class</kbd> `MetricZScoreFilter`

Filter that compares a metric's z-score against a user-defined threshold.

### <kbd>method</kbd> `MetricZScoreFilter.__init__`

```python theme={null}
__init__(
    name: 'str',
    window: 'int' = 30,
    threshold: 'Annotated | Annotated' = 3.0,
    change_dir: 'ChangeDir' = ANY
) → None
```

**Args:**

* `name` (str): Name of the observed metric.
* `window` (int): Size of the window to calculate the metric mean and standard deviation over.
* `threshold` (Union\[Annotated, Annotated]): Threshold for the z-score.
* `change_dir` (ChangeDir): Direction of the z-score change to watch for.

**Returns:**
An `MetricZScoreFilter` object.


# NewAutomation
Source: https://docs.wandb.ai/models/ref/python/automations/newautomation



<GitHubLink />

## <kbd>class</kbd> `NewAutomation`

A new automation to be created.

### <kbd>method</kbd> `NewAutomation.__init__`

```python theme={null}
__init__(
    name: 'str | None' = None,
    description: 'str | None' = None,
    enabled: 'bool | None' = None,
    event: 'Annotated | None' = None,
    action: 'Annotated | None' = None
) → None
```

**Args:**

* `name` (str | None): The name of this automation.
* `description` (str | None): An optional description of this automation.
* `enabled` (bool | None): Whether this automation is enabled.  Only enabled automations will trigger.
* `event` (Optional\[Annotated]): The event that will trigger this automation.
* `action` (Optional\[Annotated]): The action that will execute when this automation is triggered.

**Returns:**
An `NewAutomation` object.

### <kbd>property</kbd> `NewAutomation.scope`

The scope in which the triggering event must occur.

**Returns:**

* `AutomationScope | None`: The scope property value.


# OnAddArtifactAlias
Source: https://docs.wandb.ai/models/ref/python/automations/onaddartifactalias



<GitHubLink />

## <kbd>class</kbd> `OnAddArtifactAlias`

A new alias is assigned to an artifact.

Examples:
Define an event that triggers whenever the alias "prod" is assigned to
any artifact in the collection "my-collection":

```python theme={null}
from wandb import Api
from wandb.automations import OnAddArtifactAlias, ArtifactEvent

api = Api()
collection = api.artifact_collection(name="my-collection", type_name="model")

event = OnAddArtifactAlias(
    scope=collection,
    filter=ArtifactEvent.alias.eq("prod"),
)
```

### <kbd>method</kbd> `OnAddArtifactAlias.__init__`

```python theme={null}
__init__(
    event_type: 'Literal[ADD_ARTIFACT_ALIAS]' = ADD_ARTIFACT_ALIAS,
    scope: 'wandb.automations.scopes._ArtifactSequenceScope | wandb.automations.scopes._ArtifactPortfolioScope | wandb.automations.scopes.ProjectScope',
    filter: 'wandb.automations._filters.operators.And | wandb.automations._filters.operators.Or | wandb.automations._filters.operators.Nor | wandb.automations._filters.operators.Not | wandb.automations._filters.operators.Lt | wandb.automations._filters.operators.Gt | wandb.automations._filters.operators.Lte | wandb.automations._filters.operators.Gte | wandb.automations._filters.operators.Eq | wandb.automations._filters.operators.Ne | wandb.automations._filters.operators.In | wandb.automations._filters.operators.NotIn | wandb.automations._filters.operators.Exists | wandb.automations._filters.operators.Regex | wandb.automations._filters.operators.Contains | wandb.automations._filters.expressions.FilterExpr | dict[str, Any]' = And(())
) → None
```

**Args:**

* `event_type` (Literal\[ADD\_ARTIFACT\_ALIAS]):
* `scope` (wandb.automations.scopes.\_ArtifactSequenceScope | wandb.automations.scopes.\_ArtifactPortfolioScope | wandb.automations.scopes.ProjectScope): The scope of the event.
* `filter` (wandb.automations.\_filters.operators.And | wandb.automations.\_filters.operators.Or | wandb.automations.\_filters.operators.Nor | wandb.automations.\_filters.operators.Not | wandb.automations.\_filters.operators.Lt | wandb.automations.\_filters.operators.Gt | wandb.automations.\_filters.operators.Lte | wandb.automations.\_filters.operators.Gte | wandb.automations.\_filters.operators.Eq | wandb.automations.\_filters.operators.Ne | wandb.automations.\_filters.operators.In | wandb.automations.\_filters.operators.NotIn | wandb.automations.\_filters.operators.Exists | wandb.automations.\_filters.operators.Regex | wandb.automations.\_filters.operators.Contains | wandb.automations.\_filters.expressions.FilterExpr | dict\[str, Any]): Additional conditions(s), if any, that are required for this event to trigger.

**Returns:**
An `OnAddArtifactAlias` object.


# OnAddArtifactTag
Source: https://docs.wandb.ai/models/ref/python/automations/onaddartifacttag



<GitHubLink />

## <kbd>class</kbd> `OnAddArtifactTag`

A new tag is assigned to an artifact version.

Examples:
Define an event that triggers whenever the tag "prod" is assigned to
any artifact version in the collection "my-collection":

```python theme={null}
from wandb import Api
from wandb.automations import OnAddArtifactTag, ArtifactEvent

api = Api()
collection = api.artifact_collection(name="my-collection", type_name="model")

event = OnAddArtifactTag(
    scope=collection,
    filter=ArtifactEvent.tag.eq("prod"),
)
```

### <kbd>method</kbd> `OnAddArtifactTag.__init__`

```python theme={null}
__init__(
    event_type: 'Literal[ADD_ARTIFACT_TAG]' = ADD_ARTIFACT_TAG,
    scope: 'wandb.automations.scopes._ArtifactSequenceScope | wandb.automations.scopes._ArtifactPortfolioScope | wandb.automations.scopes.ProjectScope',
    filter: 'wandb.automations._filters.operators.And | wandb.automations._filters.operators.Or | wandb.automations._filters.operators.Nor | wandb.automations._filters.operators.Not | wandb.automations._filters.operators.Lt | wandb.automations._filters.operators.Gt | wandb.automations._filters.operators.Lte | wandb.automations._filters.operators.Gte | wandb.automations._filters.operators.Eq | wandb.automations._filters.operators.Ne | wandb.automations._filters.operators.In | wandb.automations._filters.operators.NotIn | wandb.automations._filters.operators.Exists | wandb.automations._filters.operators.Regex | wandb.automations._filters.operators.Contains | wandb.automations._filters.expressions.FilterExpr | dict[str, Any]' = And(())
) → None
```

**Args:**

* `event_type` (Literal\[ADD\_ARTIFACT\_TAG]):
* `scope` (wandb.automations.scopes.\_ArtifactSequenceScope | wandb.automations.scopes.\_ArtifactPortfolioScope | wandb.automations.scopes.ProjectScope): The scope of the event.
* `filter` (wandb.automations.\_filters.operators.And | wandb.automations.\_filters.operators.Or | wandb.automations.\_filters.operators.Nor | wandb.automations.\_filters.operators.Not | wandb.automations.\_filters.operators.Lt | wandb.automations.\_filters.operators.Gt | wandb.automations.\_filters.operators.Lte | wandb.automations.\_filters.operators.Gte | wandb.automations.\_filters.operators.Eq | wandb.automations.\_filters.operators.Ne | wandb.automations.\_filters.operators.In | wandb.automations.\_filters.operators.NotIn | wandb.automations.\_filters.operators.Exists | wandb.automations.\_filters.operators.Regex | wandb.automations.\_filters.operators.Contains | wandb.automations.\_filters.expressions.FilterExpr | dict\[str, Any]): Additional conditions(s), if any, that are required for this event to trigger.

**Returns:**
An `OnAddArtifactTag` object.


# OnAddCollectionTag
Source: https://docs.wandb.ai/models/ref/python/automations/onaddcollectiontag



<GitHubLink />

## <kbd>class</kbd> `OnAddCollectionTag`

A new tag is assigned to an artifact collection.

### <kbd>method</kbd> `OnAddCollectionTag.__init__`

```python theme={null}
__init__(
    event_type: 'Literal[ADD_COLLECTION_TAG]' = ADD_COLLECTION_TAG,
    scope: 'wandb.automations.scopes._ArtifactSequenceScope | wandb.automations.scopes._ArtifactPortfolioScope | wandb.automations.scopes.ProjectScope',
    filter: 'wandb.automations._filters.operators.And | wandb.automations._filters.operators.Or | wandb.automations._filters.operators.Nor | wandb.automations._filters.operators.Not | wandb.automations._filters.operators.Lt | wandb.automations._filters.operators.Gt | wandb.automations._filters.operators.Lte | wandb.automations._filters.operators.Gte | wandb.automations._filters.operators.Eq | wandb.automations._filters.operators.Ne | wandb.automations._filters.operators.In | wandb.automations._filters.operators.NotIn | wandb.automations._filters.operators.Exists | wandb.automations._filters.operators.Regex | wandb.automations._filters.operators.Contains | wandb.automations._filters.expressions.FilterExpr | dict[str, Any]' = And(())
) → None
```

**Args:**

* `event_type` (Literal\[ADD\_COLLECTION\_TAG]):
* `scope` (wandb.automations.scopes.\_ArtifactSequenceScope | wandb.automations.scopes.\_ArtifactPortfolioScope | wandb.automations.scopes.ProjectScope): The scope of the event.
* `filter` (wandb.automations.\_filters.operators.And | wandb.automations.\_filters.operators.Or | wandb.automations.\_filters.operators.Nor | wandb.automations.\_filters.operators.Not | wandb.automations.\_filters.operators.Lt | wandb.automations.\_filters.operators.Gt | wandb.automations.\_filters.operators.Lte | wandb.automations.\_filters.operators.Gte | wandb.automations.\_filters.operators.Eq | wandb.automations.\_filters.operators.Ne | wandb.automations.\_filters.operators.In | wandb.automations.\_filters.operators.NotIn | wandb.automations.\_filters.operators.Exists | wandb.automations.\_filters.operators.Regex | wandb.automations.\_filters.operators.Contains | wandb.automations.\_filters.expressions.FilterExpr | dict\[str, Any]): Additional conditions(s), if any, that are required for this event to trigger.

**Returns:**
An `OnAddCollectionTag` object.


# OnCreateArtifact
Source: https://docs.wandb.ai/models/ref/python/automations/oncreateartifact



<GitHubLink />

## <kbd>class</kbd> `OnCreateArtifact`

A new artifact is created.

Examples:
Define an event that triggers when a new artifact is created in the
collection "my-collection":

```python theme={null}
from wandb import Api
from wandb.automations import OnCreateArtifact

api = Api()
collection = api.artifact_collection(name="my-collection", type_name="model")

event = OnCreateArtifact(scope=collection)
```

### <kbd>method</kbd> `OnCreateArtifact.__init__`

```python theme={null}
__init__(
    event_type: 'Literal[CREATE_ARTIFACT]' = CREATE_ARTIFACT,
    scope: 'wandb.automations.scopes._ArtifactSequenceScope | wandb.automations.scopes._ArtifactPortfolioScope',
    filter: 'wandb.automations._filters.operators.And | wandb.automations._filters.operators.Or | wandb.automations._filters.operators.Nor | wandb.automations._filters.operators.Not | wandb.automations._filters.operators.Lt | wandb.automations._filters.operators.Gt | wandb.automations._filters.operators.Lte | wandb.automations._filters.operators.Gte | wandb.automations._filters.operators.Eq | wandb.automations._filters.operators.Ne | wandb.automations._filters.operators.In | wandb.automations._filters.operators.NotIn | wandb.automations._filters.operators.Exists | wandb.automations._filters.operators.Regex | wandb.automations._filters.operators.Contains | wandb.automations._filters.expressions.FilterExpr | dict[str, Any]' = And(())
) → None
```

**Args:**

* `event_type` (Literal\[CREATE\_ARTIFACT]):
* `scope` (wandb.automations.scopes.\_ArtifactSequenceScope | wandb.automations.scopes.\_ArtifactPortfolioScope): The scope of the event: must be an artifact collection.
* `filter` (wandb.automations.\_filters.operators.And | wandb.automations.\_filters.operators.Or | wandb.automations.\_filters.operators.Nor | wandb.automations.\_filters.operators.Not | wandb.automations.\_filters.operators.Lt | wandb.automations.\_filters.operators.Gt | wandb.automations.\_filters.operators.Lte | wandb.automations.\_filters.operators.Gte | wandb.automations.\_filters.operators.Eq | wandb.automations.\_filters.operators.Ne | wandb.automations.\_filters.operators.In | wandb.automations.\_filters.operators.NotIn | wandb.automations.\_filters.operators.Exists | wandb.automations.\_filters.operators.Regex | wandb.automations.\_filters.operators.Contains | wandb.automations.\_filters.expressions.FilterExpr | dict\[str, Any]): Additional conditions(s), if any, that are required for this event to trigger.

**Returns:**
An `OnCreateArtifact` object.


# OnLinkArtifact
Source: https://docs.wandb.ai/models/ref/python/automations/onlinkartifact



<GitHubLink />

## <kbd>class</kbd> `OnLinkArtifact`

A new artifact is linked to a collection.

Examples:
Define an event that triggers when an artifact is linked to the
collection "my-collection" with the alias "prod":

```python theme={null}
from wandb import Api
from wandb.automations import OnLinkArtifact, ArtifactEvent

api = Api()
collection = api.artifact_collection(name="my-collection", type_name="model")

event = OnLinkArtifact(
    scope=collection,
    filter=ArtifactEvent.alias.eq("prod"),
)
```

### <kbd>method</kbd> `OnLinkArtifact.__init__`

```python theme={null}
__init__(
    event_type: 'Literal[LINK_ARTIFACT]' = LINK_ARTIFACT,
    scope: 'wandb.automations.scopes._ArtifactSequenceScope | wandb.automations.scopes._ArtifactPortfolioScope | wandb.automations.scopes.ProjectScope',
    filter: 'wandb.automations._filters.operators.And | wandb.automations._filters.operators.Or | wandb.automations._filters.operators.Nor | wandb.automations._filters.operators.Not | wandb.automations._filters.operators.Lt | wandb.automations._filters.operators.Gt | wandb.automations._filters.operators.Lte | wandb.automations._filters.operators.Gte | wandb.automations._filters.operators.Eq | wandb.automations._filters.operators.Ne | wandb.automations._filters.operators.In | wandb.automations._filters.operators.NotIn | wandb.automations._filters.operators.Exists | wandb.automations._filters.operators.Regex | wandb.automations._filters.operators.Contains | wandb.automations._filters.expressions.FilterExpr | dict[str, Any]' = And(())
) → None
```

**Args:**

* `event_type` (Literal\[LINK\_ARTIFACT]):
* `scope` (wandb.automations.scopes.\_ArtifactSequenceScope | wandb.automations.scopes.\_ArtifactPortfolioScope | wandb.automations.scopes.ProjectScope): The scope of the event.
* `filter` (wandb.automations.\_filters.operators.And | wandb.automations.\_filters.operators.Or | wandb.automations.\_filters.operators.Nor | wandb.automations.\_filters.operators.Not | wandb.automations.\_filters.operators.Lt | wandb.automations.\_filters.operators.Gt | wandb.automations.\_filters.operators.Lte | wandb.automations.\_filters.operators.Gte | wandb.automations.\_filters.operators.Eq | wandb.automations.\_filters.operators.Ne | wandb.automations.\_filters.operators.In | wandb.automations.\_filters.operators.NotIn | wandb.automations.\_filters.operators.Exists | wandb.automations.\_filters.operators.Regex | wandb.automations.\_filters.operators.Contains | wandb.automations.\_filters.expressions.FilterExpr | dict\[str, Any]): Additional conditions(s), if any, that are required for this event to trigger.

**Returns:**
An `OnLinkArtifact` object.


# OnRemoveArtifactTag
Source: https://docs.wandb.ai/models/ref/python/automations/onremoveartifacttag



<GitHubLink />

## <kbd>class</kbd> `OnRemoveArtifactTag`

A tag is removed from an artifact version.

### <kbd>method</kbd> `OnRemoveArtifactTag.__init__`

```python theme={null}
__init__(
    event_type: 'Literal[REMOVE_ARTIFACT_TAG]' = REMOVE_ARTIFACT_TAG,
    scope: 'wandb.automations.scopes._ArtifactSequenceScope | wandb.automations.scopes._ArtifactPortfolioScope | wandb.automations.scopes.ProjectScope',
    filter: 'wandb.automations._filters.operators.And | wandb.automations._filters.operators.Or | wandb.automations._filters.operators.Nor | wandb.automations._filters.operators.Not | wandb.automations._filters.operators.Lt | wandb.automations._filters.operators.Gt | wandb.automations._filters.operators.Lte | wandb.automations._filters.operators.Gte | wandb.automations._filters.operators.Eq | wandb.automations._filters.operators.Ne | wandb.automations._filters.operators.In | wandb.automations._filters.operators.NotIn | wandb.automations._filters.operators.Exists | wandb.automations._filters.operators.Regex | wandb.automations._filters.operators.Contains | wandb.automations._filters.expressions.FilterExpr | dict[str, Any]' = And(())
) → None
```

**Args:**

* `event_type` (Literal\[REMOVE\_ARTIFACT\_TAG]):
* `scope` (wandb.automations.scopes.\_ArtifactSequenceScope | wandb.automations.scopes.\_ArtifactPortfolioScope | wandb.automations.scopes.ProjectScope): The scope of the event.
* `filter` (wandb.automations.\_filters.operators.And | wandb.automations.\_filters.operators.Or | wandb.automations.\_filters.operators.Nor | wandb.automations.\_filters.operators.Not | wandb.automations.\_filters.operators.Lt | wandb.automations.\_filters.operators.Gt | wandb.automations.\_filters.operators.Lte | wandb.automations.\_filters.operators.Gte | wandb.automations.\_filters.operators.Eq | wandb.automations.\_filters.operators.Ne | wandb.automations.\_filters.operators.In | wandb.automations.\_filters.operators.NotIn | wandb.automations.\_filters.operators.Exists | wandb.automations.\_filters.operators.Regex | wandb.automations.\_filters.operators.Contains | wandb.automations.\_filters.expressions.FilterExpr | dict\[str, Any]): Additional conditions(s), if any, that are required for this event to trigger.

**Returns:**
An `OnRemoveArtifactTag` object.


# OnRemoveCollectionTag
Source: https://docs.wandb.ai/models/ref/python/automations/onremovecollectiontag



<GitHubLink />

## <kbd>class</kbd> `OnRemoveCollectionTag`

A tag is removed from an artifact collection.

### <kbd>method</kbd> `OnRemoveCollectionTag.__init__`

```python theme={null}
__init__(
    event_type: 'Literal[REMOVE_COLLECTION_TAG]' = REMOVE_COLLECTION_TAG,
    scope: 'wandb.automations.scopes._ArtifactSequenceScope | wandb.automations.scopes._ArtifactPortfolioScope | wandb.automations.scopes.ProjectScope',
    filter: 'wandb.automations._filters.operators.And | wandb.automations._filters.operators.Or | wandb.automations._filters.operators.Nor | wandb.automations._filters.operators.Not | wandb.automations._filters.operators.Lt | wandb.automations._filters.operators.Gt | wandb.automations._filters.operators.Lte | wandb.automations._filters.operators.Gte | wandb.automations._filters.operators.Eq | wandb.automations._filters.operators.Ne | wandb.automations._filters.operators.In | wandb.automations._filters.operators.NotIn | wandb.automations._filters.operators.Exists | wandb.automations._filters.operators.Regex | wandb.automations._filters.operators.Contains | wandb.automations._filters.expressions.FilterExpr | dict[str, Any]' = And(())
) → None
```

**Args:**

* `event_type` (Literal\[REMOVE\_COLLECTION\_TAG]):
* `scope` (wandb.automations.scopes.\_ArtifactSequenceScope | wandb.automations.scopes.\_ArtifactPortfolioScope | wandb.automations.scopes.ProjectScope): The scope of the event.
* `filter` (wandb.automations.\_filters.operators.And | wandb.automations.\_filters.operators.Or | wandb.automations.\_filters.operators.Nor | wandb.automations.\_filters.operators.Not | wandb.automations.\_filters.operators.Lt | wandb.automations.\_filters.operators.Gt | wandb.automations.\_filters.operators.Lte | wandb.automations.\_filters.operators.Gte | wandb.automations.\_filters.operators.Eq | wandb.automations.\_filters.operators.Ne | wandb.automations.\_filters.operators.In | wandb.automations.\_filters.operators.NotIn | wandb.automations.\_filters.operators.Exists | wandb.automations.\_filters.operators.Regex | wandb.automations.\_filters.operators.Contains | wandb.automations.\_filters.expressions.FilterExpr | dict\[str, Any]): Additional conditions(s), if any, that are required for this event to trigger.

**Returns:**
An `OnRemoveCollectionTag` object.


# OnRunMetric
Source: https://docs.wandb.ai/models/ref/python/automations/onrunmetric



<GitHubLink />

## <kbd>class</kbd> `OnRunMetric`

A run metric satisfies a user-defined condition.

Examples:
Define an event that triggers for any run in project "my-project" when
the average of the last 5 values of metric "my-metric" exceeds 123.45:

```python theme={null}
from wandb import Api
from wandb.automations import OnRunMetric, RunEvent

api = Api()
project = api.project(name="my-project")

event = OnRunMetric(
    scope=project,
    filter=RunEvent.metric("my-metric").avg(5).gt(123.45),
)
```

### <kbd>method</kbd> `OnRunMetric.__init__`

```python theme={null}
__init__(
    event_type: 'Literal[RUN_METRIC_THRESHOLD, RUN_METRIC_CHANGE, RUN_METRIC_ZSCORE]',
    scope: 'ProjectScope',
    filter: 'RunMetricFilter'
) → None
```

**Args:**

* `event_type` (Literal\[RUN\_METRIC\_THRESHOLD, RUN\_METRIC\_CHANGE, RUN\_METRIC\_ZSCORE]):
* `scope` (ProjectScope): The scope of the event: must be a project.
* `filter` (RunMetricFilter): Run and/or metric condition(s) that must be satisfied for this event to trigger.

**Returns:**
An `OnRunMetric` object.


# OnRunState
Source: https://docs.wandb.ai/models/ref/python/automations/onrunstate



<GitHubLink />

## <kbd>class</kbd> `OnRunState`

A run state changes.

Examples:
Define an event that triggers for any run in project "my-project" when
its state changes to "finished" (i.e. succeeded) or "failed":

```python theme={null}
from wandb import Api
from wandb.automations import OnRunState

api = Api()
project = api.project(name="my-project")

event = OnRunState(
    scope=project,
    filter=RunEvent.state.in_(["finished", "failed"]),
)
```

### <kbd>method</kbd> `OnRunState.__init__`

```python theme={null}
__init__(
    event_type: 'Literal[RUN_STATE]' = RUN_STATE,
    scope: 'ProjectScope',
    filter: 'RunStateFilter'
) → None
```

**Args:**

* `event_type` (Literal\[RUN\_STATE]):
* `scope` (ProjectScope): The scope of the event: must be a project.
* `filter` (RunStateFilter): Run state condition(s) that must be satisfied for this event to trigger.

**Returns:**
An `OnRunState` object.


# OnUnlinkArtifact
Source: https://docs.wandb.ai/models/ref/python/automations/onunlinkartifact



<GitHubLink />

## <kbd>class</kbd> `OnUnlinkArtifact`

An artifact version is unlinked from a collection.

### <kbd>method</kbd> `OnUnlinkArtifact.__init__`

```python theme={null}
__init__(
    event_type: 'Literal[UNLINK_ARTIFACT]' = UNLINK_ARTIFACT,
    scope: 'wandb.automations.scopes._ArtifactSequenceScope | wandb.automations.scopes._ArtifactPortfolioScope | wandb.automations.scopes.ProjectScope',
    filter: 'wandb.automations._filters.operators.And | wandb.automations._filters.operators.Or | wandb.automations._filters.operators.Nor | wandb.automations._filters.operators.Not | wandb.automations._filters.operators.Lt | wandb.automations._filters.operators.Gt | wandb.automations._filters.operators.Lte | wandb.automations._filters.operators.Gte | wandb.automations._filters.operators.Eq | wandb.automations._filters.operators.Ne | wandb.automations._filters.operators.In | wandb.automations._filters.operators.NotIn | wandb.automations._filters.operators.Exists | wandb.automations._filters.operators.Regex | wandb.automations._filters.operators.Contains | wandb.automations._filters.expressions.FilterExpr | dict[str, Any]' = And(())
) → None
```

**Args:**

* `event_type` (Literal\[UNLINK\_ARTIFACT]):
* `scope` (wandb.automations.scopes.\_ArtifactSequenceScope | wandb.automations.scopes.\_ArtifactPortfolioScope | wandb.automations.scopes.ProjectScope): The scope of the event.
* `filter` (wandb.automations.\_filters.operators.And | wandb.automations.\_filters.operators.Or | wandb.automations.\_filters.operators.Nor | wandb.automations.\_filters.operators.Not | wandb.automations.\_filters.operators.Lt | wandb.automations.\_filters.operators.Gt | wandb.automations.\_filters.operators.Lte | wandb.automations.\_filters.operators.Gte | wandb.automations.\_filters.operators.Eq | wandb.automations.\_filters.operators.Ne | wandb.automations.\_filters.operators.In | wandb.automations.\_filters.operators.NotIn | wandb.automations.\_filters.operators.Exists | wandb.automations.\_filters.operators.Regex | wandb.automations.\_filters.operators.Contains | wandb.automations.\_filters.expressions.FilterExpr | dict\[str, Any]): Additional conditions(s), if any, that are required for this event to trigger.

**Returns:**
An `OnUnlinkArtifact` object.


# RunStateFilter
Source: https://docs.wandb.ai/models/ref/python/automations/runstatefilter



<GitHubLink />

## <kbd>class</kbd> `RunStateFilter`

Represents a filter for triggering events based on changes in run states.

### <kbd>method</kbd> `RunStateFilter.__init__`

```python theme={null}
__init__(
    run: 'wandb.automations._filters.operators.And | wandb.automations._filters.operators.Or | wandb.automations._filters.operators.Nor | wandb.automations._filters.operators.Not | wandb.automations._filters.operators.Lt | wandb.automations._filters.operators.Gt | wandb.automations._filters.operators.Lte | wandb.automations._filters.operators.Gte | wandb.automations._filters.operators.Eq | wandb.automations._filters.operators.Ne | wandb.automations._filters.operators.In | wandb.automations._filters.operators.NotIn | wandb.automations._filters.operators.Exists | wandb.automations._filters.operators.Regex | wandb.automations._filters.operators.Contains | wandb.automations._filters.expressions.FilterExpr | dict[str, Any]' = And(()),
    state: 'StateFilter'
) → None
```

**Args:**

* `run` (wandb.automations.\_filters.operators.And | wandb.automations.\_filters.operators.Or | wandb.automations.\_filters.operators.Nor | wandb.automations.\_filters.operators.Not | wandb.automations.\_filters.operators.Lt | wandb.automations.\_filters.operators.Gt | wandb.automations.\_filters.operators.Lte | wandb.automations.\_filters.operators.Gte | wandb.automations.\_filters.operators.Eq | wandb.automations.\_filters.operators.Ne | wandb.automations.\_filters.operators.In | wandb.automations.\_filters.operators.NotIn | wandb.automations.\_filters.operators.Exists | wandb.automations.\_filters.operators.Regex | wandb.automations.\_filters.operators.Contains | wandb.automations.\_filters.expressions.FilterExpr | dict\[str, Any]): Filters that must match any runs that will trigger this event.
* `state` (StateFilter): Run state condition(s) that must be satisfied for this event to trigger.

**Returns:**
An `RunStateFilter` object.


# SendNotification
Source: https://docs.wandb.ai/models/ref/python/automations/sendnotification



<GitHubLink />

## <kbd>class</kbd> `SendNotification`

Defines an automation action that sends a (Slack) notification.

### <kbd>method</kbd> `SendNotification.__init__`

```python theme={null}
__init__(
    integration_id: 'str',
    message: 'str' = '',
    severity: 'AlertSeverity' = <AlertSeverity.INFO: 'INFO'>,
    title: 'str' = '',
    action_type: 'Literal[NOTIFICATION]' = NOTIFICATION
) → None
```

**Args:**

* `integration_id` (str): The ID of the Slack integration that will be used to send the notification.
* `message` (str): The message body of the sent notification.
* `severity` (AlertSeverity): The severity (`INFO`, `WARN`, `ERROR`) of the sent notification.
* `title` (str): The title of the sent notification.
* `action_type` (Literal\[NOTIFICATION]):

**Returns:**
An `SendNotification` object.

### <kbd>classmethod</kbd> `SendNotification.from_integration`

```python theme={null}
from_integration(
    integration: 'SlackIntegration',
    title: 'str' = '',
    text: 'str' = '',
    level: 'AlertSeverity' = <AlertSeverity.INFO: 'INFO'>
) → Self
```

Define a notification action that sends to the given (Slack) integration.


# SendWebhook
Source: https://docs.wandb.ai/models/ref/python/automations/sendwebhook



<GitHubLink />

## <kbd>class</kbd> `SendWebhook`

Defines an automation action that sends a webhook request.

### <kbd>method</kbd> `SendWebhook.__init__`

```python theme={null}
__init__(
    integration_id: 'str',
    request_payload: 'Annotated | None' = None,
    action_type: 'Literal[GENERIC_WEBHOOK]' = GENERIC_WEBHOOK
) → None
```

**Args:**

* `integration_id` (str): The ID of the webhook integration that will be used to send the request.
* `request_payload` (Optional\[Annotated]): The payload, possibly with template variables, to send in the webhook request.
* `action_type` (Literal\[GENERIC\_WEBHOOK]):

**Returns:**
An `SendWebhook` object.

### <kbd>classmethod</kbd> `SendWebhook.from_integration`

```python theme={null}
from_integration(
    integration: 'WebhookIntegration',
    payload: 'JsonEncoded[dict[str, Any]] | None' = None
) → Self
```

Define a webhook action that sends to the given (webhook) integration.


# Custom Charts overview
Source: https://docs.wandb.ai/models/ref/python/custom-charts

Use custom charts in the W&B Python SDK for interactive visualizations in project dashboards

Custom charts in W\&B are programmable through a group of functions in the `wandb.plot` namespace. These functions create interactive visualizations in W\&B project dashboards, and support common ML visualizations such as confusion matrices, ROC curves, and distribution plots.

## Available chart functions

| Function                                                                   | Description                                                                             |
| -------------------------------------------------------------------------- | --------------------------------------------------------------------------------------- |
| [`confusion_matrix()`](/models/ref/python/custom-charts/confusion_matrix/) | Generate confusion matrices for classification performance visualization.               |
| [`roc_curve()`](/models/ref/python/custom-charts/roc_curve/)               | Create Receiver Operating Characteristic curves for binary and multi-class classifiers. |
| [`pr_curve()`](/models/ref/python/custom-charts/pr_curve/)                 | Build Precision-Recall curves for classifier evaluation.                                |
| [`line()`](/models/ref/python/custom-charts/line/)                         | Construct line charts from tabular data.                                                |
| [`scatter()`](/models/ref/python/custom-charts/scatter/)                   | Create scatter plots for variable relationships.                                        |
| [`bar()`](/models/ref/python/custom-charts/bar/)                           | Generate bar charts for categorical data.                                               |
| [`histogram()`](/models/ref/python/custom-charts/histogram/)               | Build histograms for data distribution analysis.                                        |
| [`line_series()`](/models/ref/python/custom-charts/line_series/)           | Plot multiple line series on a single chart.                                            |
| [`plot_table()`](/models/ref/python/custom-charts/plot_table/)             | Create custom charts using Vega-Lite specifications.                                    |

## Common use cases

### Model evaluation

* **Classification**: `confusion_matrix()`, `roc_curve()`, and `pr_curve()` for classifier evaluation
* **Regression**: `scatter()` for prediction vs. actual plots and `histogram()` for residual analysis
* **Vega-Lite Charts**: `plot_table()` for domain-specific visualizations

### Training monitoring

* **Learning Curves**: `line()` or `line_series()` for tracking metrics over epochs
* **Hyperparameter Comparison**: `bar()` charts for comparing configurations

### Data analysis

* **Distribution Analysis**: `histogram()` for feature distributions
* **Correlation Analysis**: `scatter()` plots for variable relationships

## Getting started

### Log a confusion matrix

```python theme={null}
import wandb

y_true = [0, 1, 2, 0, 1, 2]
y_pred = [0, 2, 2, 0, 1, 1]
class_names = ["class_0", "class_1", "class_2"]

# Initialize a run
with wandb.init(project="custom-charts-demo") as run:
    run.log({
        "conf_mat": wandb.plot.confusion_matrix(
            y_true=y_true, 
            preds=y_pred,
            class_names=class_names
        )
    })
```

### Build a scatter plot for feature analysis

```python theme={null}
import numpy as np

# Generate synthetic data
data_table = wandb.Table(columns=["feature_1", "feature_2", "label"])

with wandb.init(project="custom-charts-demo") as run:

    for _ in range(100):
        data_table.add_data(
            np.random.randn(), 
            np.random.randn(), 
            np.random.choice(["A", "B"])
        )

    run.log({
        "feature_scatter": wandb.plot.scatter(
            data_table, x="feature_1", y="feature_2",
            title="Feature Distribution"
        )
    })

```


# bar()
Source: https://docs.wandb.ai/models/ref/python/custom-charts/bar



<GitHubLink />

### <kbd>function</kbd> `bar`

```python theme={null}
bar(
    table: 'wandb.Table',
    label: 'str',
    value: 'str',
    title: 'str' = '',
    split_table: 'bool' = False
) → CustomChart
```

Constructs a bar chart from a wandb.Table of data.

**Args:**

* `table`:  A table containing the data for the bar chart.
* `label`:  The name of the column to use for the labels of each bar.
* `value`:  The name of the column to use for the values of each bar.
* `title`:  The title of the bar chart.
* `split_table`:  Whether the table should be split into a separate section  in the W\&B UI. If `True`, the table will be displayed in a section named  "Custom Chart Tables". Default is `False`.

**Returns:**

* `CustomChart`:  A custom chart object that can be logged to W\&B. To log the  chart, pass it to `wandb.log()`.

**Example:**

```python theme={null}
import random
import wandb

# Generate random data for the table
data = [
    ["car", random.uniform(0, 1)],
    ["bus", random.uniform(0, 1)],
    ["road", random.uniform(0, 1)],
    ["person", random.uniform(0, 1)],
]

# Create a table with the data
table = wandb.Table(data=data, columns=["class", "accuracy"])

# Initialize a W&B run and log the bar plot
with wandb.init(project="bar_chart") as run:
    # Create a bar plot from the table
    bar_plot = wandb.plot.bar(
         table=table,
         label="class",
         value="accuracy",
         title="Object Classification Accuracy",
    )

    # Log the bar chart to W&B
    run.log({"bar_plot": bar_plot})
```


# confusion_matrix()
Source: https://docs.wandb.ai/models/ref/python/custom-charts/confusion_matrix



<GitHubLink />

### <kbd>function</kbd> `confusion_matrix`

```python theme={null}
confusion_matrix(
    probs: 'Sequence[Sequence[float]] | None' = None,
    y_true: 'Sequence[T] | None' = None,
    preds: 'Sequence[T] | None' = None,
    class_names: 'Sequence[str] | None' = None,
    title: 'str' = 'Confusion Matrix Curve',
    split_table: 'bool' = False
) → CustomChart
```

Constructs a confusion matrix from a sequence of probabilities or predictions.

**Args:**

* `probs`:  A sequence of predicted probabilities for each  class. The sequence shape should be (N, K) where N is the number of samples  and K is the number of classes. If provided, `preds` should not be provided.
* `y_true`:  A sequence of true labels.
* `preds`:  A sequence of predicted class labels. If provided,  `probs` should not be provided.
* `class_names`:  Sequence of class names. If not  provided, class names will be defined as "Class\_1", "Class\_2", etc.
* `title`:  Title of the confusion matrix chart.
* `split_table`:  Whether the table should be split into a separate section  in the W\&B UI. If `True`, the table will be displayed in a section named  "Custom Chart Tables". Default is `False`.

**Returns:**

* `CustomChart`:  A custom chart object that can be logged to W\&B. To log the  chart, pass it to `wandb.log()`.

**Raises:**

* `ValueError`:  If both `probs` and `preds` are provided or if the number of  predictions and true labels are not equal. If the number of unique  predicted classes exceeds the number of class names or if the number of  unique true labels exceeds the number of class names.
* `wandb.Error`:  If numpy is not installed.

**Examples:**
Logging a confusion matrix with random probabilities for wildlife classification:

```python theme={null}
import numpy as np
import wandb

# Define class names for wildlife
wildlife_class_names = ["Lion", "Tiger", "Elephant", "Zebra"]

# Generate random true labels (0 to 3 for 10 samples)
wildlife_y_true = np.random.randint(0, 4, size=10)

# Generate random probabilities for each class (10 samples x 4 classes)
wildlife_probs = np.random.rand(10, 4)
wildlife_probs = np.exp(wildlife_probs) / np.sum(
    np.exp(wildlife_probs),
    axis=1,
    keepdims=True,
)

# Initialize W&B run and log confusion matrix
with wandb.init(project="wildlife_classification") as run:
    confusion_matrix = wandb.plot.confusion_matrix(
         probs=wildlife_probs,
         y_true=wildlife_y_true,
         class_names=wildlife_class_names,
         title="Wildlife Classification Confusion Matrix",
    )
    run.log({"wildlife_confusion_matrix": confusion_matrix})
```

In this example, random probabilities are used to generate a confusion matrix.

Logging a confusion matrix with simulated model predictions and 85% accuracy:

```python theme={null}
import numpy as np
import wandb

# Define class names for wildlife
wildlife_class_names = ["Lion", "Tiger", "Elephant", "Zebra"]

# Simulate true labels for 200 animal images (imbalanced distribution)
wildlife_y_true = np.random.choice(
    [0, 1, 2, 3],
    size=200,
    p=[0.2, 0.3, 0.25, 0.25],
)

# Simulate model predictions with 85% accuracy
wildlife_preds = [
    y_t
    if np.random.rand() < 0.85
    else np.random.choice([x for x in range(4) if x != y_t])
    for y_t in wildlife_y_true
]

# Initialize W&B run and log confusion matrix
with wandb.init(project="wildlife_classification") as run:
    confusion_matrix = wandb.plot.confusion_matrix(
         preds=wildlife_preds,
         y_true=wildlife_y_true,
         class_names=wildlife_class_names,
         title="Simulated Wildlife Classification Confusion Matrix",
    )
    run.log({"wildlife_confusion_matrix": confusion_matrix})
```

In this example, predictions are simulated with 85% accuracy to generate a confusion matrix.


# histogram
Source: https://docs.wandb.ai/models/ref/python/custom-charts/histogram



<GitHubLink />

### <kbd>function</kbd> `histogram`

```python theme={null}
histogram(
    table: 'wandb.Table',
    value: 'str',
    title: 'str' = '',
    split_table: 'bool' = False
) → CustomChart
```

Constructs a histogram chart from a W\&B Table.

**Args:**

* `table`:  The W\&B Table containing the data for the histogram.
* `value`:  The label for the bin axis (x-axis).
* `title`:  The title of the histogram plot.
* `split_table`:  Whether the table should be split into a separate section  in the W\&B UI. If `True`, the table will be displayed in a section named  "Custom Chart Tables". Default is `False`.

**Returns:**

* `CustomChart`:  A custom chart object that can be logged to W\&B. To log the  chart, pass it to `wandb.log()`.

**Example:**

```python theme={null}
import math
import random
import wandb

# Generate random data
data = [[i, random.random() + math.sin(i / 10)] for i in range(100)]

# Create a W&B Table
table = wandb.Table(
    data=data,
    columns=["step", "height"],
)

# Create a histogram plot
histogram = wandb.plot.histogram(
    table,
    value="height",
    title="My Histogram",
)

# Log the histogram plot to W&B
with wandb.init(...) as run:
    run.log({"histogram-plot1": histogram})
```


# line()
Source: https://docs.wandb.ai/models/ref/python/custom-charts/line



<GitHubLink />

### <kbd>function</kbd> `line`

```python theme={null}
line(
    table: 'wandb.Table',
    x: 'str',
    y: 'str',
    stroke: 'str | None' = None,
    title: 'str' = '',
    split_table: 'bool' = False
) → CustomChart
```

Constructs a customizable line chart.

**Args:**

* `table`:   The table containing data for the chart.
* `x`:  Column name for the x-axis values.
* `y`:  Column name for the y-axis values.
* `stroke`:  Column name to differentiate line strokes (e.g., for  grouping lines).
* `title`:  Title of the chart.
* `split_table`:  Whether the table should be split into a separate section  in the W\&B UI. If `True`, the table will be displayed in a section named  "Custom Chart Tables". Default is `False`.

**Returns:**

* `CustomChart`:  A custom chart object that can be logged to W\&B. To log the  chart, pass it to `wandb.log()`.

**Example:**

```python theme={null}
import math
import random
import wandb

# Create multiple series of data with different patterns
data = []
for i in range(100):
     # Series 1: Sinusoidal pattern with random noise
     data.append([i, math.sin(i / 10) + random.uniform(-0.1, 0.1), "series_1"])
     # Series 2: Cosine pattern with random noise
     data.append([i, math.cos(i / 10) + random.uniform(-0.1, 0.1), "series_2"])
     # Series 3: Linear increase with random noise
     data.append([i, i / 10 + random.uniform(-0.5, 0.5), "series_3"])

# Define the columns for the table
table = wandb.Table(data=data, columns=["step", "value", "series"])

# Initialize wandb run and log the line chart
with wandb.init(project="line_chart_example") as run:
     line_chart = wandb.plot.line(
         table=table,
         x="step",
         y="value",
         stroke="series",  # Group by the "series" column
         title="Multi-Series Line Plot",
     )
     run.log({"line-chart": line_chart})
```


# line_series()
Source: https://docs.wandb.ai/models/ref/python/custom-charts/line_series



<GitHubLink />

### <kbd>function</kbd> `line_series`

```python theme={null}
line_series(
    xs: 'Iterable[Iterable[Any]] | Iterable[Any]',
    ys: 'Iterable[Iterable[Any]]',
    keys: 'Iterable[str] | None' = None,
    title: 'str' = '',
    xname: 'str' = 'x',
    split_table: 'bool' = False
) → CustomChart
```

Constructs a line series chart.

**Args:**

* `xs`:  Sequence of x values. If a singular  array is provided, all y values are plotted against that x array. If  an array of arrays is provided, each y value is plotted against the  corresponding x array.
* `ys`:  Sequence of y values, where each iterable represents  a separate line series.
* `keys`:  Sequence of keys for labeling each line series. If  not provided, keys will be automatically generated as "line\_1",  "line\_2", etc.
* `title`:  Title of the chart.
* `xname`:  Label for the x-axis.
* `split_table`:  Whether the table should be split into a separate section  in the W\&B UI. If `True`, the table will be displayed in a section named  "Custom Chart Tables". Default is `False`.

**Returns:**

* `CustomChart`:  A custom chart object that can be logged to W\&B. To log the  chart, pass it to `wandb.log()`.

**Examples:**
Logging a single x array where all y series are plotted against the same x values:

```python theme={null}
import wandb

# Initialize W&B run
with wandb.init(project="line_series_example") as run:
    # x values shared across all y series
    xs = list(range(10))

    # Multiple y series to plot
    ys = [
         [i for i in range(10)],  # y = x
         [i**2 for i in range(10)],  # y = x^2
         [i**3 for i in range(10)],  # y = x^3
    ]

    # Generate and log the line series chart
    line_series_chart = wandb.plot.line_series(
         xs,
         ys,
         title="title",
         xname="step",
    )
    run.log({"line-series-single-x": line_series_chart})
```

In this example, a single `xs` series (shared x-values) is used for all `ys` series. This results in each y-series being plotted against the same x-values (0-9).

Logging multiple x arrays where each y series is plotted against its corresponding x array:

```python theme={null}
import wandb

# Initialize W&B run
with wandb.init(project="line_series_example") as run:
    # Separate x values for each y series
    xs = [
         [i for i in range(10)],  # x for first series
         [2 * i for i in range(10)],  # x for second series (stretched)
         [3 * i for i in range(10)],  # x for third series (stretched more)
    ]

    # Corresponding y series
    ys = [
         [i for i in range(10)],  # y = x
         [i**2 for i in range(10)],  # y = x^2
         [i**3 for i in range(10)],  # y = x^3
    ]

    # Generate and log the line series chart
    line_series_chart = wandb.plot.line_series(
         xs, ys, title="Multiple X Arrays Example", xname="Step"
    )
    run.log({"line-series-multiple-x": line_series_chart})
```

In this example, each y series is plotted against its own unique x series. This allows for more flexibility when the x values are not uniform across the data series.

Customizing line labels using `keys`:

```python theme={null}
import wandb

# Initialize W&B run
with wandb.init(project="line_series_example") as run:
    xs = list(range(10))  # Single x array
    ys = [
         [i for i in range(10)],  # y = x
         [i**2 for i in range(10)],  # y = x^2
         [i**3 for i in range(10)],  # y = x^3
    ]

    # Custom labels for each line
    keys = ["Linear", "Quadratic", "Cubic"]

    # Generate and log the line series chart
    line_series_chart = wandb.plot.line_series(
         xs,
         ys,
         keys=keys,  # Custom keys (line labels)
         title="Custom Line Labels Example",
         xname="Step",
    )
    run.log({"line-series-custom-keys": line_series_chart})
```

This example shows how to provide custom labels for the lines using the `keys` argument. The keys will appear in the legend as "Linear", "Quadratic", and "Cubic".


# plot_table()
Source: https://docs.wandb.ai/models/ref/python/custom-charts/plot_table



<GitHubLink />

### <kbd>function</kbd> `plot_table`

```python theme={null}
plot_table(
    vega_spec_name: 'str',
    data_table: 'wandb.Table',
    fields: 'dict[str, Any]',
    string_fields: 'dict[str, Any] | None' = None,
    split_table: 'bool' = False
) → CustomChart
```

Creates a custom charts using a Vega-Lite specification and a `wandb.Table`.

This function creates a custom chart based on a Vega-Lite specification and a data table represented by a `wandb.Table` object. The specification needs to be predefined and stored in the W\&B backend. The function returns a custom chart object that can be logged to W\&B using `wandb.Run.log()`.

**Args:**

* `vega_spec_name`:  The name or identifier of the Vega-Lite spec  that defines the visualization structure.
* `data_table`:  A `wandb.Table` object containing the data to be  visualized.
* `fields`:  A mapping between the fields in the Vega-Lite spec and the  corresponding columns in the data table to be visualized.
* `string_fields`:  A dictionary for providing values for any string constants  required by the custom visualization.
* `split_table`:  Whether the table should be split into a separate section  in the W\&B UI. If `True`, the table will be displayed in a section named  "Custom Chart Tables". Default is `False`.

**Returns:**

* `CustomChart`:  A custom chart object that can be logged to W\&B. To log the  chart, pass the chart object as argument to `wandb.Run.log()`.

**Raises:**

* `wandb.Error`:  If `data_table` is not a `wandb.Table` object.

**Example:**

```python theme={null}
# Create a custom chart using a Vega-Lite spec and the data table.
import wandb

data = [[1, 1], [2, 2], [3, 3], [4, 4], [5, 5]]
table = wandb.Table(data=data, columns=["x", "y"])
fields = {"x": "x", "y": "y", "title": "MY TITLE"}

with wandb.init() as run:
   # Training code goes here

   # Create a custom title with `string_fields`.
   my_custom_chart = wandb.plot_table(
        vega_spec_name="wandb/line/v0",
        data_table=table,
        fields=fields,
        string_fields={"title": "Title"},
   )

   run.log({"custom_chart": my_custom_chart})
```


# pr_curve()
Source: https://docs.wandb.ai/models/ref/python/custom-charts/pr_curve



<GitHubLink />

### <kbd>function</kbd> `pr_curve`

```python theme={null}
pr_curve(
    y_true: 'Iterable[T] | None' = None,
    y_probas: 'Iterable[numbers.Number] | None' = None,
    labels: 'list[str] | None' = None,
    classes_to_plot: 'list[T] | None' = None,
    interp_size: 'int' = 21,
    title: 'str' = 'Precision-Recall Curve',
    split_table: 'bool' = False
) → CustomChart
```

Constructs a Precision-Recall (PR) curve.

The Precision-Recall curve is particularly useful for evaluating classifiers on imbalanced datasets. A high area under the PR curve signifies both high precision (a low false positive rate) and high recall (a low false negative rate). The curve provides insights into the balance between false positives and false negatives at various threshold levels, aiding in the assessment of a model's performance.

**Args:**

* `y_true`:  True binary labels. The shape should be (`num_samples`,).
* `y_probas`:  Predicted scores or probabilities for each class.  These can be probability estimates, confidence scores, or non-thresholded  decision values. The shape should be (`num_samples`, `num_classes`).
* `labels`:  Optional list of class names to replace  numeric values in `y_true` for easier plot interpretation.  For example, `labels = ['dog', 'cat', 'owl']` will replace 0 with  'dog', 1 with 'cat', and 2 with 'owl' in the plot. If not provided,  numeric values from `y_true` will be used.
* `classes_to_plot`:  Optional list of unique class values from  y\_true to be included in the plot. If not specified, all unique  classes in y\_true will be plotted.
* `interp_size`:  Number of points to interpolate recall values. The  recall values will be fixed to `interp_size` uniformly distributed  points in the range \[0, 1], and the precision will be interpolated  accordingly.
* `title`:  Title of the plot. Defaults to "Precision-Recall Curve".
* `split_table`:  Whether the table should be split into a separate section  in the W\&B UI. If `True`, the table will be displayed in a section named  "Custom Chart Tables". Default is `False`.

**Returns:**

* `CustomChart`:  A custom chart object that can be logged to W\&B. To log the  chart, pass it to `wandb.log()`.

**Raises:**

* `wandb.Error`:  If NumPy, pandas, or scikit-learn is not installed.

**Example:**

```python theme={null}
import wandb

# Example for spam detection (binary classification)
y_true = [0, 1, 1, 0, 1]  # 0 = not spam, 1 = spam
y_probas = [
    [0.9, 0.1],  # Predicted probabilities for the first sample (not spam)
    [0.2, 0.8],  # Second sample (spam), and so on
    [0.1, 0.9],
    [0.8, 0.2],
    [0.3, 0.7],
]

labels = ["not spam", "spam"]  # Optional class names for readability

with wandb.init(project="spam-detection") as run:
    pr_curve = wandb.plot.pr_curve(
         y_true=y_true,
         y_probas=y_probas,
         labels=labels,
         title="Precision-Recall Curve for Spam Detection",
    )
    run.log({"pr-curve": pr_curve})
```


# roc_curve()
Source: https://docs.wandb.ai/models/ref/python/custom-charts/roc_curve



<GitHubLink />

### <kbd>function</kbd> `roc_curve`

```python theme={null}
roc_curve(
    y_true: 'Sequence[numbers.Number]',
    y_probas: 'Sequence[Sequence[float]] | None' = None,
    labels: 'list[str] | None' = None,
    classes_to_plot: 'list[numbers.Number] | None' = None,
    title: 'str' = 'ROC Curve',
    split_table: 'bool' = False
) → CustomChart
```

Constructs Receiver Operating Characteristic (ROC) curve chart.

**Args:**

* `y_true`:  The true class labels (ground truth)  for the target variable. Shape should be (num\_samples,).
* `y_probas`:  The predicted probabilities or  decision scores for each class. Shape should be (num\_samples, num\_classes).
* `labels`:  Human-readable labels corresponding to the class  indices in `y_true`. For example, if `labels=['dog', 'cat']`,  class 0 will be displayed as 'dog' and class 1 as 'cat' in the plot.  If None, the raw class indices from `y_true` will be used.  Default is None.
* `classes_to_plot`:  A subset of unique class labels  to include in the ROC curve. If None, all classes in `y_true` will  be plotted. Default is None.
* `title`:  Title of the ROC curve plot. Default is "ROC Curve".
* `split_table`:  Whether the table should be split into a separate  section in the W\&B UI. If `True`, the table will be displayed in a  section named "Custom Chart Tables". Default is `False`.

**Returns:**

* `CustomChart`:  A custom chart object that can be logged to W\&B. To log the  chart, pass it to `wandb.log()`.

**Raises:**

* `wandb.Error`:  If numpy, pandas, or scikit-learn are not found.

**Example:**

```python theme={null}
import numpy as np
import wandb

# Simulate a medical diagnosis classification problem with three diseases
n_samples = 200
n_classes = 3

# True labels: assign "Diabetes", "Hypertension", or "Heart Disease" to
# each sample
disease_labels = ["Diabetes", "Hypertension", "Heart Disease"]
# 0: Diabetes, 1: Hypertension, 2: Heart Disease
y_true = np.random.choice([0, 1, 2], size=n_samples)

# Predicted probabilities: simulate predictions, ensuring they sum to 1
# for each sample
y_probas = np.random.dirichlet(np.ones(n_classes), size=n_samples)

# Specify classes to plot (plotting all three diseases)
classes_to_plot = [0, 1, 2]

# Initialize a W&B run and log a ROC curve plot for disease classification
with wandb.init(project="medical_diagnosis") as run:
   roc_plot = wandb.plot.roc_curve(
        y_true=y_true,
        y_probas=y_probas,
        labels=disease_labels,
        classes_to_plot=classes_to_plot,
        title="ROC Curve for Disease Classification",
   )
   run.log({"roc-curve": roc_plot})
```


# scatter()
Source: https://docs.wandb.ai/models/ref/python/custom-charts/scatter



<GitHubLink />

### <kbd>function</kbd> `scatter`

```python theme={null}
scatter(
    table: 'wandb.Table',
    x: 'str',
    y: 'str',
    title: 'str' = '',
    split_table: 'bool' = False
) → CustomChart
```

Constructs a scatter plot from a wandb.Table of data.

**Args:**

* `table`:  The W\&B Table containing the data to visualize.
* `x`:  The name of the column used for the x-axis.
* `y`:  The name of the column used for the y-axis.
* `title`:  The title of the scatter chart.
* `split_table`:  Whether the table should be split into a separate section  in the W\&B UI. If `True`, the table will be displayed in a section named  "Custom Chart Tables". Default is `False`.

**Returns:**

* `CustomChart`:  A custom chart object that can be logged to W\&B. To log the  chart, pass it to `wandb.log()`.

**Example:**

```python theme={null}
import math
import random
import wandb

# Simulate temperature variations at different altitudes over time
data = [
   [i, random.uniform(-10, 20) - 0.005 * i + 5 * math.sin(i / 50)]
   for i in range(300)
]

# Create W&B table with altitude (m) and temperature (°C) columns
table = wandb.Table(data=data, columns=["altitude (m)", "temperature (°C)"])

# Initialize W&B run and log the scatter plot
with wandb.init(project="temperature-altitude-scatter") as run:
   # Create and log the scatter plot
   scatter_plot = wandb.plot.scatter(
        table=table,
        x="altitude (m)",
        y="temperature (°C)",
        title="Altitude vs Temperature",
   )
   run.log({"altitude-temperature-scatter": scatter_plot})
```


# Data Types overview
Source: https://docs.wandb.ai/models/ref/python/data-types

Data Types in the W&B Python SDK for logging media and structured data

Data Types in W\&B are classes that wrap media and structured data for logging to runs. They include visualization components in the W\&B UI and handle data serialization, storage, and retrieval.

## Available data types

| Data Type                                            | Description                                                          |
| ---------------------------------------------------- | -------------------------------------------------------------------- |
| [`Image`](/models/ref/python/data-types/image)       | Log images with support for masks, bounding boxes, and segmentation. |
| [`Video`](/models/ref/python/data-types/video)       | Track video data for model outputs or dataset samples.               |
| [`Audio`](/models/ref/python/data-types/audio)       | Log audio samples for audio processing tasks.                        |
| [`Table`](/models/ref/python/data-types/table)       | Create tables that can contain mixed media types.                    |
| [`Plotly`](/models/ref/python/data-types/plotly)     | Log Plotly charts for data visualization.                            |
| [`Html`](/models/ref/python/data-types/html)         | Embed custom HTML content.                                           |
| [`Object3D`](/models/ref/python/data-types/object3d) | Visualize 3D point clouds and meshes.                                |
| [`Molecule`](/models/ref/python/data-types/molecule) | Log molecular structures for computational chemistry.                |

## Examples

This example uses an `Image`:

```python theme={null}
import wandb
import matplotlib.pyplot as plt

# Generate an image for demonstration purposes
path_to_img = "/path/to/cafe.png"
im = plt.imread(path_to_img)

# Initialize a new run
with wandb.init(project="awesome-project") as run:

    # Log the image
    run.log({"img": [wandb.Image(im, caption="Cafe")]})
```

This example uses a `Table` to log a table with mixed text and labels:

```python theme={null}
import wandb

# Initialize a new run
with wandb.init(project="visualize-predictions", name="tables") as run:

    # Create tabular data, using a list of lists
    data = [["Cat", "1", "1"],["Dog", "0", "-1"]]
    run.log({"Table 1": wandb.Table(data=data, columns=["Text", "Predicted Label", "True Label"])})

    # Create tabular data, using `wandb.Table.add_data()` method
    table = wandb.Table(columns=["Text", "Predicted Label", "True Label"])
    table.add_data("Cat", "1", "1")
    table.add_data("Dog", "0", "-1")
    run.log({"Table 2": table})
```


# Audio
Source: https://docs.wandb.ai/models/ref/python/data-types/audio



<GitHubLink />

## <kbd>class</kbd> `Audio`

W\&B class for audio clips.

### <kbd>method</kbd> `Audio.__init__`

```python theme={null}
__init__(
    data_or_path: 'str | pathlib.Path | list | np.ndarray',
    sample_rate: 'int | None' = None,
    caption: 'str | None' = None
)
```

Accept a path to an audio file or a numpy array of audio data.

**Args:**

* `data_or_path`:  A path to an audio file or a NumPy array of audio data.
* `sample_rate`:  Sample rate, required when passing in raw NumPy array of audio data.
* `caption`:  Caption to display with audio.

***

### <kbd>classmethod</kbd> `Audio.durations`

```python theme={null}
durations(audio_list)
```

Calculate the duration of the audio files.

***

### <kbd>classmethod</kbd> `Audio.sample_rates`

```python theme={null}
sample_rates(audio_list)
```

Get sample rates of the audio files.

***


# box3d()
Source: https://docs.wandb.ai/models/ref/python/data-types/box3d



<GitHubLink />

### <kbd>function</kbd> `box3d`

```python theme={null}
box3d(
    center: 'npt.ArrayLike',
    size: 'npt.ArrayLike',
    orientation: 'npt.ArrayLike',
    color: 'RGBColor',
    label: 'str | None' = None,
    score: 'numeric | None' = None
) → Box3D
```

A 3D bounding box. The box is specified by its center, size and orientation.

**Args:**

* `center`:  The center point of the box as a length-3 ndarray.
* `size`:  The box's X, Y and Z dimensions as a length-3 ndarray.
* `orientation`:  The rotation transforming global XYZ coordinates  into the box's local XYZ coordinates, given as a length-4  ndarray \[r, x, y, z] corresponding to the non-zero quaternion  r + xi + yj + zk.
* `color`:  The box's color as an (r, g, b) tuple with `0 <= r,g,b <= 1`.
* `label`:  An optional label for the box.
* `score`:  An optional score for the box. Typically used to indicate  the confidence of a detection.

**Returns:**
A Box3D object.

**Example:**
The following example creates a point cloud with 60 boxes rotating around the X, Y and Z axes.

```python theme={null}
import wandb

import math
import numpy as np
from scipy.spatial.transform import Rotation


with wandb.init() as run:
    run.log(
         {
             "points": wandb.Object3D.from_point_cloud(
                 points=np.random.uniform(-5, 5, size=(100, 3)),
                 boxes=[
                     wandb.box3d(
                         center=(0.3 * t - 3, 0, 0),
                         size=(0.1, 0.1, 0.1),
                         orientation=Rotation.from_euler(
                             "xyz", [t * math.pi / 10, 0, 0]
                         ).as_quat(),
                         color=(0.5 + t / 40, 0.5, 0.5),
                         label=f"box {t}",
                         score=0.9,
                     )
                     for t in range(20)
                 ]
                 + [
                     wandb.box3d(
                         center=(0, 0.3 * t - 3, 0.3),
                         size=(0.1, 0.1, 0.1),
                         orientation=Rotation.from_euler(
                             "xyz", [0, t * math.pi / 10, 0]
                         ).as_quat(),
                         color=(0.5, 0.5 + t / 40, 0.5),
                         label=f"box {t}",
                         score=0.9,
                     )
                     for t in range(20)
                 ]
                 + [
                     wandb.box3d(
                         center=(0.3, 0.3, 0.3 * t - 3),
                         size=(0.1, 0.1, 0.1),
                         orientation=Rotation.from_euler(
                             "xyz", [0, 0, t * math.pi / 10]
                         ).as_quat(),
                         color=(0.5, 0.5, 0.5 + t / 40),
                         label=f"box {t}",
                         score=0.9,
                     )
                     for t in range(20)
                 ],
             ),
         }
    )
```


# Histogram
Source: https://docs.wandb.ai/models/ref/python/data-types/histogram



<GitHubLink />

## <kbd>class</kbd> `Histogram`

W\&B class for histograms.

This object works just like numpy's histogram function [https://docs.scipy.org/doc/numpy/reference/generated/numpy.histogram.html](https://docs.scipy.org/doc/numpy/reference/generated/numpy.histogram.html)

**Attributes:**

* `bins` (\[float]):  Edges of bins
* `histogram` (\[int]):  Number of elements falling in each bin.

### <kbd>method</kbd> `Histogram.__init__`

```python theme={null}
__init__(
    sequence: 'Sequence | None' = None,
    np_histogram: 'NumpyHistogram | None' = None,
    num_bins: 'int' = 64
) → None
```

Initialize a Histogram object.

**Args:**
sequence: Input data for histogram. np\_histogram: Alternative input of a precomputed histogram. num\_bins: Number of bins for the histogram.  The default number of bins  is 64. The maximum number of bins is 512.

**Examples:**
Generate histogram from a sequence.

```python theme={null}
import wandb

wandb.Histogram([1, 2, 3])
```

Efficiently initialize from np.histogram.

```python theme={null}
import numpy as np
import wandb

hist = np.histogram(data)
wandb.Histogram(np_histogram=hist)
```

***


# Html
Source: https://docs.wandb.ai/models/ref/python/data-types/html



<GitHubLink />

## <kbd>class</kbd> `Html`

W\&B class for logging HTML content to W\&B.

### <kbd>method</kbd> `Html.__init__`

```python theme={null}
__init__(
    data: 'str | pathlib.Path | TextIO',
    inject: 'bool' = True,
    data_is_not_path: 'bool' = False
) → None
```

Creates a W\&B HTML object.

**Args:**
data:  A string that is a path to a file with the extension ".html",  or a string or IO object containing literal HTML.

* `inject`:  Add a stylesheet to the HTML object. If set  to False the HTML will pass through unchanged.
* `data_is_not_path`:  If set to False, the data will be  treated as a path to a file.

**Examples:**
It can be initialized by providing a path to a file:

```python theme={null}
with wandb.init() as run:
    run.log({"html": wandb.Html("./index.html")})
```

Alternatively, it can be initialized by providing literal HTML, in either a string or IO object:

```python theme={null}
with wandb.init() as run:
    run.log({"html": wandb.Html("<h1>Hello, world!</h1>")})
```

***


# Image
Source: https://docs.wandb.ai/models/ref/python/data-types/image



<GitHubLink />

## <kbd>class</kbd> `Image`

A class for logging images to W\&B.

### <kbd>method</kbd> `Image.__init__`

```python theme={null}
__init__(
    data_or_path: 'ImageDataOrPathType',
    mode: 'str | None' = None,
    caption: 'str | None' = None,
    grouping: 'int | None' = None,
    classes: 'Classes | Sequence[dict] | None' = None,
    boxes: 'dict[str, BoundingBoxes2D] | dict[str, dict] | None' = None,
    masks: 'dict[str, ImageMask] | dict[str, dict] | None' = None,
    file_type: 'str | None' = None,
    normalize: 'bool' = True
) → None
```

Initialize a `wandb.Image` object.

This class handles various image data formats and automatically normalizes pixel values to the range \[0, 255] when needed, ensuring compatibility with the W\&B backend.

* Data in range \[0, 1] is multiplied by 255 and converted to uint8 \* Data in range \[-1, 1] is rescaled from \[-1, 1] to \[0, 255] by mapping
  -1 to 0 and 1 to 255, then converted to uint8 \* Data outside \[-1, 1] but not in \[0, 255] is clipped to \[0, 255] and  converted to uint8 (with a warning if values fall outside \[0, 255]) \* Data already in \[0, 255] is converted to uint8 without modification

**Args:**

* `data_or_path`:  Accepts NumPy array/pytorch tensor of image data,  a PIL image object, or a path to an image file. If a NumPy  array or pytorch tensor is provided,  the image data will be saved to the given file type.  If the values are not in the range \[0, 255] or all values are in the range \[0, 1],  the image pixel values will be normalized to the range \[0, 255]  unless `normalize` is set to `False`.
  * pytorch tensor should be in the format (channel, height, width)
  * NumPy array should be in the format (height, width, channel)
* `mode`:  The PIL mode for an image. Most common are "L", "RGB", "RGBA".
* `Full Pillow docs for more information https`: //pillow\.readthedocs.io/en/stable/handbook/concepts.html#modes
* `caption`:  Label for display of image.
* `grouping`:  The grouping number for the image.
* `classes`:  A list of class information for the image,  used for labeling bounding boxes, and image masks.
* `boxes`:  A dictionary containing bounding box information for the image.
* `see https`: //docs.wandb.ai/models/ref/python/data-types/boundingboxes2d
* `masks`:  A dictionary containing mask information for the image.
* `see https`: //docs.wandb.ai/models/ref/python/data-types/imagemask
* `file_type`:  The file type to save the image as.  This parameter has no effect if `data_or_path` is a path to an image file.
* `normalize`:  If `True`, normalize the image pixel values to fall within the range of \[0, 255].  Normalize is only applied if `data_or_path` is a numpy array or pytorch tensor.

**Examples:**
Create a wandb.Image from a numpy array

```python theme={null}
import numpy as np
import wandb

with wandb.init() as run:
    examples = []
    for i in range(3):
         pixels = np.random.randint(low=0, high=256, size=(100, 100, 3))
         image = wandb.Image(pixels, caption=f"random field {i}")
         examples.append(image)
    run.log({"examples": examples})
```

Create a wandb.Image from a PILImage

```python theme={null}
import numpy as np
from PIL import Image as PILImage
import wandb

with wandb.init() as run:
    examples = []
    for i in range(3):
         pixels = np.random.randint(
             low=0, high=256, size=(100, 100, 3), dtype=np.uint8
         )
         pil_image = PILImage.fromarray(pixels, mode="RGB")
         image = wandb.Image(pil_image, caption=f"random field {i}")
         examples.append(image)
    run.log({"examples": examples})
```

Log .jpg rather than .png (default)

```python theme={null}
import numpy as np
import wandb

with wandb.init() as run:
    examples = []
    for i in range(3):
         pixels = np.random.randint(low=0, high=256, size=(100, 100, 3))
         image = wandb.Image(
             pixels, caption=f"random field {i}", file_type="jpg"
         )
         examples.append(image)
    run.log({"examples": examples})
```

***

### <kbd>property</kbd> Image.image

***


# Molecule
Source: https://docs.wandb.ai/models/ref/python/data-types/molecule



<GitHubLink />

## <kbd>class</kbd> `Molecule`

W\&B class for 3D Molecular data.

### <kbd>method</kbd> `Molecule.__init__`

```python theme={null}
__init__(
    data_or_path: 'str | pathlib.Path | TextIO',
    caption: 'str | None' = None,
    **kwargs: 'str'
) → None
```

Initialize a Molecule object.

**Args:**

* `data_or_path`:  Molecule can be initialized from a file name or an io object.
* `caption`:  Caption associated with the molecule for display.

***


# Object3D
Source: https://docs.wandb.ai/models/ref/python/data-types/object3d



<GitHubLink />

## <kbd>class</kbd> `Object3D`

W\&B class for 3D point clouds.

### <kbd>method</kbd> `Object3D.__init__`

```python theme={null}
__init__(
    data_or_path: 'np.ndarray | str | pathlib.Path | TextIO | dict',
    caption: 'str | None' = None,
    **kwargs: 'str | FileFormat3D | None'
) → None
```

Creates a W\&B Object3D object.

**Args:**

* `data_or_path`:  Object3D can be initialized from a file or a numpy array.
* `caption`:  Caption associated with the object for display.

**Examples:**
The shape of the numpy array must be one of either

```text theme={null}
[[x y z],       ...] nx3
[[x y z c],     ...] nx4 where c is a category with supported range [1, 14]
[[x y z r g b], ...] nx6 where is rgb is color
```

***


# Plotly
Source: https://docs.wandb.ai/models/ref/python/data-types/plotly



<GitHubLink />

## <kbd>class</kbd> `Plotly`

W\&B class for Plotly plots.

### <kbd>method</kbd> `Plotly.__init__`

```python theme={null}
__init__(val: 'plotly.Figure | matplotlib.artist.Artist')
```

Initialize a Plotly object.

**Args:**

* `val`:  Matplotlib or Plotly figure.

***


# Table
Source: https://docs.wandb.ai/models/ref/python/data-types/table



<GitHubLink />

## <kbd>class</kbd> `Table`

The Table class used to display and analyze tabular data.

Unlike traditional spreadsheets, Tables support numerous types of data: scalar values, strings, numpy arrays, and most subclasses of `wandb.data_types.Media`. This means you can embed `Images`, `Video`, `Audio`, and other sorts of rich, annotated media directly in Tables, alongside other traditional scalar values.

This class is the primary class used to generate W\&B Tables [https://docs.wandb.ai/models/tables](https://docs.wandb.ai/models/tables)

### <kbd>method</kbd> `Table.__init__`

```python theme={null}
__init__(
    columns=None,
    data=None,
    rows=None,
    dataframe=None,
    dtype=None,
    optional=True,
    allow_mixed_types=False,
    log_mode: "Literal['IMMUTABLE', 'MUTABLE', 'INCREMENTAL'] | None" = 'IMMUTABLE'
)
```

Initializes a Table object.

The rows is available for legacy reasons and should not be used. The Table class uses data to mimic the Pandas API.

**Args:**

* `columns`:  (List\[str]) Names of the columns in the table.  Defaults to \["Input", "Output", "Expected"].
* `data`:  (List\[List\[any]]) 2D row-oriented array of values.
* `dataframe`:  (pandas.DataFrame) DataFrame object used to create the table.  When set, `data` and `columns` arguments are ignored.
* `rows`:  (List\[List\[any]]) 2D row-oriented array of values.
* `optional`:  (Union\[bool,List\[bool]]) Determines if `None` values are allowed. Default to True
  * If a singular bool value, then the optionality is enforced for all  columns specified at construction time
  * If a list of bool values, then the optionality is applied to each  column - should be the same length as `columns`  applies to all columns. A list of bool values applies to each respective column.
* `allow_mixed_types`:  (bool) Determines if columns are allowed to have mixed types  (disables type validation). Defaults to False
* `log_mode`:  Optional\[str] Controls how the Table is logged when mutations occur.  Options:
  * "IMMUTABLE" (default): Table can only be logged once; subsequent  logging attempts after the table has been mutated will be no-ops.
  * "MUTABLE": Table can be re-logged after mutations, creating  a new artifact version each time it's logged.
  * "INCREMENTAL": Table data is logged incrementally, with each log creating  a new artifact entry containing the new data since the last log.

***

### <kbd>method</kbd> `Table.add_column`

```python theme={null}
add_column(name, data, optional=False)
```

Adds a column of data to the table.

**Args:**

* `name`:  (str) - the unique name of the column
* `data`:  (list | np.array) - a column of homogeneous data
* `optional`:  (bool) - if null-like values are permitted

***

### <kbd>method</kbd> `Table.add_computed_columns`

```python theme={null}
add_computed_columns(fn)
```

Adds one or more computed columns based on existing data.

**Args:**

* `fn`:  A function which accepts one or two parameters, ndx (int) and  row (dict), which is expected to return a dict representing  new columns for that row, keyed by the new column names.
  * `ndx` is an integer representing the index of the row. Only included if `include_ndx`  is set to `True`.
  * `row` is a dictionary keyed by existing columns

***

### <kbd>method</kbd> `Table.add_data`

```python theme={null}
add_data(*data)
```

Adds a new row of data to the table.

The maximum amount ofrows in a table is determined by `wandb.Table.MAX_ARTIFACT_ROWS`.

The length of the data should match the length of the table column.

***

### <kbd>method</kbd> `Table.add_row`

```python theme={null}
add_row(*row)
```

Deprecated. Use `Table.add_data` method instead.

***

### <kbd>method</kbd> `Table.cast`

```python theme={null}
cast(col_name, dtype, optional=False)
```

Casts a column to a specific data type.

This can be one of the normal python classes, an internal W\&B type, or an example object, like an instance of wandb.Image or wandb.Classes.

**Args:**

* `col_name` (str):  The name of the column to cast.
* `dtype` (class, wandb.wandb\_sdk.interface.\_dtypes.Type, any):  The  target dtype.
* `optional` (bool):  If the column should allow Nones.

***

### <kbd>method</kbd> `Table.get_column`

```python theme={null}
get_column(name, convert_to=None)
```

Retrieves a column from the table and optionally converts it to a NumPy object.

**Args:**

* `name`:  (str) - the name of the column
* `convert_to`:  (str, optional)
  * "numpy": will convert the underlying data to numpy object

***

### <kbd>method</kbd> `Table.get_dataframe`

```python theme={null}
get_dataframe()
```

Returns a `pandas.DataFrame` of the table.

***

### <kbd>method</kbd> `Table.get_index`

```python theme={null}
get_index()
```

Returns an array of row indexes for use in other tables to create links.

***


# Video
Source: https://docs.wandb.ai/models/ref/python/data-types/video



<GitHubLink />

## <kbd>class</kbd> `Video`

A class for logging videos to W\&B.

### <kbd>method</kbd> `Video.__init__`

```python theme={null}
__init__(
    data_or_path: 'str | pathlib.Path | np.ndarray | TextIO | BytesIO',
    caption: 'str | None' = None,
    fps: 'int | None' = None,
    format: "Literal['gif', 'mp4', 'webm', 'ogg'] | None" = None
)
```

Initialize a W\&B Video object.

**Args:**

* `data_or_path`:  Video can be initialized with a path to a file or an io object.  Video can be initialized with a numpy tensor. The numpy tensor  must be either 4 dimensional or 5 dimensional.  The dimensions should be (number of frames, channel, height, width) or  (batch, number of frames, channel, height, width)  The format parameter must be specified with the format argument  when initializing with a numpy array  or io object.
* `caption`:  Caption associated with the video for display.
* `fps`:  The frame rate to use when encoding raw video frames.  Default value is 4.  This parameter has no effect when data\_or\_path is a string, or bytes.
* `format`:  Format of video, necessary if initializing with a numpy array  or io object. This parameter will be used to determine the format  to use when encoding the video data. Accepted values are "gif",  "mp4", "webm", or "ogg".  If no value is provided, the default format will be "gif".

**Examples:**
Log a numpy array as a video

```python theme={null}
import numpy as np
import wandb

with wandb.init() as run:
    # axes are (number of frames, channel, height, width)
    frames = np.random.randint(
         low=0, high=256, size=(10, 3, 100, 100), dtype=np.uint8
    )
    run.log({"video": wandb.Video(frames, format="mp4", fps=4)})
```

***


# Experiments overview
Source: https://docs.wandb.ai/models/ref/python/experiments

Use foundational classes in the W&B Python SDK for tracking experiments and managing artifacts

These classes comprise the core building blocks for tracking machine learning experiments, managing artifacts, and configuring SDK behavior. These foundational classes enable you to log metrics, store model checkpoints, version datasets, and manage experiment configurations with full reproducibility and collaboration features.

> For more details on using these classes in ML experiments, consult the [Experiments](/models/ref/python/experiments/) and [Artifacts](/models/artifacts/) docs.

## Core classes

| Class                                                  | Description                                                                                                                    |
| ------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------ |
| [`Run`](/models/ref/python/experiments/run/)           | The primary unit of computation logged by W\&B, representing a single ML experiment with metrics, configurations, and outputs. |
| [`Artifact`](/models/ref/python/experiments/artifact/) | Flexible and lightweight building block for dataset and model versioning with automatic deduplication and lineage tracking.    |
| [`Settings`](/models/ref/python/experiments/settings/) | Configuration management for the W\&B SDK, controlling behavior from logging to API interactions.                              |

## Getting started

### Track an experiment

Create and track a machine learning experiment with metrics logging:

```python theme={null}
import wandb

# Initialize a new run
with wandb.init(project="my-experiments", config={"learning_rate": 0.001}) as run:
    # Access configuration
    config = run.config
    
    # Log metrics during training
    for epoch in range(10):
        metrics = train_one_epoch()  # Your training logic
        run.log({
            "loss": metrics["loss"],
            "accuracy": metrics["accuracy"],
            "epoch": epoch
        })
    
    # Log summary metrics
    run.summary["best_accuracy"] = max_accuracy
```

### Version a model artifact

Create and log a versioned model artifact with metadata:

```python theme={null}
import wandb

with wandb.init(project="my-models") as run:
    # Train your model
    model = train_model()
    
    # Create an artifact for the model
    model_artifact = wandb.Artifact(
        name="my-model",
        type="model",
        description="ResNet-50 trained on ImageNet subset",
        metadata={
            "architecture": "ResNet-50",
            "dataset": "ImageNet-1K",
            "accuracy": 0.95
        }
    )
    
    # Add model files to the artifact
    model_artifact.add_file("model.pt")
    model_artifact.add_dir("model_configs/")
    
    # Log the artifact to W&B
    run.log_artifact(model_artifact)
```

### Configure SDK settings

Customize W\&B SDK behavior for your specific requirements:

```python theme={null}
import wandb

# Configure settings programmatically
wandb.Settings(
    project="production-runs",
    entity="my-team",
    mode="offline",  # Run offline, sync later
    save_code=True,   # Save source code
    quiet=True        # Reduce console output
)

# Or use environment variables
# export WANDB_PROJECT=production-runs
# export WANDB_MODE=offline

# Initialize with custom settings
with wandb.init() as run:
    # Your experiment code here
    pass
```

### Link artifacts for lineage tracking

Track relationships between datasets, models, and evaluations:

```python theme={null}
import wandb

with wandb.init(project="ml-pipeline") as run:
    # Use a dataset artifact
    dataset = run.use_artifact("dataset:v1")
    dataset_dir = dataset.download()
    
    # Train model using the dataset
    model = train_on_dataset(dataset_dir)
    
    # Create model artifact with dataset lineage
    model_artifact = wandb.Artifact(
        name="trained-model",
        type="model"
    )
    model_artifact.add_file("model.pt")
    
    # Log with automatic lineage tracking
    run.log_artifact(model_artifact)
```


# Artifact
Source: https://docs.wandb.ai/models/ref/python/experiments/artifact



<GitHubLink />

## <kbd>class</kbd> `Artifact`

Flexible and lightweight building block for dataset and model versioning.

Construct an empty W\&B Artifact. Populate an artifacts contents with methods that begin with `add`. Once the artifact has all the desired files, you can call `run.log_artifact()` to log it.

### <kbd>method</kbd> `Artifact.__init__`

```python theme={null}
__init__(
    name: 'str',
    type: 'str',
    description: 'str | None' = None,
    metadata: 'dict[str, Any] | None' = None,
    incremental: 'bool' = False,
    use_as: 'str | None' = None,
    storage_region: 'str | None' = None
) → None
```

**Args:**

* `name` (str):  A human-readable name for the artifact. Use the name to identify  a specific artifact in the W\&B App UI or programmatically. You can  interactively reference an artifact with the `use_artifact` Public API.  A name can contain letters, numbers, underscores, hyphens, and dots.  The name must be unique across a project.
* `type` (str):  The artifact's type. Use the type of an artifact to both organize  and differentiate artifacts. You can use any string that contains letters,  numbers, underscores, hyphens, and dots. Common types include `dataset` or  `model`. Include `model` within your type string if you want to link the  artifact to the W\&B Model Registry.  Note that some types reserved for internal use and cannot be set by users.  Such types include `job` and types that start with `wandb-`.
* `description (str | None) = None`:  A description of the artifact. For Model or  Dataset Artifacts, add documentation for your standardized team model or  dataset card. View an artifact's description programmatically with the  `Artifact.description` attribute or programmatically with the W\&B App UI.  W\&B renders the description as markdown in the W\&B App.
* `metadata (dict[str, Any] | None) = None`:  Additional information about an artifact.  Specify metadata as a dictionary of key-value pairs. You can specify no more  than 100 total keys.
* `incremental`:  Use `Artifact.new_draft()` method instead to modify an  existing artifact.
* `use_as`:  Deprecated.

**Returns:**
An `Artifact` object.

***

### <kbd>property</kbd> Artifact.aliases

List of one or more semantically-friendly references or

identifying "nicknames" assigned to an artifact version.

Aliases are mutable references that you can programmatically reference. Change an artifact's alias with the W\&B App UI or programmatically. See [Create new artifact versions](https://docs.wandb.ai/models/artifacts/create-a-new-artifact-version) for more information.

**Returns:**

* `list[str]`: The aliases property value.

***

### <kbd>property</kbd> Artifact.collection

The collection this artifact is retrieved from.

A collection is an ordered group of artifact versions. If this artifact is retrieved from a collection that it is linked to, return that collection. Otherwise, return the collection that the artifact version originates from.

The collection that an artifact originates from is known as the source sequence.

**Returns:**

* `ArtifactCollection`: The collection property value.

***

### <kbd>property</kbd> Artifact.commit\_hash

The hash returned when this artifact was committed.

**Returns:**

* `str`: The commit\_hash property value.

***

### <kbd>property</kbd> Artifact.created\_at

Timestamp when the artifact was created.

**Returns:**

* `str`: The created\_at property value.

***

### <kbd>property</kbd> Artifact.description

A description of the artifact.

**Returns:**

* `str | None`: The description property value.

***

### <kbd>property</kbd> Artifact.digest

The logical digest of the artifact.

The digest is the checksum of the artifact's contents. If an artifact has the same digest as the current `latest` version, then `log_artifact` is a no-op.

**Returns:**

* `str`: The digest property value.

***

### <kbd>property</kbd> Artifact.entity

The name of the entity that the artifact collection belongs to.

If the artifact is a link, the entity will be the entity of the linked artifact.

**Returns:**

* `str`: The entity property value.

***

### <kbd>property</kbd> Artifact.file\_count

The number of files (including references).

**Returns:**

* `int`: The file\_count property value.

***

### <kbd>property</kbd> Artifact.history\_step

The nearest step which logged history metrics for this artifact's source run.

**Examples:**

```python theme={null}
run = artifact.logged_by()
if run and (artifact.history_step is not None):
    history = run.sample_history(
        min_step=artifact.history_step,
        max_step=artifact.history_step + 1,
        keys=["my_metric"],
    )
```

**Returns:**

* `int | None`: The history\_step property value.

***

### <kbd>property</kbd> Artifact.id

The artifact's ID.

**Returns:**

* `str | None`: The id property value.

***

### <kbd>property</kbd> Artifact.is\_link

Boolean flag indicating if the artifact is a link artifact.

True: The artifact is a link artifact to a source artifact. False: The artifact is a source artifact.

**Returns:**

* `bool`: The is\_link property value.

***

### <kbd>property</kbd> Artifact.linked\_artifacts

Returns a list of all the linked artifacts of a source artifact.

If this artifact is a link artifact (`artifact.is_link == True`), it will return an empty list.

Limited to 500 results.

**Returns:**

* `list[Artifact]`: The linked\_artifacts property value.

***

### <kbd>property</kbd> Artifact.manifest

The artifact's manifest.

The manifest lists all of its contents, and can't be changed once the artifact has been logged.

**Returns:**

* `ArtifactManifest`: The manifest property value.

***

### <kbd>property</kbd> Artifact.metadata

User-defined artifact metadata.

Structured data associated with the artifact.

**Returns:**

* `dict`: The metadata property value.

***

### <kbd>property</kbd> Artifact.name

The artifact name and version of the artifact.

A string with the format `{collection}:{alias}`. If fetched before an artifact is logged/saved, the name won't contain the alias. If the artifact is a link, the name will be the name of the linked artifact.

**Returns:**

* `str`: The name property value.

***

### <kbd>property</kbd> Artifact.project

The name of the project that the artifact collection belongs to.

If the artifact is a link, the project will be the project of the linked artifact.

**Returns:**

* `str`: The project property value.

***

### <kbd>property</kbd> Artifact.qualified\_name

The entity/project/name of the artifact.

If the artifact is a link, the qualified name will be the qualified name of the linked artifact path.

**Returns:**

* `str`: The qualified\_name property value.

***

### <kbd>property</kbd> Artifact.size

The total size of the artifact in bytes.

Includes any references tracked by this artifact.

**Returns:**

* `int`: The size property value.

***

### <kbd>property</kbd> Artifact.source\_artifact

Returns the source artifact, which is the original logged artifact.

If this artifact is a source artifact (`artifact.is_link == False`), it will return itself.

**Returns:**

* `Artifact`: The source\_artifact property value.

***

### <kbd>property</kbd> Artifact.source\_collection

The artifact's source collection.

The source collection is the collection that the artifact was logged from.

**Returns:**

* `ArtifactCollection`: The source\_collection property value.

***

### <kbd>property</kbd> Artifact.source\_entity

The name of the entity of the source artifact.

**Returns:**

* `str`: The source\_entity property value.

***

### <kbd>property</kbd> Artifact.source\_name

The artifact name and version of the source artifact.

A string with the format `{source_collection}:{alias}`. Before the artifact is saved, contains only the name since the version is not yet known.

**Returns:**

* `str`: The source\_name property value.

***

### <kbd>property</kbd> Artifact.source\_project

The name of the project of the source artifact.

**Returns:**

* `str`: The source\_project property value.

***

### <kbd>property</kbd> Artifact.source\_qualified\_name

The source\_entity/source\_project/source\_name of the source artifact.

**Returns:**

* `str`: The source\_qualified\_name property value.

***

### <kbd>property</kbd> Artifact.source\_version

The source artifact's version.

A string with the format `v{number}`.

**Returns:**

* `str`: The source\_version property value.

***

### <kbd>property</kbd> Artifact.state

The status of the artifact. One of: "PENDING", "COMMITTED", or "DELETED".

**Returns:**

* `str`: The state property value.

***

### <kbd>property</kbd> Artifact.tags

List of one or more tags assigned to this artifact version.

**Returns:**

* `list[str]`: The tags property value.

***

### <kbd>property</kbd> Artifact.ttl

The time-to-live (TTL) policy of an artifact.

Artifacts are deleted shortly after a TTL policy's duration passes. If set to `None`, the artifact deactivates TTL policies and will be not scheduled for deletion, even if there is a team default TTL. An artifact inherits a TTL policy from the team default if the team administrator defines a default TTL and there is no custom policy set on an artifact.

**Raises:**

* `ArtifactNotLoggedError`:  Unable to fetch inherited TTL if the artifact has not been logged or saved.

**Returns:**

* `timedelta | None`: The ttl property value.

***

### <kbd>property</kbd> Artifact.type

The artifact's type. Common types include `dataset` or `model`.

**Returns:**

* `str`: The type property value.

***

### <kbd>property</kbd> Artifact.updated\_at

The time when the artifact was last updated.

**Returns:**

* `str`: The updated\_at property value.

***

### <kbd>property</kbd> Artifact.url

Constructs the URL of the artifact.

**Returns:**

* `str`:  The URL of the artifact.

**Returns:**

* `str`: The url property value.

***

### <kbd>property</kbd> Artifact.use\_as

Deprecated.

**Returns:**

* `str | None`: The use\_as property value.

***

### <kbd>property</kbd> Artifact.version

The artifact's version.

A string with the format `v{number}`. If this is a link artifact, the version will be from the linked collection.

**Returns:**

* `str`: The version property value.

***

### <kbd>method</kbd> `Artifact.add`

```python theme={null}
add(
    obj: 'WBValue',
    name: 'StrPath',
    overwrite: 'bool' = False
) → ArtifactManifestEntry
```

Add wandb.WBValue `obj` to the artifact.

**Args:**

* `obj`:  The object to add. Currently support one of Bokeh, JoinedTable,  PartitionedTable, Table, Classes, ImageMask, BoundingBoxes2D,  Audio, Image, Video, Html, Object3D
* `name`:  The path within the artifact to add the object.
* `overwrite`:  If True, overwrite existing objects with the same file  path if applicable.

**Returns:**
The added manifest entry

**Raises:**

* `ArtifactFinalizedError`:  You cannot make changes to the current artifact version because it is finalized. Log a new artifact version instead.

***

### <kbd>method</kbd> `Artifact.add_dir`

```python theme={null}
add_dir(
    local_path: 'str',
    name: 'str | None' = None,
    skip_cache: 'bool | None' = False,
    policy: "Literal['mutable', 'immutable'] | None" = 'mutable',
    merge: 'bool' = False
) → None
```

Add a local directory to the artifact.

**Args:**

* `local_path`:  The path of the local directory.
* `name`:  The subdirectory name within an artifact. The name you  specify appears in the W\&B App UI nested by artifact's `type`.  Defaults to the root of the artifact.
* `skip_cache`:  If set to `True`, W\&B will not copy/move files to  the cache while uploading
* `policy`:  By default, "mutable".
  * mutable: Create a temporary copy of the file to prevent  corruption during upload.
  * immutable: Disable protection, rely on the user not to delete  or change the file.
* `merge`:  If `False` (default), throws ValueError if a file was already added  in a previous add\_dir call and its content has changed. If `True`,  overwrites existing files with changed content. Always adds new files  and never removes files. To replace an entire directory, pass a name  when adding the directory using `add_dir(local_path, name=my_prefix)`  and call `remove(my_prefix)` to remove the directory, then add it again.

**Raises:**

* `ArtifactFinalizedError`:  You cannot make changes to the current  artifact version because it is finalized. Log a new artifact  version instead.
* `ValueError`:  Policy must be "mutable" or "immutable"

***

### <kbd>method</kbd> `Artifact.add_file`

```python theme={null}
add_file(
    local_path: 'str',
    name: 'str | None' = None,
    is_tmp: 'bool | None' = False,
    skip_cache: 'bool | None' = False,
    policy: "Literal['mutable', 'immutable'] | None" = 'mutable',
    overwrite: 'bool' = False
) → ArtifactManifestEntry
```

Add a local file to the artifact.

**Args:**

* `local_path`:  The path to the file being added.
* `name`:  The path within the artifact to use for the file being added.  Defaults to the basename of the file.
* `is_tmp`:  If true, then the file is renamed deterministically to avoid  collisions.
* `skip_cache`:  If `True`, do not copy files to the cache  after uploading.
* `policy`:  By default, set to "mutable". If set to "mutable",  create a temporary copy of the file to prevent corruption  during upload. If set to "immutable", disable  protection and rely on the user not to delete or change the  file.
* `overwrite`:  If `True`, overwrite the file if it already exists.

**Returns:**
The added manifest entry.

**Raises:**

* `ArtifactFinalizedError`:  You cannot make changes to the current  artifact version because it is finalized. Log a new artifact  version instead.
* `ValueError`:  Policy must be "mutable" or "immutable"

***

### <kbd>method</kbd> `Artifact.add_reference`

```python theme={null}
add_reference(
    uri: 'ArtifactManifestEntry | str',
    name: 'StrPath | None' = None,
    checksum: 'bool' = True,
    max_objects: 'int | None' = None
) → Sequence[ArtifactManifestEntry]
```

Add a reference denoted by a URI to the artifact.

Unlike files or directories that you add to an artifact, references are not uploaded to W\&B. For more information, see [Track external files](https://docs.wandb.ai/models/artifacts/track-external-files).

By default, the following schemes are supported:

* http(s): The size and digest of the file will be inferred by the  `Content-Length` and the `ETag` response headers returned by the server.
* s3: The checksum and size are pulled from the object metadata.  If bucket versioning is enabled, then the version ID is also tracked.
* gs: The checksum and size are pulled from the object metadata. If bucket  versioning is enabled, then the version ID is also tracked.
* https, domain matching `*.blob.core.windows.net`
* Azure: The checksum and size are be pulled from the blob metadata.  If storage account versioning is enabled, then the version ID is  also tracked.
* file: The checksum and size are pulled from the file system. This scheme  is useful if you have an NFS share or other externally mounted volume  containing files you wish to track but not necessarily upload.

For any other scheme, the digest is just a hash of the URI and the size is left blank.

**Args:**

* `uri`:  The URI path of the reference to add. The URI path can be an object  returned from `Artifact.get_entry` to store a reference to another  artifact's entry.
* `name`:  The path within the artifact to place the contents of this reference.
* `checksum`:  Whether or not to checksum the resource(s) located at the  reference URI. Checksumming is strongly recommended as it enables  automatic integrity validation. Disabling checksumming will speed up  artifact creation but reference directories will not iterated through so  the objects in the directory will not be saved to the artifact.  We recommend setting `checksum=False` when adding reference objects,  in which case a new version will only be created if the reference URI  changes.
* `max_objects`:  The maximum number of objects to consider when adding a  reference that points to directory or bucket store prefix.  By default, the maximum number of objects allowed for Amazon S3,  GCS, Azure, and local files is 10,000,000. Other URI schemas  do not have a maximum.

**Returns:**
The added manifest entries.

**Raises:**

* `ArtifactFinalizedError`:  You cannot make changes to the current artifact version because it is finalized. Log a new artifact version instead.

***

### <kbd>method</kbd> `Artifact.checkout`

```python theme={null}
checkout(root: 'str | None' = None) → str
```

Replace the specified root directory with the contents of the artifact.

WARNING: This will delete all files in `root` that are not included in the artifact.

**Args:**

* `root`:  The directory to replace with this artifact's files.

**Returns:**
The path of the checked out contents.

**Raises:**

* `ArtifactNotLoggedError`:  If the artifact is not logged.

***

### <kbd>method</kbd> `Artifact.delete`

```python theme={null}
delete(delete_aliases: 'bool' = False) → None
```

Delete an artifact and its files.

If called on a linked artifact, only the link is deleted, and the source artifact is unaffected.

Use `Artifact.unlink()` instead of `Artifact.delete()` to remove a link between a source artifact and a collection.

**Args:**

* `delete_aliases`:  If set to `True`, delete all aliases associated  with the artifact. If `False`, raise an exception if  the artifact has existing aliases. This parameter is ignored  if the artifact is retrieved from a collection it is linked to.

**Raises:**

* `ArtifactNotLoggedError`:  If the artifact is not logged.

***

### <kbd>method</kbd> `Artifact.download`

```python theme={null}
download(
    root: 'StrPath | None' = None,
    allow_missing_references: 'bool' = False,
    skip_cache: 'bool | None' = None,
    path_prefix: 'StrPath | None' = None,
    multipart: 'bool | None' = None
) → FilePathStr
```

Download the contents of the artifact to the specified root directory.

Existing files located within `root` are not modified. Explicitly delete `root` before you call `download` if you want the contents of `root` to exactly match the artifact.

**Args:**

* `root`:  The directory W\&B stores the artifact's files.
* `allow_missing_references`:  If set to `True`, any invalid reference paths  will be ignored while downloading referenced files.
* `skip_cache`:  If set to `True`, the artifact cache will be skipped when  downloading and W\&B will download each file into the default root or  specified download directory.
* `path_prefix`:  If specified, only files with a path that starts with the given  prefix will be downloaded. Uses unix format (forward slashes).
* `multipart`:  If set to `None` (default), the artifact will be downloaded  in parallel using multipart download if individual file size is greater  than 2GB. If set to `True` or `False`, the artifact will be downloaded in  parallel or serially regardless of the file size.

**Returns:**
The path to the downloaded contents.

**Raises:**

* `ArtifactNotLoggedError`:  If the artifact is not logged.

***

### <kbd>method</kbd> `Artifact.file`

```python theme={null}
file(root: 'str | None' = None) → StrPath
```

Download a single file artifact to the directory you specify with `root`.

**Args:**

* `root`:  The root directory to store the file. Defaults to  `./artifacts/self.name/`.

**Returns:**
The full path of the downloaded file.

**Raises:**

* `ArtifactNotLoggedError`:  If the artifact is not logged.
* `ValueError`:  If the artifact contains more than one file.

***

### <kbd>method</kbd> `Artifact.files`

```python theme={null}
files(
    names: 'list[str] | None' = None,
    per_page: 'int' = 50,
    start: 'str | None' = None
) → ArtifactFiles
```

Iterate over all files stored in this artifact.

**Args:**

* `names`:  The filename paths relative to the root of the artifact you wish to  list.
* `per_page`:  The number of files to return per request.
* `start`:  Pagination cursor for resuming a past query, captured  from a previous paginator's `.cursor` attribute.

**Returns:**
An iterator containing `File` objects.

**Raises:**

* `ArtifactNotLoggedError`:  If the artifact is not logged.

***

### <kbd>method</kbd> `Artifact.finalize`

```python theme={null}
finalize() → None
```

Finalize the artifact version.

You cannot modify an artifact version once it is finalized because the artifact is logged as a specific artifact version. Create a new artifact version to log more data to an artifact. An artifact is automatically finalized when you log the artifact with `log_artifact`.

***

### <kbd>method</kbd> `Artifact.get`

```python theme={null}
get(name: 'str') → WBValue | None
```

Get the WBValue object located at the artifact relative `name`.

**Args:**

* `name`:  The artifact relative name to retrieve.

**Returns:**
W\&B object that can be logged with `run.log()` and visualized in the W\&B UI.

**Raises:**

* `ArtifactNotLoggedError`:  if the artifact isn't logged or the  run is offline.

***

### <kbd>method</kbd> `Artifact.get_added_local_path_name`

```python theme={null}
get_added_local_path_name(local_path: 'str') → str | None
```

Get the artifact relative name of a file added by a local filesystem path.

**Args:**

* `local_path`:  The local path to resolve into an artifact relative name.

**Returns:**
The artifact relative name.

***

### <kbd>method</kbd> `Artifact.get_entry`

```python theme={null}
get_entry(name: 'StrPath') → ArtifactManifestEntry
```

Get the entry with the given name.

**Args:**

* `name`:  The artifact relative name to get

**Returns:**
A `W&B` object.

**Raises:**

* `ArtifactNotLoggedError`:  if the artifact isn't logged or the run is offline.
* `KeyError`:  if the artifact doesn't contain an entry with the given name.

***

### <kbd>method</kbd> `Artifact.get_path`

```python theme={null}
get_path(name: 'StrPath') → ArtifactManifestEntry
```

Deprecated. Use `get_entry(name)`.

***

### <kbd>method</kbd> `Artifact.is_draft`

```python theme={null}
is_draft() → bool
```

Check if artifact is not saved.

**Returns:**
Boolean. `False` if artifact is saved. `True` if artifact is not saved.

***

### <kbd>method</kbd> `Artifact.json_encode`

```python theme={null}
json_encode() → dict[str, Any]
```

Returns the artifact encoded to the JSON format.

**Returns:**
A `dict` with `string` keys representing attributes of the artifact.

***

### <kbd>method</kbd> `Artifact.link`

```python theme={null}
link(target_path: 'str', aliases: 'Iterable[str] | None' = None) → Artifact
```

Link this artifact to a collection.

**Args:**

* `target_path`:  The path of the collection. Path consists of the prefix  "wandb-registry-" along with the registry name and the  collection name `wandb-registry-{REGISTRY_NAME}/{COLLECTION_NAME}`.
* `aliases`:  Add one or more aliases to the linked artifact. The  "latest" alias is automatically applied to the most recent artifact  you link.

**Raises:**

* `ArtifactNotLoggedError`:  If the artifact is not logged.

**Returns:**
The linked artifact.

***

### <kbd>method</kbd> `Artifact.logged_by`

```python theme={null}
logged_by() → Run | None
```

Get the W\&B run that originally logged the artifact.

**Returns:**
The name of the W\&B run that originally logged the artifact.

**Raises:**

* `ArtifactNotLoggedError`:  If the artifact is not logged.

***

### <kbd>method</kbd> `Artifact.new_draft`

```python theme={null}
new_draft() → Artifact
```

Create a new draft artifact with the same content as this committed artifact.

Modifying an existing artifact creates a new artifact version known as an "incremental artifact". The artifact returned can be extended or modified and logged as a new version.

**Returns:**
An `Artifact` object.

**Raises:**

* `ArtifactNotLoggedError`:  If the artifact is not logged.

***

### <kbd>method</kbd> `Artifact.new_file`

```python theme={null}
new_file(
    name: 'str',
    mode: 'str' = 'x',
    encoding: 'str | None' = None
) → Generator[IO]
```

Open a new temporary file and add it to the artifact.

**Args:**

* `name`:  The name of the new file to add to the artifact.
* `mode`:  The file access mode to use to open the new file.
* `encoding`:  The encoding used to open the new file.

**Returns:**
A new file object that can be written to. Upon closing, the file is automatically added to the artifact.

**Raises:**

* `ArtifactFinalizedError`:  You cannot make changes to the current artifact version because it is finalized. Log a new artifact version instead.

***

### <kbd>method</kbd> `Artifact.remove`

```python theme={null}
remove(item: 'StrPath | ArtifactManifestEntry') → None
```

Remove an item from the artifact.

**Args:**

* `item`:  The item to remove. Can be a specific manifest entry  or the name of an artifact-relative path. If the item  matches a directory all items in that directory will be removed.

**Raises:**

* `ArtifactFinalizedError`:  You cannot make changes to the current  artifact version because it is finalized. Log a new artifact  version instead.
* `FileNotFoundError`:  If the item isn't found in the artifact.

***

### <kbd>method</kbd> `Artifact.save`

```python theme={null}
save(
    project: 'str | None' = None,
    settings: 'wandb.Settings | None' = None
) → None
```

Persist any changes made to the artifact.

If currently in a run, that run will log this artifact. If not currently in a run, a run of type "auto" is created to track this artifact.

**Args:**

* `project`:  A project to use for the artifact in the case that a run is not  already in context.
* `settings`:  A settings object to use when initializing an automatic run. Most  commonly used in testing harness.

***

### <kbd>method</kbd> `Artifact.unlink`

```python theme={null}
unlink() → None
```

Unlink this artifact if it is a linked member of an artifact collection.

**Raises:**

* `ArtifactNotLoggedError`:  If the artifact is not logged.
* `ValueError`:  If the artifact is not linked to any collection.

***

### <kbd>method</kbd> `Artifact.used_by`

```python theme={null}
used_by() → list[Run]
```

Get a list of the runs that have used this artifact and its linked artifacts.

**Returns:**
A list of `Run` objects.

**Raises:**

* `ArtifactNotLoggedError`:  If the artifact is not logged.

***

### <kbd>method</kbd> `Artifact.verify`

```python theme={null}
verify(root: 'str | None' = None) → None
```

Verify that the contents of an artifact match the manifest.

All files in the directory are checksummed and the checksums are then cross-referenced against the artifact's manifest. References are not verified.

**Args:**

* `root`:  The directory to verify. If None artifact will be downloaded to  './artifacts/self.name/'.

**Raises:**

* `ArtifactNotLoggedError`:  If the artifact is not logged.
* `ValueError`:  If the verification fails.

***

### <kbd>method</kbd> `Artifact.wait`

```python theme={null}
wait(timeout: 'int | None' = None) → Artifact
```

If needed, wait for this artifact to finish logging.

**Args:**

* `timeout`:  The time, in seconds, to wait.

**Returns:**
An `Artifact` object.


# Run
Source: https://docs.wandb.ai/models/ref/python/experiments/run



<GitHubLink />

## <kbd>class</kbd> `Run`

A unit of computation logged by W\&B. Typically, this is an ML experiment.

Call [`wandb.init()`](https://docs.wandb.ai/models/ref/python/functions/init) to create a new run. `wandb.init()` starts a new run and returns a `wandb.Run` object. Each run is associated with a unique ID (run ID). W\&B recommends using a context (`with` statement) manager to automatically finish the run.

For distributed training experiments, you can either track each process separately using one run per process or track all processes to a single run. See [Log distributed training experiments](https://docs.wandb.ai/models/track/log/distributed-training) for more information.

You can log data to a run with `wandb.Run.log()`. Anything you log using `wandb.Run.log()` is sent to that run. See [Create an experiment](https://docs.wandb.ai/models/track/create-an-experiment) or [`wandb.init`](https://docs.wandb.ai/models/ref/python/functions/init) API reference page or more information.

There is a another `Run` object in the [`wandb.apis.public`](https://docs.wandb.ai/models/ref/python/public-api/api) namespace. Use this object is to interact with runs that have already been created.

**Attributes:**

* `summary`:  (Summary) A summary of the run, which is a dictionary-like  object. For more information, see
* `[Log summary metrics](https`: //docs.wandb.ai/models/track/log/log-summary).

**Examples:**
Create a run with `wandb.init()`:

```python theme={null}
import wandb

# Start a new run and log some data
# Use context manager (`with` statement) to automatically finish the run
with wandb.init(entity="entity", project="project") as run:
    run.log({"accuracy": acc, "loss": loss})
```

### <kbd>property</kbd> Run.config

Config object associated with this run.

**Returns:**

* `wandb_config.Config`: The config property value.

***

### <kbd>property</kbd> Run.config\_static

Static config object associated with this run.

**Returns:**

* `wandb_config.ConfigStatic`: The config\_static property value.

***

### <kbd>property</kbd> Run.dir

The directory where files associated with the run are saved.

**Returns:**

* `str`: The dir property value.

***

### <kbd>property</kbd> Run.disabled

True if the run is disabled, False otherwise.

**Returns:**

* `bool`: The disabled property value.

***

### <kbd>property</kbd> Run.entity

The name of the W\&B entity associated with the run.

Entity can be a username or the name of a team or organization.

**Returns:**

* `str`: The entity property value.

***

### <kbd>property</kbd> Run.group

Returns the name of the group associated with this run.

Grouping runs together allows related experiments to be organized and visualized collectively in the W\&B UI. This is especially useful for scenarios such as distributed training or cross-validation, where multiple runs should be viewed and managed as a unified experiment.

In shared mode, where all processes share the same run object, setting a group is usually unnecessary, since there is only one run and no grouping is required.

**Returns:**

* `str`: The group property value.

***

### <kbd>property</kbd> Run.id

Identifier for this run.

**Returns:**

* `str`: The id property value.

***

### <kbd>property</kbd> Run.job\_type

Name of the job type associated with the run.

View a run's job type in the run's Overview page in the W\&B App.

You can use this to categorize runs by their job type, such as "training", "evaluation", or "inference". This is useful for organizing and filtering runs in the W\&B UI, especially when you have multiple runs with different job types in the same project. For more information, see [Organize runs](https://docs.wandb.ai/models/runs#organize-runs).

**Returns:**

* `str`: The job\_type property value.

***

### <kbd>property</kbd> Run.name

Display name of the run.

Display names are not guaranteed to be unique and may be descriptive. By default, they are randomly generated.

**Returns:**

* `str | None`: The name property value.

***

### <kbd>property</kbd> Run.notes

Notes associated with the run, if there are any.

Notes can be a multiline string and can also use markdown and latex equations inside `$$`, like `$x + 3$`.

**Returns:**

* `str | None`: The notes property value.

***

### <kbd>property</kbd> Run.offline

True if the run is offline, False otherwise.

**Returns:**

* `bool`: The offline property value.

***

### <kbd>property</kbd> Run.path

Path to the run.

Run paths include entity, project, and run ID, in the format `entity/project/run_id`.

**Returns:**

* `str`: The path property value.

***

### <kbd>property</kbd> Run.project

Name of the W\&B project associated with the run.

**Returns:**

* `str`: The project property value.

***

### <kbd>property</kbd> Run.project\_url

URL of the W\&B project associated with the run, if there is one.

Offline runs do not have a project URL.

**Returns:**

* `str | None`: The project\_url property value.

***

### <kbd>property</kbd> Run.resumed

True if the run was resumed, False otherwise.

**Returns:**

* `bool`: The resumed property value.

***

### <kbd>property</kbd> Run.settings

A frozen copy of run's Settings object.

**Returns:**

* `Settings`: The settings property value.

***

### <kbd>property</kbd> Run.start\_time

Unix timestamp (in seconds) of when the run started.

**Returns:**

* `float`: The start\_time property value.

***

### <kbd>property</kbd> Run.sweep\_id

Identifier for the sweep associated with the run, if there is one.

**Returns:**

* `str | None`: The sweep\_id property value.

***

### <kbd>property</kbd> Run.sweep\_url

URL of the sweep associated with the run, if there is one.

Offline runs do not have a sweep URL.

**Returns:**

* `str | None`: The sweep\_url property value.

***

### <kbd>property</kbd> Run.tags

Tags associated with the run, if there are any.

**Returns:**

* `tuple | None`: The tags property value.

***

### <kbd>property</kbd> Run.url

The url for the W\&B run, if there is one.

Offline runs will not have a url.

**Returns:**

* `str | None`: The url property value.

***

### <kbd>method</kbd> `Run.alert`

```python theme={null}
alert(
    title: 'str',
    text: 'str',
    level: 'str | AlertLevel | None' = None,
    wait_duration: 'int | float | timedelta | None' = None
) → None
```

Create an alert with the given title and text.

**Args:**

* `title`:  The title of the alert, must be less than 64 characters long.
* `text`:  The text body of the alert.
* `level`:  The alert level to use, either: `INFO`, `WARN`, or `ERROR`.
* `wait_duration`:  The time to wait (in seconds) before sending another  alert with this title.

***

### <kbd>method</kbd> `Run.define_metric`

```python theme={null}
define_metric(
    name: 'str',
    step_metric: 'str | wandb_metric.Metric | None' = None,
    step_sync: 'bool | None' = None,
    hidden: 'bool | None' = None,
    summary: 'str | None' = None,
    goal: 'str | None' = None,
    overwrite: 'bool | None' = None
) → wandb_metric.Metric
```

Customize metrics logged with `wandb.Run.log()`.

**Args:**

* `name`:  The name of the metric to customize.
* `step_metric`:  The name of another metric to serve as the X-axis  for this metric in automatically generated charts.
* `step_sync`:  Automatically insert the last value of step\_metric into  `wandb.Run.log()` if it is not provided explicitly. Defaults to True  if step\_metric is specified.
* `hidden`:  Hide this metric from automatic plots.
* `summary`:  Specify aggregate metrics added to summary.  Supported aggregations include "min", "max", "mean", "last",  "first", "best", "copy" and "none". "none" prevents a summary  from being generated. "best" is used together with the goal  parameter, "best" is deprecated and should not be used, use  "min" or "max" instead. "copy" is deprecated and should not be  used.
* `goal`:  Specify how to interpret the "best" summary type.  Supported options are "minimize" and "maximize". "goal" is  deprecated and should not be used, use "min" or "max" instead.
* `overwrite`:  If false, then this call is merged with previous  `define_metric` calls for the same metric by using their  values for any unspecified parameters. If true, then  unspecified parameters overwrite values specified by  previous calls.

**Returns:**
An object that represents this call but can otherwise be discarded.

***

### <kbd>method</kbd> `Run.display`

```python theme={null}
display(height: 'int' = 420, hidden: 'bool' = False) → bool
```

Display this run in Jupyter.

***

### <kbd>method</kbd> `Run.finish`

```python theme={null}
finish(exit_code: 'int | None' = None, quiet: 'bool | None' = None) → None
```

Finish a run and upload any remaining data.

Marks the completion of a W\&B run and ensures all data is synced to the server. The run's final state is determined by its exit conditions and sync status.

Run States:

* Running: Active run that is logging data and/or sending heartbeats.
* Crashed: Run that stopped sending heartbeats unexpectedly.
* Finished: Run completed successfully (`exit_code=0`) with all data synced.
* Failed: Run completed with errors (`exit_code!=0`).
* Killed: Run was forcibly stopped before it could finish.

**Args:**

* `exit_code`:  Integer indicating the run's exit status. Use 0 for success,  any other value marks the run as failed.
* `quiet`:  Deprecated. Configure logging verbosity using `wandb.Settings(quiet=...)`.

***

### <kbd>method</kbd> `Run.finish_artifact`

```python theme={null}
finish_artifact(
    artifact_or_path: 'Artifact | str',
    name: 'str | None' = None,
    type: 'str | None' = None,
    aliases: 'list[str] | None' = None,
    distributed_id: 'str | None' = None
) → Artifact
```

Finishes a non-finalized artifact as output of a run.

Subsequent "upserts" with the same distributed ID will result in a new version.

**Args:**

* `artifact_or_path`:  A path to the contents of this artifact,  can be in the following forms:
  * `/local/directory`
  * `/local/directory/file.txt`
  * `s3://bucket/path`  You can also pass an Artifact object created by calling  `wandb.Artifact`.
* `name`:  An artifact name. May be prefixed with entity/project.  Valid names can be in the following forms:
  * name:version
  * name:alias
  * digest  This will default to the basename of the path prepended with the current  run id  if not specified.
* `type`:  The type of artifact to log, examples include `dataset`, `model`
* `aliases`:  Aliases to apply to this artifact,  defaults to `["latest"]`
* `distributed_id`:  Unique string that all distributed jobs share. If None,  defaults to the run's group name.

**Returns:**
An `Artifact` object.

***

### <kbd>method</kbd> `Run.link_artifact`

```python theme={null}
link_artifact(
    artifact: 'Artifact',
    target_path: 'str',
    aliases: 'list[str] | None' = None
) → Artifact
```

Link the artifact to a collection.

The term “link” refers to pointers that connect where W\&B stores the artifact and where the artifact is accessible in the registry. W\&B does not duplicate artifacts when you link an artifact to a collection.

View linked artifacts in the Registry UI for the specified collection.

**Args:**

* `artifact`:  The artifact object to link to the collection.
* `target_path`:  The path of the collection. Path consists of the prefix  "wandb-registry-" along with the registry name and the  collection name `wandb-registry-{REGISTRY_NAME}/{COLLECTION_NAME}`.
* `aliases`:  Add one or more aliases to the linked artifact. The  "latest" alias is automatically applied to the most recent artifact  you link.

**Returns:**
The linked artifact.

***

### <kbd>method</kbd> `Run.link_model`

```python theme={null}
link_model(
    path: 'StrPath',
    registered_model_name: 'str',
    name: 'str | None' = None,
    aliases: 'list[str] | None' = None
) → Artifact | None
```

Log a model artifact version and link it to a registered model in the model registry.

Linked model versions are visible in the UI for the specified registered model.

This method will:

* Check if 'name' model artifact has been logged. If so, use the artifact version that matches the files located at 'path' or log a new version. Otherwise log files under 'path' as a new model artifact, 'name' of type 'model'.
* Check if registered model with name 'registered\_model\_name' exists in the 'model-registry' project. If not, create a new registered model with name 'registered\_model\_name'.
* Link version of model artifact 'name' to registered model, 'registered\_model\_name'.
* Attach aliases from 'aliases' list to the newly linked model artifact version.

**Args:**

* `path`:  (str) A path to the contents of this model, can be in the  following forms:
  * `/local/directory`
  * `/local/directory/file.txt`
  * `s3://bucket/path`
* `registered_model_name`:  The name of the registered model that the  model is to be linked to. A registered model is a collection of  model versions linked to the model registry, typically  representing a team's specific ML Task. The entity that this  registered model belongs to will be derived from the run.
* `name`:  The name of the model artifact that files in 'path' will be  logged to. This will default to the basename of the path  prepended with the current run id  if not specified.
* `aliases`:  Aliases that will only be applied on this linked artifact  inside the registered model. The alias "latest" will always be  applied to the latest version of an artifact that is linked.

**Raises:**

* `AssertionError`:  If registered\_model\_name is a path or  if model artifact 'name' is of a type that does not contain  the substring 'model'.
* `ValueError`:  If name has invalid special characters.

**Returns:**
The linked artifact if linking was successful, otherwise `None`.

***

### <kbd>method</kbd> `Run.log`

```python theme={null}
log(
    data: 'dict[str, Any]',
    step: 'int | None' = None,
    commit: 'bool | None' = None
) → None
```

Upload run data.

Use `log` to log data from runs, such as scalars, images, video, histograms, plots, and tables. See [Log objects and media](https://docs.wandb.ai/models/track/log) for code snippets, best practices, and more.

Basic usage:

```python theme={null}
import wandb

with wandb.init() as run:
     run.log({"train-loss": 0.5, "accuracy": 0.9})
```

The previous code snippet saves the loss and accuracy to the run's history and updates the summary values for these metrics.

Visualize logged data in a workspace at [wandb.ai](https://wandb.ai), or locally on a [self-hosted instance](https://docs.wandb.ai/platform/hosting) of the W\&B app, or export data to visualize and explore locally, such as in a Jupyter notebook, with the [Public API](https://docs.wandb.ai/models/track/public-api-guide).

Logged values don't have to be scalars. You can log any [W\&B supported Data Type](https://docs.wandb.ai/models/ref/python/data-types) such as images, audio, video, and more. For example, you can use `wandb.Table` to log structured data. See [Log tables, visualize and query data](https://docs.wandb.ai/models/tables/tables-walkthrough) tutorial for more details.

W\&B organizes metrics with a forward slash (`/`) in their name into sections named using the text before the final slash. For example, the following results in two sections named "train" and "validate":

```python theme={null}
with wandb.init() as run:
     # Log metrics in the "train" section.
     run.log(
         {
             "train/accuracy": 0.9,
             "train/loss": 30,
             "validate/accuracy": 0.8,
             "validate/loss": 20,
         }
     )
```

Only one level of nesting is supported; `run.log({"a/b/c": 1})` produces a section named "a".

`run.log()` is not intended to be called more than a few times per second. For optimal performance, limit your logging to once every N iterations, or collect data over multiple iterations and log it in a single step.

By default, each call to `log` creates a new "step". The step must always increase, and it is not possible to log to a previous step. You can use any metric as the X axis in charts. See [Custom log axes](https://docs.wandb.ai/models/track/log/customize-logging-axes) for more details.

In many cases, it is better to treat the W\&B step like you'd treat a timestamp rather than a training step.

```python theme={null}
with wandb.init() as run:
     # Example: log an "epoch" metric for use as an X axis.
     run.log({"epoch": 40, "train-loss": 0.5})
```

It is possible to use multiple `wandb.Run.log()` invocations to log to the same step with the `step` and `commit` parameters. The following are all equivalent:

```python theme={null}
with wandb.init() as run:
     # Normal usage:
     run.log({"train-loss": 0.5, "accuracy": 0.8})
     run.log({"train-loss": 0.4, "accuracy": 0.9})

     # Implicit step without auto-incrementing:
     run.log({"train-loss": 0.5}, commit=False)
     run.log({"accuracy": 0.8})
     run.log({"train-loss": 0.4}, commit=False)
     run.log({"accuracy": 0.9})

     # Explicit step:
     run.log({"train-loss": 0.5}, step=current_step)
     run.log({"accuracy": 0.8}, step=current_step)
     current_step += 1
     run.log({"train-loss": 0.4}, step=current_step)
     run.log({"accuracy": 0.9}, step=current_step, commit=True)
```

**Args:**

* `data`:  A `dict` with `str` keys and values that are serializable
* `Python objects including`:  `int`, `float` and `string`; any of the `wandb.data_types`; lists, tuples and NumPy arrays of serializable Python objects; other `dict`s of this structure.
* `step`:  The step number to log. If `None`, then an implicit  auto-incrementing step is used. See the notes in  the description.
* `commit`:  If true, finalize and upload the step. If false, then  accumulate data for the step. See the notes in the description.  If `step` is `None`, then the default is `commit=True`;  otherwise, the default is `commit=False`.

**Examples:**
For more and more detailed examples, see [our guides to logging](https://docs.wandb.ai/models/track/log).

Basic usage

```python theme={null}
import wandb

with wandb.init() as run:
    run.log({"train-loss": 0.5, "accuracy": 0.9
```

Incremental logging

```python theme={null}
import wandb

with wandb.init() as run:
    run.log({"loss": 0.2}, commit=False)
    # Somewhere else when I'm ready to report this step:
    run.log({"accuracy": 0.8})
```

Histogram

```python theme={null}
import numpy as np
import wandb

# sample gradients at random from normal distribution
gradients = np.random.randn(100, 100)
with wandb.init() as run:
    run.log({"gradients": wandb.Histogram(gradients)})
```

Image from NumPy

```python theme={null}
import numpy as np
import wandb

with wandb.init() as run:
    examples = []
    for i in range(3):
         pixels = np.random.randint(low=0, high=256, size=(100, 100, 3))
         image = wandb.Image(pixels, caption=f"random field {i}")
         examples.append(image)
    run.log({"examples": examples})
```

Image from PIL

```python theme={null}
import numpy as np
from PIL import Image as PILImage
import wandb

with wandb.init() as run:
    examples = []
    for i in range(3):
         pixels = np.random.randint(
             low=0,
             high=256,
             size=(100, 100, 3),
             dtype=np.uint8,
         )
         pil_image = PILImage.fromarray(pixels, mode="RGB")
         image = wandb.Image(pil_image, caption=f"random field {i}")
         examples.append(image)
    run.log({"examples": examples})
```

Video from NumPy

```python theme={null}
import numpy as np
import wandb

with wandb.init() as run:
    # axes are (time, channel, height, width)
    frames = np.random.randint(
         low=0,
         high=256,
         size=(10, 3, 100, 100),
         dtype=np.uint8,
    )
    run.log({"video": wandb.Video(frames, fps=4)})
```

Matplotlib plot

```python theme={null}
from matplotlib import pyplot as plt
import numpy as np
import wandb

with wandb.init() as run:
    fig, ax = plt.subplots()
    x = np.linspace(0, 10)
    y = x * x
    ax.plot(x, y)  # plot y = x^2
    run.log({"chart": fig})
```

PR Curve

```python theme={null}
import wandb

with wandb.init() as run:
    run.log({"pr": wandb.plot.pr_curve(y_test, y_probas, labels)})
```

3D Object

```python theme={null}
import wandb

with wandb.init() as run:
    run.log(
         {
             "generated_samples": [
                 wandb.Object3D(open("sample.obj")),
                 wandb.Object3D(open("sample.gltf")),
                 wandb.Object3D(open("sample.glb")),
             ]
         }
    )
```

**Raises:**

* `wandb.Error`:  If called before `wandb.init()`.
* `ValueError`:  If invalid data is passed.

***

### <kbd>method</kbd> `Run.log_artifact`

```python theme={null}
log_artifact(
    artifact_or_path: 'Artifact | StrPath',
    name: 'str | None' = None,
    type: 'str | None' = None,
    aliases: 'list[str] | None' = None,
    tags: 'list[str] | None' = None
) → Artifact
```

Declare an artifact as an output of a run.

**Args:**

* `artifact_or_path`:  (str or Artifact) A path to the contents of this artifact,  can be in the following forms:
  * `/local/directory`
  * `/local/directory/file.txt`
  * `s3://bucket/path`  You can also pass an Artifact object created by calling  `wandb.Artifact`.
* `name`:  (str, optional) An artifact name. Valid names can be in the following forms:
  * name:version
  * name:alias
  * digest  This will default to the basename of the path prepended with the current  run id  if not specified.
* `type`:  (str) The type of artifact to log, examples include `dataset`, `model`
* `aliases`:  (list, optional) Aliases to apply to this artifact,  defaults to `["latest"]`
* `tags`:  (list, optional) Tags to apply to this artifact, if any.

**Returns:**
An `Artifact` object.

***

### <kbd>method</kbd> `Run.log_code`

```python theme={null}
log_code(
    root: 'str | None' = '.',
    name: 'str | None' = None,
    include_fn: 'Callable[[str, str], bool] | Callable[[str], bool]' = <function _is_py_requirements_or_dockerfile at 0x106b82d40>,
    exclude_fn: 'Callable[[str, str], bool] | Callable[[str], bool]' = <function exclude_wandb_fn at 0x10814af20>
) → Artifact | None
```

Save the current state of your code to a W\&B Artifact.

By default, it walks the current directory and logs all files that end with `.py`.

**Args:**

* `root`:  The relative (to `os.getcwd()`) or absolute path to recursively find code from.
* `name`:  (str, optional) The name of our code artifact. By default, we'll name  the artifact `source-$PROJECT_ID-$ENTRYPOINT_RELPATH`. There may be scenarios where you want  many runs to share the same artifact. Specifying name allows you to achieve that.
* `include_fn`:  A callable that accepts a file path and (optionally) root path and  returns True when it should be included and False otherwise. This
* `defaults to `lambda path, root`:  path.endswith(".py")`.
* `exclude_fn`:  A callable that accepts a file path and (optionally) root path and  returns `True` when it should be excluded and `False` otherwise. This  defaults to a function that excludes all files within `<root>/.wandb/`  and `<root>/wandb/` directories.

**Examples:**
Basic usage

```python theme={null}
import wandb

with wandb.init() as run:
    run.log_code()
```

Advanced usage

```python theme={null}
import wandb

with wandb.init() as run:
    run.log_code(
         root="../",
         include_fn=lambda path: path.endswith(".py") or path.endswith(".ipynb"),
         exclude_fn=lambda path, root: os.path.relpath(path, root).startswith(
             "cache/"
         ),
    )
```

**Returns:**
An `Artifact` object if code was logged

***

### <kbd>method</kbd> `Run.log_model`

```python theme={null}
log_model(
    path: 'StrPath',
    name: 'str | None' = None,
    aliases: 'list[str] | None' = None
) → None
```

Logs a model artifact containing the contents inside the 'path' to a run and marks it as an output to this run.

The name of model artifact can only contain alphanumeric characters, underscores, and hyphens.

**Args:**

* `path`:  (str) A path to the contents of this model,  can be in the following forms:
  * `/local/directory`
  * `/local/directory/file.txt`
  * `s3://bucket/path`
* `name`:  A name to assign to the model artifact that  the file contents will be added to. This will default to the  basename of the path prepended with the current run id if  not specified.
* `aliases`:  Aliases to apply to the created model artifact,  defaults to `["latest"]`

**Raises:**

* `ValueError`:  If name has invalid special characters.

**Returns:**
None

***

### <kbd>method</kbd> `Run.mark_preempting`

```python theme={null}
mark_preempting() → None
```

Mark this run as preempting.

Also tells the internal process to immediately report this to server.

***

### <kbd>method</kbd> `Run.pin_config_keys`

```python theme={null}
pin_config_keys(keys: 'Sequence[str]' = ()) → None
```

Pin config keys to display in the References section on Run Overview.

Pinned keys appear prominently above Notes on the Run Overview page. String values are rendered as markdown; non-strings are rendered as plain text. Calling this again replaces the previously pinned list.

**Args:**

* `keys`:  Config key names to pin, matching keys set via  `run.config`. These are exact key strings (dots and  slashes are treated literally, not as path separators).  Order is preserved and determines display order.

***

### <kbd>method</kbd> `Run.restore`

```python theme={null}
restore(
    name: 'str',
    run_path: 'str | None' = None,
    replace: 'bool' = False,
    root: 'str | None' = None
) → None | TextIO
```

Download the specified file from cloud storage.

File is placed into the current directory or run directory. By default, will only download the file if it doesn't already exist.

**Args:**

* `name`:  The name of the file.
* `run_path`:  Optional path to a run to pull files from, i.e. `username/project_name/run_id`  if wandb.init has not been called, this is required.
* `replace`:  Whether to download the file even if it already exists locally
* `root`:  The directory to download the file to.  Defaults to the current  directory or the run directory if wandb.init was called.

**Returns:**
None if it can't find the file, otherwise a file object open for reading.

**Raises:**

* `CommError`:  If W\&B can't connect to the W\&B backend.
* `ValueError`:  If the file is not found or can't find run\_path.

***

### <kbd>method</kbd> `Run.save`

```python theme={null}
save(
    glob_str: 'str | os.PathLike',
    base_path: 'str | os.PathLike | None' = None,
    policy: 'PolicyName' = 'live'
) → bool | list[str]
```

Sync one or more files to W\&B.

Relative paths are relative to the current working directory.

A Unix glob, such as "myfiles/\*", is expanded at the time `save` is called regardless of the `policy`. In particular, new files are not picked up automatically.

A `base_path` may be provided to control the directory structure of uploaded files. It should be a prefix of `glob_str`, and the directory structure beneath it is preserved.

When given an absolute path or glob and no `base_path`, one directory level is preserved as in the example above.

Files are automatically deduplicated: calling `save()` multiple times on the same file without modifications will not re-upload it.

**Args:**

* `glob_str`:  A relative or absolute path or Unix glob.
* `base_path`:  A path to use to infer a directory structure; see examples.
* `policy`:  One of `live`, `now`, or `end`.
  * live: upload the file as it changes, overwriting the previous version
  * now: upload the file once now
  * end: upload file when the run ends

**Returns:**
Paths to the symlinks created for the matched files.

For historical reasons, this may return a boolean in legacy code.

```python theme={null}
import wandb

run = wandb.init()

run.save("these/are/myfiles/*")
# => Saves files in a "these/are/myfiles/" folder in the run.

run.save("these/are/myfiles/*", base_path="these")
# => Saves files in an "are/myfiles/" folder in the run.

run.save("/Users/username/Documents/run123/*.txt")
# => Saves files in a "run123/" folder in the run. See note below.

run.save("/Users/username/Documents/run123/*.txt", base_path="/Users")
# => Saves files in a "username/Documents/run123/" folder in the run.

run.save("files/*/saveme.txt")
# => Saves each "saveme.txt" file in an appropriate subdirectory
#    of "files/".

# Explicitly finish the run since a context manager is not used.
run.finish()
```

***

### <kbd>method</kbd> `Run.status`

```python theme={null}
status() → RunStatus
```

Get sync info from the internal backend, about the current run's sync status.

***

### <kbd>method</kbd> `Run.unwatch`

```python theme={null}
unwatch(
    models: 'torch.nn.Module | Sequence[torch.nn.Module] | None' = None
) → None
```

Remove pytorch model topology, gradient and parameter hooks.

**Args:**

* `models`:  Optional list of pytorch models that have had watch called on them.

***

### <kbd>method</kbd> `Run.upsert_artifact`

```python theme={null}
upsert_artifact(
    artifact_or_path: 'Artifact | str',
    name: 'str | None' = None,
    type: 'str | None' = None,
    aliases: 'list[str] | None' = None,
    distributed_id: 'str | None' = None
) → Artifact
```

Declare (or append to) a non-finalized artifact as output of a run.

Note that you must call run.finish\_artifact() to finalize the artifact. This is useful when distributed jobs need to all contribute to the same artifact.

**Args:**

* `artifact_or_path`:  A path to the contents of this artifact,  can be in the following forms:
  * `/local/directory`
  * `/local/directory/file.txt`
  * `s3://bucket/path`
* `name`:  An artifact name. May be prefixed with "entity/project". Defaults  to the basename of the path prepended with the current run ID  if not specified. Valid names can be in the following forms:
  * name:version
  * name:alias
  * digest
* `type`:  The type of artifact to log. Common examples include `dataset`, `model`.
* `aliases`:  Aliases to apply to this artifact, defaults to `["latest"]`.
* `distributed_id`:  Unique string that all distributed jobs share. If None,  defaults to the run's group name.

**Returns:**
An `Artifact` object.

***

### <kbd>method</kbd> `Run.use_artifact`

```python theme={null}
use_artifact(
    artifact_or_name: 'str | Artifact',
    type: 'str | None' = None,
    aliases: 'list[str] | None' = None,
    use_as: 'str | None' = None
) → Artifact
```

Declare an artifact as an input to a run.

Call `download` or `file` on the returned object to get the contents locally.

**Args:**

* `artifact_or_name`:  The name of the artifact to use. May be prefixed  with the name of the project the artifact was logged to  ("entity" or "entity/project"). If no  entity is specified in the name, the Run or API setting's entity is used.  Valid names can be in the following forms
  * name:version
  * name:alias
* `type`:  The type of artifact to use.
* `aliases`:  Aliases to apply to this artifact
* `use_as`:  This argument is deprecated and does nothing.

**Returns:**
An `Artifact` object.

**Examples:**

```python theme={null}
import wandb

run = wandb.init(project="<example>")

# Use an artifact by name and alias
artifact_a = run.use_artifact(artifact_or_name="<name>:<alias>")

# Use an artifact by name and version
artifact_b = run.use_artifact(artifact_or_name="<name>:v<version>")

# Use an artifact by entity/project/name:alias
artifact_c = run.use_artifact(
   artifact_or_name="<entity>/<project>/<name>:<alias>"
)

# Use an artifact by entity/project/name:version
artifact_d = run.use_artifact(
   artifact_or_name="<entity>/<project>/<name>:v<version>"
)

# Explicitly finish the run since a context manager is not used.
run.finish()
```

***

### <kbd>method</kbd> `Run.use_model`

```python theme={null}
use_model(name: 'str') → FilePathStr
```

Download the files logged in a model artifact 'name'.

**Args:**

* `name`:  A model artifact name. 'name' must match the name of an existing logged  model artifact. May be prefixed with `entity/project/`. Valid names  can be in the following forms
  * model\_artifact\_name:version
  * model\_artifact\_name:alias

**Returns:**

* `path` (str):  Path to downloaded model artifact file(s).

**Raises:**

* `AssertionError`:  If model artifact 'name' is of a type that does  not contain the substring 'model'.

***

### <kbd>method</kbd> `Run.watch`

```python theme={null}
watch(
    models: 'torch.nn.Module | Sequence[torch.nn.Module]',
    criterion: 'torch.F | None' = None,
    log: "Literal['gradients', 'parameters', 'all'] | None" = 'gradients',
    log_freq: 'int' = 1000,
    idx: 'int | None' = None,
    log_graph: 'bool' = False
) → None
```

Hook into given PyTorch model to monitor gradients and the model's computational graph.

This function can track parameters, gradients, or both during training.

**Args:**

* `models`:  A single model or a sequence of models to be monitored.
* `criterion`:  The loss function being optimized (optional).
* `log`:  Specifies whether to log "gradients", "parameters", or "all".  Set to None to disable logging. (default="gradients").
* `log_freq`:  Frequency (in batches) to log gradients and parameters. (default=1000)
* `idx`:  Index used when tracking multiple models with `wandb.watch`. (default=None)
* `log_graph`:  Whether to log the model's computational graph. (default=False)

**Raises:**
ValueError:  If `wandb.init()` has not been called or if any of the models are not instances  of `torch.nn.Module`.

***

### <kbd>method</kbd> `Run.write_logs`

```python theme={null}
write_logs(text: 'str') → None
```

Write text to the run's Logs tab.

Use `write_logs` to directly write text to the Logs tab instead of relying on automatic stdout/stderr capture. Calls after the run has finished are silently ignored.

Consider using the `capture_loggers` setting which integrates with Python's `logging` module.

**Args:**

* `text`:  The text to write. A trailing newline is added if not present.


# Settings
Source: https://docs.wandb.ai/models/ref/python/experiments/settings



<GitHubLink />

## <kbd>class</kbd> `Settings`

Settings for the W\&B SDK.

This class manages configuration settings for the W\&B SDK,
ensuring type safety and validation of all settings. Settings are accessible
as attributes and can be initialized programmatically, through environment
variables (`WANDB_ prefix`), and with configuration files.

The settings are organized into three categories:

1. Public settings: Core configuration options that users can safely modify to customize
   W\&B's behavior for their specific needs.
2. Internal settings: Settings prefixed with 'x\_' that handle low-level SDK behavior.
   These settings are primarily for internal use and debugging. While they can be modified,
   they are not considered part of the public API and may change without notice in future
   versions.
3. Computed settings: Read-only settings that are automatically derived from other settings or
   the environment.

### <kbd>method</kbd> `Settings.__init__`

```python theme={null}
__init__(
    allow_media_symlink: 'bool' = False,
    allow_offline_artifacts: 'bool' = True,
    allow_val_change: 'bool' = False,
    anonymous: 'object' = <object object at 0x104fe52d0>,
    api_key: 'str | None' = None,
    azure_account_url_to_access_key: 'dict[str, str] | None' = None,
    app_url_override: 'str | None' = None,
    base_url: 'str' = 'https://api.wandb.ai',
    code_dir: 'str | None' = None,
    config_paths: 'collections.abc.Sequence[str] | None' = None,
    console: 'Literal['auto', 'off', 'wrap', 'redirect', 'wrap_raw', 'wrap_emu']' = 'auto',
    console_multipart: 'bool' = False,
    console_chunk_max_bytes: 'int' = 0,
    console_chunk_max_seconds: 'int' = 0,
    capture_loggers: 'dict[str, str] | None' = None,
    credentials_file: 'str' = None,
    disable_code: 'bool' = False,
    disable_git: 'bool' = False,
    disable_git_fork_point: 'bool' = True,
    disable_job_creation: 'bool' = True,
    docker: 'str | None' = None,
    email: 'str | None' = None,
    entity: 'str | None' = None,
    organization: 'str | None' = None,
    force: 'bool' = False,
    fork_from: 'wandb.sdk.lib.run_moment.RunMoment | None' = None,
    git_commit: 'str | None' = None,
    git_remote: 'str' = 'origin',
    git_remote_url: 'str | None' = None,
    git_root: 'str | None' = None,
    heartbeat_seconds: 'int' = 30,
    host: 'str | None' = None,
    http_proxy: 'str | None' = None,
    https_proxy: 'str | None' = None,
    identity_token_file: 'str | None' = None,
    ignore_globs: 'Sequence' = (),
    init_timeout: 'float' = 90.0,
    finish_timeout: 'float' = 0.0,
    finish_timeout_raises: 'bool' = False,
    insecure_disable_ssl: 'bool' = False,
    job_name: 'str | None' = None,
    job_source: 'Literal['repo', 'artifact', 'image'] | None' = None,
    label_disable: 'bool' = False,
    launch: 'bool' = False,
    launch_config_path: 'str | None' = None,
    login_timeout: 'float | None' = None,
    mode: 'Literal['online', 'offline', 'shared', 'disabled', 'dryrun', 'run']' = 'online',
    notebook_name: 'str | None' = None,
    program: 'str | None' = None,
    program_abspath: 'str | None' = None,
    program_relpath: 'str | None' = None,
    project: 'str | None' = None,
    quiet: 'bool' = False,
    reinit: 'Literal['default', 'return_previous', 'finish_previous', 'create_new'] | bool' = 'default',
    relogin: 'bool' = False,
    resume: 'Literal['allow', 'must', 'never', 'auto'] | None' = None,
    resume_from: 'wandb.sdk.lib.run_moment.RunMoment | None' = None,
    resumed: 'bool' = False,
    root_dir: 'str' = None,
    run_group: 'str | None' = None,
    run_id: 'str | None' = None,
    run_job_type: 'str | None' = None,
    run_name: 'str | None' = None,
    run_notes: 'str | None' = None,
    run_tags: 'tuple[str, ...] | None' = None,
    sagemaker_disable: 'bool' = False,
    save_code: 'bool | None' = None,
    settings_system: 'str | None' = None,
    stop_fn: 'collections.abc.Callable[[], None] | None' = None,
    max_end_of_run_history_metrics: 'int' = 10,
    max_end_of_run_summary_metrics: 'int' = 10,
    show_colors: 'bool | None' = None,
    show_emoji: 'bool | None' = None,
    show_errors: 'bool' = True,
    show_info: 'bool' = True,
    show_warnings: 'bool' = True,
    silent: 'bool' = False,
    start_method: 'str | None' = None,
    stop_on_fatal_error: 'bool' = False,
    strict: 'bool | None' = None,
    summary_timeout: 'int' = 60,
    summary_warnings: 'int' = 5,
    sweep_id: 'str | None' = None,
    sweep_param_path: 'str | None' = None,
    symlink: 'bool' = None,
    sync_tensorboard: 'bool | None' = None,
    table_raise_on_max_row_limit_exceeded: 'bool' = False,
    use_dot_wandb: 'bool | None' = None,
    username: 'str | None' = None,
    x_cli_only_mode: 'bool' = False,
    x_disable_meta: 'bool' = False,
    x_disable_stats: 'bool' = False,
    x_disable_viewer: 'bool' = False,
    x_disable_machine_info: 'bool' = False,
    x_executable: 'str | None' = None,
    x_extra_http_headers: 'dict[str, str] | None' = None,
    x_file_stream_max_bytes: 'int | None' = None,
    x_file_stream_max_line_bytes: 'int | None' = None,
    x_file_stream_transmit_interval: 'float | None' = None,
    x_file_stream_retry_max: 'int | None' = None,
    x_file_stream_retry_wait_min_seconds: 'float | None' = None,
    x_file_stream_retry_wait_max_seconds: 'float | None' = None,
    x_file_stream_timeout_seconds: 'float | None' = None,
    x_file_transfer_retry_max: 'int | None' = None,
    x_file_transfer_retry_wait_min_seconds: 'float | None' = None,
    x_file_transfer_retry_wait_max_seconds: 'float | None' = None,
    x_file_transfer_timeout_seconds: 'float | None' = None,
    x_files_dir: 'str | None' = None,
    x_flow_control_custom: 'bool | None' = None,
    x_flow_control_disabled: 'bool | None' = None,
    x_graphql_retry_max: 'int | None' = None,
    x_graphql_retry_wait_min_seconds: 'float | None' = None,
    x_graphql_retry_wait_max_seconds: 'float | None' = None,
    x_graphql_timeout_seconds: 'float | None' = None,
    x_internal_check_process: 'float' = 8.0,
    x_jupyter_name: 'str | None' = None,
    x_jupyter_path: 'str | None' = None,
    x_jupyter_root: 'str | None' = None,
    x_label: 'str | None' = None,
    x_live_policy_rate_limit: 'int | None' = None,
    x_live_policy_wait_time: 'int | None' = None,
    x_log_level: 'int' = 20,
    x_network_buffer: 'int | None' = None,
    x_primary: 'bool' = True,
    x_proxies: 'dict[str, str] | None' = None,
    x_runqueue_item_id: 'str | None' = None,
    x_save_requirements: 'bool' = True,
    x_server_side_derived_summary: 'bool' = False,
    x_server_side_expand_glob_metrics: 'bool' = True,
    x_service_transport: 'str | None' = None,
    x_service_wait: 'float' = 30.0,
    x_skip_transaction_log: 'bool' = False,
    x_start_time: 'float | None' = None,
    x_stats_pid: 'int' = 61958,
    x_stats_sampling_interval: 'float' = 15.0,
    x_stats_neuron_monitor_config_path: 'str | None' = None,
    x_stats_dcgm_exporter: 'str | None' = None,
    x_stats_open_metrics_endpoints: 'dict[str, str] | None' = None,
    x_stats_open_metrics_filters: 'dict[str, dict[str, str]] | collections.abc.Sequence[str] | None' = None,
    x_stats_open_metrics_http_headers: 'dict[str, str] | None' = None,
    x_stats_disk_paths: 'collections.abc.Sequence[str] | None' = ('/',),
    x_stats_cpu_count: 'int | None' = None,
    x_stats_cpu_logical_count: 'int | None' = None,
    x_stats_gpu_count: 'int | None' = None,
    x_stats_gpu_type: 'str | None' = None,
    x_stats_gpu_device_ids: 'collections.abc.Sequence[int] | None' = None,
    x_stats_buffer_size: 'int' = 0,
    x_stats_coreweave_metadata_base_url: 'str' = 'http://169.254.169.254',
    x_stats_coreweave_metadata_endpoint: 'str' = '/api/v2/cloud-init/meta-data',
    x_stats_track_process_tree: 'bool' = False,
    x_stats_no_cgroup: 'bool' = False,
    x_sync: 'bool' = False,
    x_sync_dir_suffix: 'str' = '',
    x_update_finish_state: 'bool' = True
) → None
```

**Args:**

* `allow_media_symlink` (bool): Whether to symlink media files to the run directory.
  If true, media files will be symlinked or hardlinked to the
  run directory instead of copied. This may result in faster
  logging and reduced disk usage. However, deleting or modifying
  the original files before upload to the W\&B server will be
  reflected in the uploaded data.

* `allow_offline_artifacts` (bool): Flag to allow table artifacts to be synced in offline mode.
  To revert to the old behavior, set this to False.

* `allow_val_change` (bool): Flag to allow modification of `Config` values after they've been set.

* `anonymous` (object): Deprecated and will be removed.

* `api_key` (str | None): The W\&B API key.

* `azure_account_url_to_access_key` (dict\[str, str] | None): Mapping of Azure account URLs to their corresponding access keys for Azure integration.

* `app_url_override` (str | None): Override for the 'app' URL for the W\&B UI.
  The `app_url` is normally computed based on `base_url`, but this can be
  used to set it explicitly.
  WANDB\_APP\_URL is the corresponding environment variable.

* `base_url` (str): The URL of the W\&B backend for data synchronization.

* `code_dir` (str | None): Directory containing the code to be tracked by W\&B.

* `config_paths` (collections.abc.Sequence\[str] | None): Paths to files to load configuration from into the `Config` object.

* `console` (Literal\['auto', 'off', 'wrap', 'redirect', 'wrap\_raw', 'wrap\_emu']): The type of console capture to be applied.
  Possible values are:
  * "auto" - Automatically selects the console capture method based on the
    system environment and settings.
  * "off" - Disables console capture.
  * "redirect" - Redirects low-level file descriptors for capturing output.
  * "wrap" - Overrides the write methods of sys.stdout/sys.stderr. Will be
    mapped to either "wrap\_raw" or "wrap\_emu" based on the state of the system.
  * "wrap\_raw" - Same as "wrap" but captures raw output directly instead of
    through an emulator. Derived from the `wrap` setting and should not be set manually.
  * "wrap\_emu" - Same as "wrap" but captures output through an emulator.
    Derived from the `wrap` setting and should not be set manually.

* `console_multipart` (bool): Enable multipart console logging.
  When True, the SDK writes console output to timestamped files
  under the `logs/` directory instead of a single `output.log`.
  Each part is uploaded as soon as it is closed, giving users live
  access to logs while the run is active. Rollover cadence is
  controlled by `console_chunk_max_bytes` and/or `console_chunk_max_seconds`.
  If both limits are `0`, all logs are uploaded once at run finish.
  Note: Uploaded chunks are immutable; terminal control sequences
  that modify previous lines (e.g., progress bars using carriage returns)
  only affect the current chunk.

* `console_chunk_max_bytes` (int): Size-based rollover threshold for multipart console logs, in bytes.
  Starts a new console log file when the current part reaches this
  size. Has an effect only when `console_multipart` is `True`.
  Can be combined with `console_chunk_max_seconds`; whichever limit is
  hit first triggers the rollover. A value of `0` disables the
  size-based limit.

* `console_chunk_max_seconds` (int): Time-based rollover threshold for multipart console logs, in seconds.
  Starts a new console log file after this many seconds have elapsed
  since the current part began. Requires `console_multipart` to be
  `True`.  May be used with `console_chunk_max_bytes`; the first limit
  reached closes the part. A value of `0` disables the time-based
  limit.

* `capture_loggers` (dict\[str, str] | None): Names of Python loggers to capture into the run's Logs tab.
  A mapping of logger name to minimum log level. When set, wandb installs a
  logging.Handler on each named logger and removes it when the run finishes.
  Log records emitted by those loggers are published as console output to the
  run, similar to stdout/stderr capture.
  Log records are formatted the same as `logging.basicConfig()`, like
  `INFO:my_module:Some message.` This is not currently customizable.
  To capture all logs, pass the name of the root logger, which is 'root'.
  This is independent of the `console` setting: both can be active
  simultaneously.
  Example:
  ```python theme={null}
  wandb.init(
  settings=wandb.Settings(
  console="off",
  capture_loggers={
  "my_app": "INFO",
  "my_app.training": "ERROR",
  },
  ),
  )
  ```

* `credentials_file` (str): Path to file for writing temporary access tokens.

* `disable_code` (bool): Whether to disable capturing the code.

* `disable_git` (bool): Whether to disable capturing the git state.

* `disable_git_fork_point` (bool): Whether to disable inferring fork point from remote branches
  When set to True, the SDK will use the latest commit from the upstream
  branch, if one is set. Otherwise skip generating the diff patch.
  When set to False, the SDK will try to use the latest commit from the upstream branch,
  if one is set.
  Otherwise, it will find the closest commit from all remote branches.
  This may impact performance for repos with many upstream branches.

* `disable_job_creation` (bool): Whether to disable the creation of a job artifact for W\&B Launch.

* `docker` (str | None): The Docker image used to execute the script.

* `email` (str | None): The email address of the user.

* `entity` (str | None): The W\&B entity, such as a user or a team.

* `organization` (str | None): The W\&B organization.

* `force` (bool): Whether to pass the `force` flag to `wandb.login()`.

* `fork_from` (wandb.sdk.lib.run\_moment.RunMoment | None): Specifies a point in a previous execution of a run to fork from.
  The point is defined by the run ID, a metric, and its value.
  Currently, only the metric '\_step' is supported.

* `git_commit` (str | None): The git commit hash to associate with the run.

* `git_remote` (str): The git remote to associate with the run.

* `git_remote_url` (str | None): The URL of the git remote repository.

* `git_root` (str | None): Root directory of the git repository.

* `host` (str | None): Hostname of the machine running the script.

* `http_proxy` (str | None): Custom proxy servers for http requests to W\&B.

* `https_proxy` (str | None): Custom proxy servers for https requests to W\&B.

* `identity_token_file` (str | None): Path to file containing an identity token (JWT) for authentication.

* `ignore_globs` (Sequence): Unix glob patterns relative to `files_dir` specifying files to exclude from upload.

* `init_timeout` (float): Time in seconds to wait for the `wandb.init` call to complete before timing out.

* `finish_timeout` (float): Time in seconds to wait for data to upload at the end of a run.
  Setting this can limit costs caused by slow uploads to W\&B at the end of a
  run, with the trade-off that the run will be marked crashed and may be
  missing some data. The default is for `run.finish()` to block until all
  data finishes uploading.
  If this is set to a number greater than zero, W\&B gives up on uploading a
  run's data after this many seconds at the end of a run, unblocking your
  script. After some time, the run becomes Crashed or Failed in the UI. Any
  unuploaded data is still stored on disk and can be uploaded with `wandb
  sync`.
  Use the `finish_timeout_raises` setting to raise an error in addition to
  printing a warning message.
  Runs shut down by `wandb.teardown()` (which automatically runs at the end
  of a script in an atexit hook) will also respect this setting.

* `finish_timeout_raises` (bool): Whether to raise a TimeoutError if finish\_timeout expires.
  Using this together with the `finish_timeout` setting causes `run.finish()`
  to raise a TimeoutError after a timeout in addition to printing a message.
  Note that `run.finish()` is called implicitly when using a Run as a context
  manager:
  with wandb.init() as run:
  ...  # run.finish() executes at the end of the `with` block
  This does not cause `wandb.teardown()` to raise an error (since it runs
  at the end of a script anyway).

* `insecure_disable_ssl` (bool): Whether to insecurely disable SSL verification.

* `job_name` (str | None): Name of the Launch job running the script.

* `job_source` (Optional\[Literal\['repo', 'artifact', 'image']]): Source type for Launch.

* `label_disable` (bool): Whether to disable automatic labeling features.

* `launch_config_path` (str | None): Path to the launch configuration file.

* `login_timeout` (float | None): Time in seconds to wait for login operations before timing out.

* `mode` (Literal\['online', 'offline', 'shared', 'disabled', 'dryrun', 'run']): The operating mode for W\&B logging and synchronization.

* `notebook_name` (str | None): Name of the notebook if running in a Jupyter-like environment.

* `program` (str | None): Path to the script that created the run, if available.

* `program_abspath` (str | None): The absolute path from the root repository directory to the script that
  created the run.
  Root repository directory is defined as the directory containing the
  .git directory, if it exists. Otherwise, it's the current working directory.

* `program_relpath` (str | None): The relative path to the script that created the run.

* `project` (str | None): The W\&B project ID.

* `quiet` (bool): Flag to suppress non-essential output.

* `reinit` (Union\[Literal\['default', 'return\_previous', 'finish\_previous', 'create\_new'], bool]): What to do when `wandb.init()` is called while a run is active.
  Options:
  * "default": Use "finish\_previous" in notebooks and "return\_previous"
    otherwise.
  * "return\_previous": Return the most recently created run
    that is not yet finished. This does not update `wandb.run`; see
    the "create\_new" option.
  * "finish\_previous": Finish all active runs, then return a new run.
  * "create\_new": Create a new run without modifying other active runs.
    Does not update `wandb.run` and top-level functions like `wandb.log`.
    Because of this, some older integrations that rely on the global run
    will not work.
    Can also be a boolean, but this is deprecated. False is the same as
    "return\_previous", and True is the same as "finish\_previous".

* `relogin` (bool): Flag to force a new login attempt.

* `resume` (Optional\[Literal\['allow', 'must', 'never', 'auto']]): Specifies the resume behavior for the run.
  Options:
  * "must": Resumes from an existing run with the same ID. If no such run exists,
    it will result in failure.
  * "allow": Attempts to resume from an existing run with the same ID. If none is
    found, a new run will be created.
  * "never": Always starts a new run. If a run with the same ID already exists,
    it will result in failure.
  * "auto": Automatically resumes from the most recent failed run on the same
    machine.

* `resume_from` (wandb.sdk.lib.run\_moment.RunMoment | None): Specifies a point in a previous execution of a run to resume from.
  The point is defined by the run ID, a metric, and its value.
  Currently, only the metric '\_step' is supported.

* `root_dir` (str): The root directory to use as the base for all run-related paths.
  In particular, this is used to derive the wandb directory and the run directory.

* `run_group` (str | None): Group identifier for related runs.
  Used for grouping runs in the UI.

* `run_id` (str | None): The ID of the run.

* `run_job_type` (str | None): Type of job being run (e.g., training, evaluation).

* `run_name` (str | None): Human-readable name for the run.

* `run_notes` (str | None): Additional notes or description for the run.

* `run_tags` (tuple\[str, ...] | None): Tags to associate with the run for organization and filtering.

* `sagemaker_disable` (bool): Flag to disable SageMaker-specific functionality.

* `save_code` (bool | None): Whether to save the code associated with the run.

* `settings_system` (str | None): Path to the system-wide settings file.

* `stop_fn` (collections.abc.Callable\[\[], None] | None): A callback to execute to stop the run.
  A run can be stopped through the web UI, or after a fatal error
  (if configured via a setting).
  By default, to stop a run, W\&B sends a SIGINT to the main thread.
  Set this callback to override this behavior, like to use a different
  signal or to take some other action before interrupting.
  The callback runs in a separate thread. It runs soon after a stop is
  requested, but not immediately.

* `max_end_of_run_history_metrics` (int): Maximum number of history sparklines to display at the end of a run.

* `max_end_of_run_summary_metrics` (int): Maximum number of summary metrics to display at the end of a run.

* `show_errors` (bool): Whether to display error messages.

* `show_info` (bool): Whether to display informational messages.

* `show_warnings` (bool): Whether to display warning messages.

* `silent` (bool): Flag to suppress all output.

* `stop_on_fatal_error` (bool): Whether to stop the run after a fatal error.
  After W\&B hits an unrecoverable error while uploading data, it prints
  a message and stops uploading, but still allows logging more data.
  This is usually desirable: your training metrics get stored on disk
  and can be recovered using `wandb sync`, even if they aren't uploaded.
  This is not useful if your files get deleted after training.
  In that case, setting this to True will stop the run after a fatal error,
  as if the stop button was pressed in the web UI.

* `strict` (bool | None): Whether to enable strict mode for validation and error checking.

* `summary_timeout` (int): Time in seconds to wait for summary operations before timing out.

* `sweep_id` (str | None): Identifier of the sweep this run belongs to.

* `sweep_param_path` (str | None): Path to the sweep parameters configuration.

* `symlink` (bool): Whether to use symlinks (True by default except on Windows).

* `sync_tensorboard` (bool | None): Whether to synchronize TensorBoard logs with W\&B.

* `table_raise_on_max_row_limit_exceeded` (bool): Whether to raise an exception when table row limits are exceeded.

* `use_dot_wandb` (bool | None): Whether to use a hidden `.wandb` or visible `wandb` directory for run data.
  If True, the SDK uses `.wandb`. If False, `wandb`.
  If not set, defaults to `.wandb` if it already exists, otherwise `wandb`.

* `username` (str | None): Username.

* `x_disable_meta` (bool): Flag to disable the collection of system metadata.

* `x_disable_stats` (bool): Flag to disable the collection of system metrics.

* `x_extra_http_headers` (dict\[str, str] | None): Additional headers to add to all outgoing HTTP requests.

* `x_label` (str | None): Label to assign to system metrics and console logs collected for the run.
  This is used to group data by on the frontend and can be used to distinguish data
  from different processes in a distributed training job.

* `x_primary` (bool): Determines whether to save internal wandb files and metadata.
  In a distributed setting, this is useful for avoiding file overwrites
  from secondary processes when only system metrics and logs are needed,
  as the primary process handles the main logging.

* `x_save_requirements` (bool): Flag to save the requirements file.

* `x_server_side_derived_summary` (bool): Flag to delegate automatic computation of summary from history to the server.
  This does not disable user-provided summary updates.

* `x_service_wait` (float): Time in seconds to wait for the wandb-core internal service to start.

* `x_skip_transaction_log` (bool): Whether to skip saving the run events to the transaction log.
  This is only relevant for online runs. Can be used to reduce the amount of
  data written to disk.
  Should be used with caution, as it removes the gurantees about
  recoverability.

* `x_stats_sampling_interval` (float): Sampling interval for the system monitor in seconds.

* `x_stats_dcgm_exporter` (str | None): Endpoint to extract Nvidia DCGM metrics from.
  Options:
  * Extract DCGM-related metrics from a query to the Prometheus `/api/v1/query` endpoint.
    It is a common practice to aggregate metrics reported by the instances of the DCGM Exporter
    running on different nodes in a cluster using Prometheus.
  * TODO: Parse metrics directly from the `/metrics` endpoint of the DCGM Exporter.
    Examples:
  * `http://localhost:9400/api/v1/query?query=DCGM_FI_DEV_GPU_TEMP{node="l1337", cluster="globular"}`.

* `x_stats_open_metrics_endpoints` (dict\[str, str] | None): OpenMetrics `/metrics` endpoints to monitor for system metrics.

* `x_stats_open_metrics_filters` (dict\[str, dict\[str, str]] | collections.abc.Sequence\[str] | None): Filter to apply to metrics collected from OpenMetrics `/metrics` endpoints.
  Supports two formats:
  * `{"metric regex pattern, including endpoint name as prefix": {"label": "label value regex pattern"}}`
  * `("metric regex pattern 1", "metric regex pattern 2", ...)`

* `x_stats_open_metrics_http_headers` (dict\[str, str] | None): HTTP headers to add to OpenMetrics requests.

* `x_stats_disk_paths` (collections.abc.Sequence\[str] | None): System paths to monitor for disk usage.

* `x_stats_cpu_count` (int | None): System CPU count.
  If set, overrides the auto-detected value in the run metadata.

* `x_stats_cpu_logical_count` (int | None): Logical CPU count.
  If set, overrides the auto-detected value in the run metadata.

* `x_stats_gpu_count` (int | None): GPU device count.
  If set, overrides the auto-detected value in the run metadata.

* `x_stats_gpu_type` (str | None): GPU device type.
  If set, overrides the auto-detected value in the run metadata.

* `x_stats_gpu_device_ids` (collections.abc.Sequence\[int] | None): GPU device indices to monitor.
  If not set, the system monitor captures metrics for all GPUs.
  Assumes 0-based indexing matching CUDA/ROCm device enumeration.

* `x_stats_track_process_tree` (bool): Monitor the entire process tree for resource usage, starting from `x_stats_pid`.
  When `True`, the system monitor aggregates the RSS, CPU%, and thread count
  from the process with PID `x_stats_pid` and all of its descendants.
  This can have a performance overhead and is disabled by default.

* `x_stats_no_cgroup` (bool): Disable cgroup v2 CPU and memory limits for system metric percentages.

* `x_sync_dir_suffix` (str): Suffix to add to the run's directory name (sync\_dir).
  This is set in wandb.init() to avoid naming conflicts.
  If set, it is joined to the default name with a dash.

* `x_update_finish_state` (bool): Flag to indicate whether this process can update the run's final state on the server.
  Set to False in distributed training when only the main process should determine the final state.

**Returns:**
An `Settings` object.

### <kbd>property</kbd> `Settings.app_url`

The URL for the W\&B UI, usually [https://wandb.ai](https://wandb.ai).

This is different from `base_url` (like [https://api.wandb.ai](https://api.wandb.ai)) which
is used to access W\&B APIs programmatically.

**Returns:**

* `str`: The app\_url property value.

### <kbd>property</kbd> `Settings.colab_url`

The URL to the Colab notebook, if running in Colab.

**Returns:**

* `str | None`: The colab\_url property value.

### <kbd>property</kbd> `Settings.deployment`

### <kbd>property</kbd> `Settings.files_dir`

Absolute path to the local directory where the run's files are stored.

**Returns:**

* `str`: The files\_dir property value.

### <kbd>property</kbd> `Settings.is_local`

### <kbd>property</kbd> `Settings.log_dir`

The directory for storing log files.

**Returns:**

* `str`: The log\_dir property value.

### <kbd>property</kbd> `Settings.log_internal`

The path to the file to use for internal logs.

**Returns:**

* `str`: The log\_internal property value.

### <kbd>property</kbd> `Settings.log_symlink_internal`

The path to the symlink to the internal log file of the most recent run.

**Returns:**

* `str`: The log\_symlink\_internal property value.

### <kbd>property</kbd> `Settings.log_symlink_user`

The path to the symlink to the user-process log file of the most recent run.

**Returns:**

* `str`: The log\_symlink\_user property value.

### <kbd>property</kbd> `Settings.log_user`

The path to the file to use for user-process logs.

**Returns:**

* `str`: The log\_user property value.

### <kbd>property</kbd> `Settings.project_url`

The W\&B URL where the project can be viewed.

**Returns:**

* `str`: The project\_url property value.

### <kbd>property</kbd> `Settings.resume_fname`

The path to the resume file.

**Returns:**

* `str`: The resume\_fname property value.

### <kbd>property</kbd> `Settings.run_mode`

The mode of the run. Can be either "run" or "offline-run".

**Returns:**

* `Literal['run', 'offline-run']`: The run\_mode property value.

### <kbd>property</kbd> `Settings.run_url`

The W\&B URL where the run can be viewed.

**Returns:**

* `str`: The run\_url property value.

### <kbd>property</kbd> `Settings.settings_workspace`

The path to the workspace settings file.

**Returns:**

* `str`: The settings\_workspace property value.

### <kbd>property</kbd> `Settings.sweep_url`

The W\&B URL where the sweep can be viewed.

**Returns:**

* `str`: The sweep\_url property value.

### <kbd>property</kbd> `Settings.sync_dir`

The directory for storing the run's files.

**Returns:**

* `str`: The sync\_dir property value.

### <kbd>property</kbd> `Settings.sync_file`

Path to the append-only binary transaction log file.

**Returns:**

* `str`: The sync\_file property value.

### <kbd>property</kbd> `Settings.sync_symlink_latest`

Path to the symlink to the most recent run's transaction log file.

**Returns:**

* `str`: The sync\_symlink\_latest property value.

### <kbd>property</kbd> `Settings.timespec`

The time specification for the run.

**Returns:**

* `str`: The timespec property value.

### <kbd>property</kbd> `Settings.wandb_dir`

Full path to the wandb directory.

**Returns:**

* `str`: The wandb\_dir property value.


# System Metrics Reference
Source: https://docs.wandb.ai/models/ref/python/experiments/system-metrics

Reference for CPU, GPU, memory, disk, and network system metrics that the W&B SDK automatically logs during runs.

This page provides detailed information about the system metrics that are tracked by the W\&B SDK.

<Note>
  `wandb` automatically logs system metrics every 15 seconds.
</Note>

## View system metrics

You can view and monitor system metrics using the W\&B App or the `wandb beta leet` terminal UI.

<Tabs>
  <Tab title="App">
    To view system metrics in the W\&B App:

    1. Navigate to your project in the W\&B App.
    2. Select a run from the **Runs** table.
    3. In the workspace, look for the **System** section which displays charts for:
       * GPU utilization and memory
       * CPU usage
       * Memory usage
       * Disk I/O
       * Network traffic

    You can customize which system metrics to display by adding panels to your workspace. See [Panels](/models/app/features/panels/) for more information on creating and customizing visualizations.
  </Tab>

  <Tab title="LEET">
    To view system metrics for a run in your terminal using the `wandb beta leet` terminal UI:

    1. If you started the run locally from a script, navigate to the directory where you ran your code. It contains a `wandb/` directory with a subdirectory per run and a `latest-run/` symbolic link. Each run directory contains a transaction log named in the format `run-<run-ID>.wandb`.

       If you did not start the run locally but downloaded a `.wandb` transaction log file instead, make a note of its location.
    2. Start `wandb beta leet` using one of these commands:

       ```bash theme={null}
       # View the latest run, stored in ./wandb/latest-run/
       wandb beta leet

       # Specify a run directory
       wandb beta leet ./wandb/run-20250813_124246-n67z9ude

       # Specify a .wandb file
       wandb beta leet ./wandb/run-20250813_124246-n67z9ude/run-n67z9ude.wandb
       ```

    LEET displays system metrics in the **right sidebar**, showing:

    * GPU utilization (%) and memory usage (GB)
    * CPU usage
    * RAM usage (GB)
    * Disk I/O
    * Network activity

    Get started with these keyboard shortcuts:

    * `h` or `?` - View all keyboard shortcuts
    * `/` - Filter metrics by pattern
    * `[` / `]` - Toggle left/right sidebars
    * `n` / `N` - Navigate between metric pages
    * `q` / `CMD+C` - Quit

    See [`wandb beta leet`](/models/ref/cli/wandb-beta/wandb-beta-leet) for more details.
  </Tab>
</Tabs>

## CPU

### Process CPU Percent (CPU)

Percentage of CPU usage by the process, normalized by the number of available CPUs.

W\&B assigns a `cpu` tag to this metric.

### Process CPU Threads

The number of threads utilized by the process.

W\&B assigns a `proc.cpu.threads` tag to this metric.

## Disk

By default, the usage metrics are collected for the `/` path. To configure the paths to be monitored, use the following setting:

```python theme={null}
run = wandb.init(
    settings=wandb.Settings(
        x_stats_disk_paths=("/System/Volumes/Data", "/home", "/mnt/data"),
    ),
)
```

### Disk Usage Percent

Represents the total system disk usage in percentage for specified paths.

W\&B assigns a `disk.{path}.usagePercent` tag to this metric.

### Disk Usage

Represents the total system disk usage in gigabytes (GB) for specified paths.
The paths that are accessible are sampled, and the disk usage (in GB) for each path is appended to the samples.

W\&B assigns a `disk.{path}.usageGB` tag to this metric.

### Disk In

Indicates the total system disk read in megabytes (MB).
The initial disk read bytes are recorded when the first sample is taken. Subsequent samples calculate the difference between the current read bytes and the initial value.

W\&B assigns a `disk.in` tag to this metric.

### Disk Out

Represents the total system disk write in megabytes (MB).
Similar to [Disk In](), the initial disk write bytes are recorded when the first sample is taken. Subsequent samples calculate the difference between the current write bytes and the initial value.

W\&B assigns a `disk.out` tag to this metric.

## Memory

### Process Memory RSS

Represents the Memory Resident Set Size (RSS) in megabytes (MB) for the process. RSS is the portion of memory occupied by a process that is held in main memory (RAM).

W\&B assigns a `proc.memory.rssMB` tag to this metric.

### Process Memory Percent

Indicates the memory usage of the process as a percentage of the total available memory.

W\&B assigns a `proc.memory.percent` tag to this metric.

### Memory Percent

Represents the total system memory usage as a percentage of the total available memory.

W\&B assigns a `memory_percent` tag to this metric.

### Memory Available

Indicates the total available system memory in megabytes (MB).

W\&B assigns a `proc.memory.availableMB` tag to this metric.

## Network

### Network Sent

Represents the total bytes sent over the network.
The initial bytes sent are recorded when the metric is first initialized. Subsequent samples calculate the difference between the current bytes sent and the initial value.

W\&B assigns a `network.sent` tag to this metric.

### Network Received

Indicates the total bytes received over the network.
Similar to [Network Sent](), the initial bytes received are recorded when the metric is first initialized. Subsequent samples calculate the difference between the current bytes received and the initial value.

W\&B assigns a `network.recv` tag to this metric.

## NVIDIA GPU

In addition to the metrics described below, if the process and/or its descendants use a particular GPU, W\&B captures the corresponding metrics as `gpu.process.{gpu_index}.{metric_name}`

### GPU Memory Utilization

Represents the GPU memory utilization in percent for each GPU.

W\&B assigns a `gpu.{gpu_index}.memory` tag to this metric.

### GPU Memory Allocated

Indicates the GPU memory allocated as a percentage of the total available memory for each GPU.

W\&B assigns a `gpu.{gpu_index}.memoryAllocated` tag to this metric.

### GPU Memory Allocated Bytes

Specifies the GPU memory allocated in bytes for each GPU.

W\&B assigns a `gpu.{gpu_index}.memoryAllocatedBytes` tag to this metric.

### GPU Utilization

Reflects the GPU utilization in percent for each GPU.

W\&B assigns a `gpu.{gpu_index}.gpu` tag to this metric.

### GPU Temperature

The GPU temperature in Celsius for each GPU.

W\&B assigns a `gpu.{gpu_index}.temp` tag to this metric.

### GPU Power Usage Watts

Indicates the GPU power usage in Watts for each GPU.

W\&B assigns a `gpu.{gpu_index}.powerWatts` tag to this metric.

### GPU Power Usage Percent

Reflects the GPU power usage as a percentage of its power capacity for each GPU.

W\&B assigns a `gpu.{gpu_index}.powerPercent` tag to this metric.

### GPU SM Clock Speed

Represents the clock speed of the Streaming Multiprocessor (SM) on the GPU in MHz. This metric is indicative of the processing speed within the GPU cores responsible for computation tasks.

W\&B assigns a `gpu.{gpu_index}.smClock` tag to this metric.

### GPU Memory Clock Speed

Represents the clock speed of the GPU memory in MHz, which influences the rate of data transfer between the GPU memory and processing cores.

W\&B assigns a `gpu.{gpu_index}.memoryClock` tag to this metric.

### GPU Graphics Clock Speed

Represents the base clock speed for graphics rendering operations on the GPU, expressed in MHz. This metric often reflects performance during visualization or rendering tasks.

W\&B assigns a `gpu.{gpu_index}.graphicsClock` tag to this metric.

### GPU Corrected Memory Errors

Tracks the count of memory errors on the GPU that W\&B automatically corrects by error-checking protocols, indicating recoverable hardware issues.

W\&B assigns a `gpu.{gpu_index}.correctedMemoryErrors` tag to this metric.

### GPU Uncorrected Memory Errors

Tracks the count of memory errors on the GPU that W\&B uncorrected, indicating non-recoverable errors which can impact processing reliability.

W\&B assigns a `gpu.{gpu_index}.unCorrectedMemoryErrors` tag to this metric.

### GPU Encoder Utilization

Represents the percentage utilization of the GPU's video encoder, indicating its load when encoding tasks (for example, video rendering) are running.

W\&B assigns a `gpu.{gpu_index}.encoderUtilization` tag to this metric.

## AMD GPU

W\&B extracts metrics from the output of the `rocm-smi` tool supplied by AMD (`rocm-smi -a --json`).

ROCm [6.x (latest)](https://rocm.docs.amd.com/en/latest/) and [5.x](https://rocm.docs.amd.com/en/docs-5.6.0/) formats are supported. Learn more about ROCm formats in the [AMD ROCm documentation](https://rocm.docs.amd.com/en/latest/compatibility/compatibility-matrix.html). The newer format includes more details.

### AMD GPU Utilization

Represents the GPU utilization in percent for each AMD GPU device.

W\&B assigns a `gpu.{gpu_index}.gpu` tag to this metric.

### AMD GPU Memory Allocated

Indicates the GPU memory allocated as a percentage of the total available memory for each AMD GPU device.

W\&B assigns a `gpu.{gpu_index}.memoryAllocated` tag to this metric.

### AMD GPU Temperature

The GPU temperature in Celsius for each AMD GPU device.

W\&B assigns a `gpu.{gpu_index}.temp` tag to this metric.

### AMD GPU Power Usage Watts

The GPU power usage in Watts for each AMD GPU device.

W\&B assigns a `gpu.{gpu_index}.powerWatts` tag to this metric.

### AMD GPU Power Usage Percent

Reflects the GPU power usage as a percentage of its power capacity for each AMD GPU device.

W\&B assigns a `gpu.{gpu_index}.powerPercent` to this metric.

## Apple ARM Mac GPU

### Apple GPU Utilization

Indicates the GPU utilization in percent for Apple GPU devices, specifically on ARM Macs.

W\&B assigns a `gpu.0.gpu` tag to this metric.

### Apple GPU Memory Allocated

The GPU memory allocated as a percentage of the total available memory for Apple GPU devices on ARM Macs.

W\&B assigns a `gpu.0.memoryAllocated` tag to this metric.

### Apple GPU Temperature

The GPU temperature in Celsius for Apple GPU devices on ARM Macs.

W\&B assigns a `gpu.0.temp` tag to this metric.

### Apple GPU Power Usage Watts

The GPU power usage in Watts for Apple GPU devices on ARM Macs.

W\&B assigns a `gpu.0.powerWatts` tag to this metric.

### Apple GPU Power Usage Percent

The GPU power usage as a percentage of its power capacity for Apple GPU devices on ARM Macs.

W\&B assigns a `gpu.0.powerPercent` tag to this metric.

## Graphcore IPU

Graphcore IPUs (Intelligence Processing Units) are unique hardware accelerators designed specifically for machine intelligence tasks.

### IPU Device Metrics

These metrics represent various statistics for a specific IPU device. Each metric has a device ID (`device_id`) and a metric key (`metric_key`) to identify it. W\&B assigns a `ipu.{device_id}.{metric_key}` tag to this metric.

Metrics are extracted using the proprietary `gcipuinfo` library, which interacts with Graphcore's `gcipuinfo` binary. The `sample` method fetches these metrics for each IPU device associated with the process ID (`pid`). Only the metrics that change over time, or the first time a device's metrics are fetched, are logged to avoid logging redundant data.

For each metric, the method `parse_metric` is used to extract the metric's value from its raw string representation. The metrics are then aggregated across multiple samples using the `aggregate` method.

The following lists available metrics and their units:

* **Average Board Temperature** (`average board temp (C)`): Temperature of the IPU board in Celsius.
* **Average Die Temperature** (`average die temp (C)`): Temperature of the IPU die in Celsius.
* **Clock Speed** (`clock (MHz)`): The clock speed of the IPU in MHz.
* **IPU Power** (`ipu power (W)`): Power consumption of the IPU in Watts.
* **IPU Utilization** (`ipu utilisation (%)`): Percentage of IPU utilization.
* **IPU Session Utilization** (`ipu utilisation (session) (%)`): IPU utilization percentage specific to the current session.
* **Data Link Speed** (`speed (GT/s)`): Speed of data transmission in Giga-transfers per second.

## Google Cloud TPU

Tensor Processing Units (TPUs) are Google's custom-developed ASICs (Application Specific Integrated Circuits) used to accelerate machine learning workloads.

<Note>
  The exact metrics reported depend on the TPU chip generation, such as v4, v5e, v5p, v6e, or 7x and on the runtime environment.
</Note>

See [Supported metrics in Google's Cloud TPU documentation](https://docs.cloud.google.com/tpu/docs/tpu-monitoring-library) for more information about the available metrics and their definitions.

### TPU Compute metrics

* `tpu.{tpu_index}.tensorcoreUtilization`: TensorCore utilization percentage per device. It is the most direct measure of whether TensorCore compute units are being fully utilized.

* `tpu.{tpu_index}.dutyCycle`: The percentage of time during the sample period when the accelerator TensorCore was actively processing. This metric is reported per chip and fanned out to all devices on multi-device chips. Higher values indicate better TensorCore utilization.

### TPU Memory metrics

* `tpu.{tpu_index}.hbmCapacityTotal`: Total High Bandwidth Memory (HBM) capacity in bytes per device.

* `tpu.{tpu_index}.hbmCapacityUsage`: Current HBM usage in bytes per device.

### TPU Interconnect health

* `tpu.{tpu_index}.iciLinkHealth`: Inter-Chip Interconnect (ICI) link health per device. This metric is available only through the libtpu SDK path.

### TPU Transfer latency

Latency distributions in microseconds. Reported statistics can include mean, p50, p90, p95, and p999. These metrics are most relevant for multi-slice TPU pod configurations.

* `tpu.bufferTransferLatency.{label}.{stat}Us`: DCN (Data Center Network) buffer transfer latencies between slices.

* `tpu.inboundBufferTransferLatency.{label}.{stat}Us`: Inbound DCN buffer transfer latencies.

* `tpu.hostToDeviceTransferLatency.{label}.{stat}Us`: Host-to-device data transfer latencies.

* `tpu.deviceToHostTransferLatency.{label}.{stat}Us`: Device-to-host data transfer latencies.

### TPU Collective communication

* `tpu.collectiveE2ELatency.{label}.{stat}Us`: End-to-end latency for collective operations, such as all-reduce and all-gather.

* `tpu.hostComputeLatency.{label}.{stat}Us`: Host-side compute latencies, including MXLA compute latencies.

### TPU Network metrics

* `tpu.grpcTcpMinRtt.{stat}Us`: Minimum TCP round-trip time for gRPC connections.

* `tpu.grpcTcpDeliveryRate.{stat}Mbps`: TCP delivery rate for gRPC connections, in megabits per second.

### TPU HLO execution metrics

* `tpu.hloExecTiming.{label}.{stat}Us`: HLO (High Level Operations) execution timing distributions in microseconds. This metric reports per-operation execution time.

* `tpu.hloQueueSize.{label}`: Current size of the HLO execution queue. This metric indicates how many operations are queued for execution.

## AWS Trainium

[AWS Trainium](https://aws.amazon.com/machine-learning/trainium/) is a specialized hardware platform offered by AWS that focuses on accelerating machine learning workloads. The `neuron-monitor` tool from AWS is used to capture the AWS Trainium metrics.

### Trainium Neuron Core Utilization

The utilization percentage of each NeuronCore, reported on a per-core basis.

W\&B assigns a `trn.{core_index}.neuroncore_utilization` tag to this metric.

### Trainium Host Memory Usage, Total

The total memory consumption on the host in bytes.

W\&B assigns a `trn.host_total_memory_usage` tag to this metric.

### Trainium Neuron Device Total Memory Usage

The total memory usage on the Neuron device in bytes.

W\&B assigns a  `trn.neuron_device_total_memory_usage)` tag to this metric.

### Trainium Host Memory Usage Breakdown:

The following is a breakdown of memory usage on the host:

* **Application Memory** (`trn.host_total_memory_usage.application_memory`): Memory used by the application.
* **Constants** (`trn.host_total_memory_usage.constants`): Memory used for constants.
* **DMA Buffers** (`trn.host_total_memory_usage.dma_buffers`): Memory used for Direct Memory Access buffers.
* **Tensors** (`trn.host_total_memory_usage.tensors`): Memory used for tensors.

### Trainium Neuron Core Memory Usage Breakdown

Detailed memory usage information for each NeuronCore:

* **Constants** (`trn.{core_index}.neuroncore_memory_usage.constants`)
* **Model Code** (`trn.{core_index}.neuroncore_memory_usage.model_code`)
* **Model Shared Scratchpad** (`trn.{core_index}.neuroncore_memory_usage.model_shared_scratchpad`)
* **Runtime Memory** (`trn.{core_index}.neuroncore_memory_usage.runtime_memory`)
* **Tensors** (`trn.{core_index}.neuroncore_memory_usage.tensors`)

## OpenMetrics

Capture and log metrics from external endpoints that expose OpenMetrics / Prometheus-compatible data with support for custom regex-based metric filters to be applied to the consumed endpoints.

Refer to [Monitoring GPU cluster performance in W\&B](https://wandb.ai/dimaduev/dcgm/reports/Monitoring-GPU-cluster-performance-with-NVIDIA-DCGM-Exporter-and-Weights-Biases--Vmlldzo0MDYxMTA1) for a detailed example of how to use this feature in a particular case of monitoring GPU cluster performance with the [NVIDIA DCGM-Exporter](https://docs.nvidia.com/datacenter/cloud-native/gpu-telemetry/latest/dcgm-exporter.html).


# controller()
Source: https://docs.wandb.ai/models/ref/python/functions/controller



<GitHubLink />

### <kbd>function</kbd> `controller`

```python theme={null}
controller(
    sweep_id_or_config: 'str | dict | None' = None,
    entity: 'str | None' = None,
    project: 'str | None' = None
) → _WandbController
```

Public sweep controller constructor.

**Examples:**

```python theme={null}
import wandb

tuner = wandb.controller(...)
print(tuner.sweep_config)
print(tuner.sweep_id)
tuner.configure_search(...)
tuner.configure_stopping(...)
```


# finish()
Source: https://docs.wandb.ai/models/ref/python/functions/finish



<GitHubLink />

### <kbd>function</kbd> `finish`

```python theme={null}
finish(exit_code: 'int | None' = None, quiet: 'bool | None' = None) → None
```

Finish a run and upload any remaining data.

Marks the completion of a W\&B run and ensures all data is synced to the server. The run's final state is determined by its exit conditions and sync status.

Run States:

* Running: Active run that is logging data and/or sending heartbeats.
* Crashed: Run that stopped sending heartbeats unexpectedly.
* Finished: Run completed successfully (`exit_code=0`) with all data synced.
* Failed: Run completed with errors (`exit_code!=0`).

**Args:**

* `exit_code`:  Integer indicating the run's exit status. Use 0 for success,  any other value marks the run as failed.
* `quiet`:  Deprecated. Configure logging verbosity using `wandb.Settings(quiet=...)`.


# init()
Source: https://docs.wandb.ai/models/ref/python/functions/init



<GitHubLink />

### <kbd>function</kbd> `init`

```python theme={null}
init(
    entity: 'str | None' = None,
    project: 'str | None' = None,
    dir: 'StrPath | None' = None,
    id: 'str | None' = None,
    name: 'str | None' = None,
    notes: 'str | None' = None,
    tags: 'Sequence[str] | None' = None,
    config: 'dict[str, Any] | str | None' = None,
    config_exclude_keys: 'list[str] | None' = None,
    config_include_keys: 'list[str] | None' = None,
    allow_val_change: 'bool | None' = None,
    group: 'str | None' = None,
    job_type: 'str | None' = None,
    mode: "Literal['online', 'offline', 'disabled', 'shared'] | None" = None,
    force: 'bool | None' = None,
    reinit: "bool | Literal[None, 'default', 'return_previous', 'finish_previous', 'create_new']" = None,
    resume: "bool | Literal['allow', 'never', 'must', 'auto'] | None" = None,
    resume_from: 'str | None' = None,
    fork_from: 'str | None' = None,
    save_code: 'bool | None' = None,
    tensorboard: 'bool | None' = None,
    sync_tensorboard: 'bool | None' = None,
    monitor_gym: 'bool | None' = None,
    settings: 'Settings | dict[str, Any] | None' = None,
    anonymous: 'DoNotSet' = <object object at 0x104fe52d0>
) → Run
```

Start a new run to track and log to W\&B.

In an ML training pipeline, you could add `wandb.init()` to the beginning of your training script as well as your evaluation script, and each piece would be tracked as a run in W\&B.

`wandb.init()` spawns a new background process to log data to a run, and it also syncs data to [https://wandb.ai](https://wandb.ai) by default, so you can see your results in real-time. When you're done logging data, call `run.finish()` to end the run, or use the run as a context manager to call it automatically:

with wandb.init() as run:  ...  # run.finish() executes at the end of the block

If you don't call `run.finish()`, the run will end when your script exits.

Run IDs must not contain any of the following special characters `/ \ # ? % :`

**Args:**

* `entity`:  The username or team name the runs are logged to.  The entity must already exist, so ensure you create your account  or team in the UI before starting to log runs. If not specified, the  run will default your default entity. To change the default entity,  go to your settings and update the  "Default location to create new projects" under "Default team".
* `project`:  The name of the project under which this run will be logged.  If not specified, we use a heuristic to infer the project name based  on the system, such as checking the git root or the current program  file. If we can't infer the project name, the project will default to  `"uncategorized"`.
* `dir`:  The absolute path to the directory where experiment logs and  metadata files are stored. If not specified, this defaults  to the `./wandb` directory. Note that this does not affect the  location where artifacts are stored when calling `download()`.
* `id`:  A unique identifier for this run, used for resuming. It must be unique  within the project and cannot be reused once a run is deleted. For  a short descriptive name, use the `name` field,  or for saving hyperparameters to compare across runs, use `config`.
* `name`:  A short display name for this run, which appears in the UI to help  you identify it. By default, we generate a random two-word name  allowing easy cross-reference runs from table to charts. Keeping these  run names brief enhances readability in chart legends and tables. For  saving hyperparameters, we recommend using the `config` field.
* `notes`:  A detailed description of the run, similar to a commit message in  Git. Use this argument to capture any context or details that may  help you recall the purpose or setup of this run in the future.
* `tags`:  A list of tags to label this run in the UI. Tags are helpful for  organizing runs or adding temporary identifiers like "baseline" or  "production." You can easily add, remove tags, or filter by tags in  the UI.  If resuming a run, the tags provided here will replace any existing  tags. To add tags to a resumed run without overwriting the current  tags, use `run.tags += ("new_tag",)` after calling `run = wandb.init()`.
* `config`:  Sets `wandb.config`, a dictionary-like object for storing input  parameters to your run, such as model hyperparameters or data  preprocessing settings.  The config appears in the UI in an overview page, allowing you to  group, filter, and sort runs based on these parameters.  Keys should not contain periods (`.`), and values should be  smaller than 10 MB.  If a dictionary, `argparse.Namespace`, or `absl.flags.FLAGS` is  provided, the key-value pairs will be loaded directly into  `wandb.config`.  If a string is provided, it is interpreted as a path to a YAML file,  from which configuration values will be loaded into `wandb.config`.
* `config_exclude_keys`:  A list of specific keys to exclude from `wandb.config`.
* `config_include_keys`:  A list of specific keys to include in `wandb.config`.
* `allow_val_change`:  Controls whether config values can be modified after their  initial set. By default, an exception is raised if a config value is  overwritten. For tracking variables that change during training, such as  a learning rate, consider using `wandb.log()` instead. By default, this  is `False` in scripts and `True` in Notebook environments.
* `group`:  Specify a group name to organize individual runs as part of a larger  experiment. This is useful for cases like cross-validation or running  multiple jobs that train and evaluate a model on different test sets.  Grouping allows you to manage related runs collectively in the UI,  making it easy to toggle and review results as a unified experiment.
* `job_type`:  Specify the type of run, especially helpful when organizing runs  within a group as part of a larger experiment. For example, in a group,  you might label runs with job types such as "train" and "eval".  Defining job types enables you to easily filter and group similar runs  in the UI, facilitating direct comparisons.
* `mode`:  Specifies how run data is managed, with the following options:
  * `"online"` (default): Enables live syncing with W\&B when a network  connection is available, with real-time updates to visualizations.
  * `"offline"`: Suitable for air-gapped or offline environments; data  is saved locally and can be synced later. Ensure the run folder  is preserved to enable future syncing.
  * `"disabled"`: Disables all W\&B functionality, making the run’s methods  no-ops. Typically used in testing to bypass W\&B operations.
  * `"shared"`: (This is an experimental feature). Allows multiple processes,  possibly on different machines, to simultaneously log to the same run.  In this approach you use a primary node and one or more worker nodes  to log data to the same run. Within the primary node you  initialize a run. For each worker node, initialize a run  using the run ID used by the primary node.
* `force`:  Determines if a W\&B login is required to run the script. If `True`,  the user must be logged in to W\&B; otherwise, the script will not  proceed. If `False` (default), the script can proceed without a login,  switching to offline mode if the user is not logged in.
* `reinit`:  Shorthand for the "reinit" setting. Determines the behavior of  `wandb.init()` when a run is active.
* `resume`:  Controls the behavior when resuming a run with the specified `id`.  Available options are:
  * `"allow"`: If a run with the specified `id` exists, it will resume  from the last step; otherwise, a new run will be created.
  * `"never"`: If a run with the specified `id` exists, an error will  be raised. If no such run is found, a new run will be created.
  * `"must"`: If a run with the specified `id` exists, it will resume  from the last step. If no run is found, an error will be raised.
  * `"auto"`: Automatically resumes the previous run if it crashed on  this machine; otherwise, starts a new run.
  * `True`: Deprecated. Use `"auto"` instead.
  * `False`: Deprecated. Use the default behavior (leaving `resume`  unset) to always start a new run.  If `resume` is set, `fork_from` and `resume_from` cannot be  used. When `resume` is unset, the system will always start a new run.
* `resume_from`:  Specifies a moment in a previous run to resume a run from,  using the format `{run_id}?_step={step}`. This allows users to truncate  the history logged to a run at an intermediate step and resume logging  from that step. The target run must be in the same project.  If an `id` argument is also provided, the `resume_from` argument will  take precedence.  `resume`, `resume_from` and `fork_from` cannot be used together, only  one of them can be used at a time.  Note that this feature is in beta and may change in the future.
* `fork_from`:  Specifies a point in a previous run from which to fork a new  run, using the format `{id}?_step={step}`. This creates a new run that  resumes logging from the specified step in the target run’s history.  The target run must be part of the current project.  If an `id` argument is also provided, it must be different from the  `fork_from` argument, an error will be raised if they are the same.  `resume`, `resume_from` and `fork_from` cannot be used together, only  one of them can be used at a time.  Note that this feature is in beta and may change in the future.
* `save_code`:  Enables saving the main script or notebook to W\&B, aiding in  experiment reproducibility and allowing code comparisons across runs in  the UI. By default, this is disabled, but you can change the default to  enable on your settings page.
* `tensorboard`:  Deprecated. Use `sync_tensorboard` instead.
* `sync_tensorboard`:  Enables automatic syncing of W\&B logs from TensorBoard  or TensorBoardX, saving relevant event files for viewing in  the W\&B UI.
* `monitor_gym`:  Enables automatic logging of videos of the environment when  using OpenAI Gym.
* `settings`:  Specifies a dictionary or `wandb.Settings` object with advanced  settings for the run.

**Returns:**
A `Run` object.

**Raises:**

* `Error`:  If some unknown or internal error happened during the run  initialization.
* `AuthenticationError`:  If the user failed to provide valid credentials.
* `CommError`:  If there was a problem communicating with the WandB server.
* `UsageError`:  If the user provided invalid arguments.
* `KeyboardInterrupt`:  If user interrupts the run.

**Examples:**
`wandb.init()` returns a `Run` object. Use the run object to log data, save artifacts, and manage the run lifecycle.

```python theme={null}
import wandb

config = {"lr": 0.01, "batch_size": 32}
with wandb.init(config=config) as run:
    # Log accuracy and loss to the run
    acc = 0.95  # Example accuracy
    loss = 0.05  # Example loss
    run.log({"accuracy": acc, "loss": loss})
```


# login()
Source: https://docs.wandb.ai/models/ref/python/functions/login



<GitHubLink />

### <kbd>function</kbd> `login`

```python theme={null}
login(
    key: 'str | None' = None,
    relogin: 'bool | None' = None,
    host: 'str | None' = None,
    force: 'bool | None' = None,
    timeout: 'int | None' = None,
    verify: 'bool' = False,
    referrer: 'str | None' = None,
    anonymous: 'DoNotSet' = <object object at 0x104fe52d0>
) → bool
```

Log into W\&B.

You generally don't have to use this because most W\&B methods that need authentication can log in implicitly. This is the programmatic counterpart to the `wandb login` CLI.

This updates global credentials for the session (affecting all wandb usage in the current Python process after this call) and possibly the .netrc file.

If the identity\_token\_file setting is set, like through the WANDB\_IDENTITY\_TOKEN\_FILE environment variable, then this is a no-op.

Otherwise, if an explicit API key is provided, it is used and written to the system .netrc file. If no key is provided, but the session is already authenticated, then the session key is used for verification (if verify is True) and the .netrc file is not updated.

If none of the above is true, then this gets the API key from the first of:

* The WANDB\_API\_KEY environment variable
* The api\_key setting in a system or workspace settings file
* The .netrc file (either \~/.netrc, \~/\_netrc or the path specified by the  NETRC environment variable)
* An interactive prompt (if available)

**Args:**

* `key`:  The API key to use.
* `relogin`:  If true, get the API key from an interactive prompt, skipping  reading .netrc, environment variables, etc.
* `host`:  The W\&B server URL to connect to.
* `force`:  If true, disallows selecting offline mode in the interactive  prompt.
* `timeout`:  Number of seconds to wait for user input in the interactive  prompt. This can be used as a failsafe if an interactive prompt  is incorrectly shown in a non-interactive environment.
* `verify`:  Verify the credentials with the W\&B server and raise an  AuthenticationError on failure.
* `referrer`:  The referrer to use in the URL login request for analytics.

**Returns:**

* `bool`:  If `key` is configured.

**Raises:**

* `AuthenticationError`:  If `api_key` fails verification with the server.
* `UsageError`:  If `api_key` cannot be configured and no tty.


# restore()
Source: https://docs.wandb.ai/models/ref/python/functions/restore



<GitHubLink />

### <kbd>function</kbd> `restore`

```python theme={null}
restore(
    name: 'str',
    run_path: 'str | None' = None,
    replace: 'bool' = False,
    root: 'str | None' = None
) → None | TextIO
```

Download the specified file from cloud storage.

File is placed into the current directory or run directory. By default, will only download the file if it doesn't already exist.

**Args:**

* `name`:  The name of the file.
* `run_path`:  Optional path to a run to pull files from, i.e. `username/project_name/run_id`  if wandb.init has not been called, this is required.
* `replace`:  Whether to download the file even if it already exists locally
* `root`:  The directory to download the file to.  Defaults to the current  directory or the run directory if wandb.init was called.

**Returns:**
None if it can't find the file, otherwise a file object open for reading.

**Raises:**

* `CommError`:  If W\&B can't connect to the W\&B backend.
* `ValueError`:  If the file is not found or can't find run\_path.


# setup()
Source: https://docs.wandb.ai/models/ref/python/functions/setup



<GitHubLink />

### <kbd>function</kbd> `setup`

```python theme={null}
setup(settings: 'Settings | None' = None) → _WandbSetup
```

Prepares W\&B for use in the current process and its children.

You can usually ignore this as it is implicitly called by `wandb.init()`.

When using wandb in multiple processes, calling `wandb.setup()` in the parent process before starting child processes may improve performance and resource utilization.

Note that `wandb.setup()` modifies `os.environ`, and it is important that child processes inherit the modified environment variables.

See also `wandb.teardown()`.

**Args:**

* `settings`:  Configuration settings to apply globally. These can be  overridden by subsequent `wandb.init()` calls.

**Example:**

```python theme={null}
import multiprocessing

import wandb


def run_experiment(params):
   with wandb.init(config=params):
        # Run experiment
        pass


if __name__ == "__main__":
   # Start backend and set global config
   wandb.setup(settings={"project": "my_project"})

   # Define experiment parameters
   experiment_params = [
        {"learning_rate": 0.01, "epochs": 10},
        {"learning_rate": 0.001, "epochs": 20},
   ]

   # Start multiple processes, each running a separate experiment
   processes = []
   for params in experiment_params:
        p = multiprocessing.Process(target=run_experiment, args=(params,))
        p.start()
        processes.append(p)

   # Wait for all processes to complete
   for p in processes:
        p.join()

   # Optional: Explicitly shut down the backend
   wandb.teardown()
```


# sweep()
Source: https://docs.wandb.ai/models/ref/python/functions/sweep



<GitHubLink />

### <kbd>function</kbd> `sweep`

```python theme={null}
sweep(
    sweep: 'dict | Callable',
    entity: 'str | None' = None,
    project: 'str | None' = None,
    prior_runs: 'list[str] | None' = None
) → str
```

Initialize a hyperparameter sweep.

Search for hyperparameters that optimizes a cost function of a machine learning model by testing various combinations.

Make note the unique identifier, `sweep_id`, that is returned. At a later step provide the `sweep_id` to a sweep agent.

See [Sweep configuration structure](https://docs.wandb.ai/models/sweeps/define-sweep-configuration) for information on how to define your sweep.

**Args:**

* `sweep`:  The configuration of a hyperparameter search.  (or configuration generator).  If you provide a callable, ensure that the callable does  not take arguments and that it returns a dictionary that  conforms to the W\&B sweep config spec.
* `entity`:  The username or team name where you want to send W\&B  runs created by the sweep to. Ensure that the entity you  specify already exists. If you don't specify an entity,  the run will be sent to your default entity,  which is usually your username.
* `project`:  The name of the project where W\&B runs created from  the sweep are sent to. If the project is not specified, the  run is sent to a project labeled 'Uncategorized'.
* `prior_runs`:  The run IDs of existing runs to add to this sweep.

**Returns:**

* `str`:  A unique identifier for the sweep.


# teardown()
Source: https://docs.wandb.ai/models/ref/python/functions/teardown



<GitHubLink />

### <kbd>function</kbd> `teardown`

```python theme={null}
teardown(exit_code: 'int | None' = None) → None
```

Waits for W\&B to finish and frees resources.

Completes any runs that were not explicitly finished using `run.finish()` and waits for all data to be uploaded.

It is recommended to call this at the end of a session that used `wandb.setup()`. It is invoked automatically in an `atexit` hook, but this is not reliable in certain setups such as when using Python's `multiprocessing` module.


# Public API overview
Source: https://docs.wandb.ai/models/ref/python/public-api

Use the W&B Public API to programmatically access and manage W&B data

The W\&B Public API provides programmatic access to query, export, and update data stored in W\&B. Use this API for post-hoc analysis, data export, and programmatic management of runs, artifacts, and sweeps. While the main SDK handles real-time logging during training, the Public API enables you to retrieve historical data, update metadata, manage artifacts, and perform analysis on completed experiments. The main `Api` class serves as the entry point to most functionality.

> Training and fine-tuning models is done elsewhere in [the W\&B Python SDK](/models/ref/python). Use the Public API for querying and managing data *after* it has been logged to W\&B.

## Available components

| Component                                             | Description                                                                                        |
| ----------------------------------------------------- | -------------------------------------------------------------------------------------------------- |
| [`Api`](/models/ref/python/public-api/api/)           | Main entry point for the Public API. Query runs, projects, and artifacts across your organization. |
| [`Runs`](/models/ref/python/public-api/runs/)         | Access and manage individual training runs, including history, logs, and metrics.                  |
| [`Artifacts`](/models/artifacts/)                     | Query and download model artifacts, datasets, and other versioned files.                           |
| [`Sweeps`](/models/sweeps/)                           | Access hyperparameter sweep data and analyze optimization results.                                 |
| [`Projects`](/models/ref/python/public-api/projects/) | Manage projects and access project-level metadata and settings.                                    |
| [`Reports`](/models/reports/create-a-report/)         | Programmatically access and manage W\&B Reports.                                                   |
| [`Team`](/models/ref/python/public-api/team)          | Query team information and manage team-level resources.                                            |
| [`User`](/models/ref/python/public-api/user)          | Access user profiles and user-specific data.                                                       |
| [`Files`](/models/ref/python/public-api/files/)       | Download and manage files associated with runs.                                                    |
| `History`                                             | Access detailed time-series metrics logged during training (see Run.history).                      |
| [`Automations`](./automations/)                       | Manage automated workflows and actions.                                                            |
| [`Integrations`](/models/integrations)                | Configure and manage third-party integrations.                                                     |

## Common use cases

### Data export and analysis

* Export run history as DataFrames for analysis in Jupyter notebooks
* Download metrics for custom visualization or reporting
* Aggregate results across multiple experiments

### Post-Hoc Updates

* Update run metadata after completion
* Add tags or notes to completed experiments
* Modify run configurations or summaries

### Artifact Management

* Query artifacts by version or alias
* Download model checkpoints programmatically
* Track artifact lineage and dependencies

### Sweep Analysis

* Access sweep results and best performing runs
* Export hyperparameter search results
* Analyze parameter importance

## Authentication

The Public API uses the same authentication mechanism as the Python SDK. You can authenticate in several ways:

Use the `WANDB_API_KEY` environment variable to set your API key:

```bash theme={null}
export WANDB_API_KEY=your_api_key
```

Pass the API key directly when initializing the `Api` class:

```python theme={null}
api = Api(api_key="your_api_key")
```

Or use `wandb.login()` to authenticate the current session:

```python theme={null}
import wandb

wandb.login()
api = Api()
```

## Example usage

### Download an Artifact by name and alias

The following example shows how to retrieve an artifact logged to W\&B by its name and alias, and then download its contents.

```python theme={null}
import wandb

api = wandb.Api()
artifact = api.artifact("entity/project/artifact:alias")
artifact.download()
```

### Download an Artifact from a registry

The following example shows how to retrieve a linked artifact from a W\&B Registry

```python theme={null}
import wandb

REGISTRY = "<registry_name>"
COLLECTION = "<collection_name>"
VERSION = "<version>"

api = wandb.Api()
artifact_name = f"wandb-registry-{REGISTRY}/{COLLECTION}:{VERSION}"

# Fetch the artifact
fetched_artifact = api.artifact(name = artifact_name)

# Download artifact. Returns path to downloaded contents
downloaded_path = fetched_artifact.download()
```

### Query W\&B Registry

Use Mongo-like filters to query W\&B Registries, Collections, and Artifacts. The following example demonstrates how to filter collections by name using a regular expression.

```python theme={null}
import wandb

# Initialize wandb API
api = wandb.Api()

# Filter all collections, independent of registry, that 
# contains the string `yolo` in the collection name
collection_filters = {
    "name": {"$regex": "yolo"}
}

# Returns an iterable of all collections that match the filters
collections = api.registries().collections(filter=collection_filters)
```

For more information on how to query a registry, collection, or artifact, see the [Find registry items](/models/registry/search_registry).


# Api
Source: https://docs.wandb.ai/models/ref/python/public-api/api



<GitHubLink />

## <kbd>class</kbd> `Api`

Used for querying the W\&B server.

**Examples:**

```python theme={null}
import wandb

wandb.Api()
```

### <kbd>method</kbd> `Api.__init__`

```python theme={null}
__init__(
    overrides: 'dict[str, Any] | None' = None,
    timeout: 'int | None' = None,
    api_key: 'str | None' = None
) → None
```

Initialize the API.

**Args:**

* `overrides`:  You can set `base_url` if you are
* `using a W&B server other than `https`: //api.wandb.ai`. You can also set defaults for `entity`, `project`, and `run`.
* `timeout`:  HTTP timeout in seconds for API requests. If not  specified, the default timeout will be used.
* `api_key`:  API key to use for authentication. If not provided,  the API key from the current environment or configuration will be used.  Prompts for an API key if none is provided  or configured in the environment.

***

### <kbd>property</kbd> Api.default\_entity

Returns the default W\&B entity.

**Returns:**

* `str | None`: The default\_entity property value.

***

### <kbd>property</kbd> Api.user\_agent

Returns W\&B public user agent.

**Returns:**

* `str`: The user\_agent property value.

***

### <kbd>property</kbd> Api.viewer

Returns the viewer object.

**Raises:**

* `ValueError`:  If viewer data is not able to be fetched from W\&B.
* `WandbApiFailedError`:  If an error occurs while making the GraphQL request.

**Returns:**

* `User`: The viewer property value.

***

### <kbd>method</kbd> `Api.artifact`

```python theme={null}
artifact(name: 'str', type: 'str | None' = None)
```

Returns a single artifact.

**Args:**

* `name`:  The artifact's name. The name of an artifact resembles a  filepath that consists, at a minimum, the name of the project  the artifact was logged to, the name of the artifact, and the  artifact's version or alias. Optionally append the entity that  logged the artifact as a prefix followed by a forward slash.  If no entity is specified in the name, the Run or API  setting's entity is used.
* `type`:  The type of artifact to fetch.

**Returns:**
An `Artifact` object.

**Raises:**

* `ValueError`:  If the artifact name is not specified.
* `ValueError`:  If the artifact type is specified but does not  match the type of the fetched artifact.

**Examples:**
In the following code snippets "entity", "project", "artifact", "version", and "alias" are placeholders for your W\&B entity, name of the project the artifact is in, the name of the artifact, and artifact's version, respectively.

```python theme={null}
import wandb

# Specify the project, artifact's name, and the artifact's alias
wandb.Api().artifact(name="project/artifact:alias")

# Specify the project, artifact's name, and a specific artifact version
wandb.Api().artifact(name="project/artifact:version")

# Specify the entity, project, artifact's name, and the artifact's alias
wandb.Api().artifact(name="entity/project/artifact:alias")

# Specify the entity, project, artifact's name, and a specific artifact version
wandb.Api().artifact(name="entity/project/artifact:version")
```

**Note:**

> This method is intended for external use only. Do not call `api.artifact()` within the wandb repository code.

***

### <kbd>method</kbd> `Api.artifact_collection`

```python theme={null}
artifact_collection(type_name: 'str', name: 'str') → ArtifactCollection
```

Returns a single artifact collection by type.

You can use the returned `ArtifactCollection` object to retrieve information about specific artifacts in that collection, and more.

**Args:**

* `type_name`:  The type of artifact collection to fetch.
* `name`:  An artifact collection name. Optionally append the entity  that logged the artifact as a prefix followed by a forward  slash.

**Returns:**
An `ArtifactCollection` object.

**Examples:**
In the proceeding code snippet "type", "entity", "project", and "artifact\_name" are placeholders for the collection type, your W\&B entity, name of the project the artifact is in, and the name of the artifact, respectively.

```python theme={null}
import wandb

collections = wandb.Api().artifact_collection(
    type_name="type", name="entity/project/artifact_name"
)

# Get the first artifact in the collection
artifact_example = collections.artifacts()[0]

# Download the contents of the artifact to the specified root directory.
artifact_example.download()
```

***

### <kbd>method</kbd> `Api.artifact_collection_exists`

```python theme={null}
artifact_collection_exists(name: 'str', type: 'str') → bool
```

Whether an artifact collection exists within a specified project and entity.

**Args:**

* `name`:  An artifact collection name. Optionally append the  entity that logged the artifact as a prefix followed by  a forward slash. If entity or project is not specified,  infer the collection from the override params if they exist.  Otherwise, entity is pulled from the user settings and project  will default to "uncategorized".
* `type`:  The type of artifact collection.

**Returns:**
True if the artifact collection exists, False otherwise.

**Examples:**
In the proceeding code snippet "type", and "collection\_name" refer to the type of the artifact collection and the name of the collection, respectively.

```python theme={null}
import wandb

wandb.Api.artifact_collection_exists(type="type", name="collection_name")
```

***

### <kbd>method</kbd> `Api.artifact_collections`

```python theme={null}
artifact_collections(
    project_name: 'str',
    type_name: 'str',
    per_page: 'int' = 50,
    start: 'str | None' = None
) → ArtifactCollections
```

Returns a collection of matching artifact collections.

**Args:**

* `project_name`:  The name of the project to filter on.
* `type_name`:  The name of the artifact type to filter on.
* `per_page`:  Sets the page size for query pagination.  Usually there is no reason to change this.
* `start`:  Pagination cursor for resuming a past query, captured  from a previous paginator's `.cursor` attribute.

**Returns:**
An iterable `ArtifactCollections` object.

***

### <kbd>method</kbd> `Api.artifact_exists`

```python theme={null}
artifact_exists(name: 'str', type: 'str | None' = None) → bool
```

Whether an artifact version exists within the specified project and entity.

**Args:**

* `name`:  The name of artifact. Add the artifact's entity and project  as a prefix. Append the version or the alias of the artifact  with a colon. If the entity or project is not specified,  W\&B uses override parameters if populated. Otherwise, the  entity is pulled from the user settings and the project is  set to "Uncategorized".
* `type`:  The type of artifact.

**Returns:**
True if the artifact version exists, False otherwise.

**Examples:**
In the proceeding code snippets "entity", "project", "artifact", "version", and "alias" are placeholders for your W\&B entity, name of the project the artifact is in, the name of the artifact, and artifact's version, respectively.

```python theme={null}
import wandb

wandb.Api().artifact_exists("entity/project/artifact:version")
wandb.Api().artifact_exists("entity/project/artifact:alias")
```

***

### <kbd>method</kbd> `Api.artifact_type`

```python theme={null}
artifact_type(type_name: 'str', project: 'str | None' = None) → ArtifactType
```

Returns the matching `ArtifactType`.

**Args:**

* `type_name`:  The name of the artifact type to retrieve.
* `project`:  If given, a project name or path to filter on.

**Returns:**
An `ArtifactType` object.

***

### <kbd>method</kbd> `Api.artifact_types`

```python theme={null}
artifact_types(
    project: 'str | None' = None,
    start: 'str | None' = None
) → ArtifactTypes
```

Returns a collection of matching artifact types.

**Args:**

* `project`:  The project name or path to filter on.
* `start`:  Pagination cursor for resuming a past query, captured  from a previous paginator's `.cursor` attribute.

**Returns:**
An iterable `ArtifactTypes` object.

***

### <kbd>method</kbd> `Api.artifact_versions`

```python theme={null}
artifact_versions(type_name, name, per_page=50)
```

Deprecated. Use `Api.artifacts(type_name, name)` method instead.

***

### <kbd>method</kbd> `Api.artifacts`

```python theme={null}
artifacts(
    type_name: 'str',
    name: 'str',
    per_page: 'int' = 50,
    tags: 'list[str] | None' = None,
    start: 'str | None' = None
) → Artifacts
```

Return an `Artifacts` collection.

**Args:**

* `type_name`:  The type of artifacts to fetch.
* `name`:  The artifact's collection name. Optionally append the  entity that logged the artifact as a prefix followed by  a forward slash.
* `per_page`:  Sets the page size for query pagination. Usually  there is no reason to change this.
* `tags`:  Only return artifacts with all of these tags.
* `start`:  Pagination cursor for resuming a past query, captured  from a previous paginator's `.cursor` attribute.

**Returns:**
An iterable `Artifacts` object.

**Examples:**
In the proceeding code snippet, "type", "entity", "project", and "artifact\_name" are placeholders for the artifact type, W\&B entity, name of the project the artifact was logged to, and the name of the artifact, respectively.

```python theme={null}
import wandb

wandb.Api().artifacts(type_name="type", name="entity/project/artifact_name")
```

Pause iteration and resume later from the same position by saving the paginator's `.cursor` and passing it as `start=`:

```python theme={null}
from itertools import islice

import wandb

api = wandb.Api()

# Consume the first page of results, then save the cursor.
page_size = 10
artifacts = api.artifacts(
    type_name="type",
    name="entity/project/artifact_name",
    per_page=page_size,
)
first_page = list(islice(artifacts, page_size))

saved_cursor = artifacts.cursor

# Later (e.g. in a new process), resume iteration from the saved cursor.
remaining_artifacts = api.artifacts(
    type_name="type",
    name="entity/project/artifact_name",
    per_page=page_size,
    start=saved_cursor,
)
```

***

### <kbd>method</kbd> `Api.automation`

```python theme={null}
automation(name: 'str', entity: 'str | None' = None) → Automation
```

Returns the only Automation matching the parameters.

**Args:**

* `name`:  The name of the automation to fetch.
* `entity`:  The entity to fetch the automation for.

**Raises:**

* `ValueError`:  If zero or multiple Automations match the search criteria.

**Examples:**
Get an existing automation named "my-automation":

```python theme={null}
import wandb

api = wandb.Api()
automation = api.automation(name="my-automation")
```

Get an existing automation named "other-automation", from the entity "my-team":

```python theme={null}
automation = api.automation(name="other-automation", entity="my-team")
```

***

### <kbd>method</kbd> `Api.automations`

```python theme={null}
automations(
    entity: 'str | None' = None,
    name: 'str | None' = None,
    per_page: 'int' = 50,
    start: 'str | None' = None
) → Iterator[Automation]
```

Returns an iterator over all Automations that match the given parameters.

If no parameters are provided, the returned iterator will contain all Automations that the user has access to.

**Args:**

* `entity`:  The entity to fetch the automations for.
* `name`:  The name of the automation to fetch.
* `per_page`:  The number of automations to fetch per page.  Defaults to 50.  Usually there is no reason to change this.
* `start`:  Pagination cursor for resuming a past query, captured  from a previous paginator's `.cursor` attribute.

**Returns:**
A list of automations.

**Examples:**
Fetch all existing automations for the entity "my-team":

```python theme={null}
import wandb

api = wandb.Api()
automations = api.automations(entity="my-team")
```

***

### <kbd>method</kbd> `Api.create_automation`

```python theme={null}
create_automation(
    obj: 'NewAutomation',
    fetch_existing: 'bool' = False,
    **kwargs: 'Unpack[WriteAutomationsKwargs]'
) → Automation
```

Create a new Automation.

**Args:**
obj:  The automation to create.  fetch\_existing:  If True, and a conflicting automation already exists, attempt  to fetch the existing automation instead of raising an error.  \*\*kwargs:  Any additional values to assign to the automation before  creating it.  If given, these will override any values that may  already be set on the automation:

* `name`: The name of the automation.
* `description`: The description of the automation.
* `enabled`: Whether the automation is enabled.
* `scope`: The scope of the automation.
* `event`: The event that triggers the automation.
* `action`: The action that is triggered by the automation.

**Returns:**
The saved Automation.

**Examples:**
Create a new automation named "my-automation" that sends a Slack notification when a run within a specific project logs a metric exceeding a custom threshold:

```python theme={null}
import wandb
from wandb.automations import OnRunMetric, RunEvent, SendNotification

api = wandb.Api()

project = api.project("my-project", entity="my-team")

# Use the first Slack integration for the team
slack_hook = next(api.slack_integrations(entity="my-team"))

event = OnRunMetric(
     scope=project,
     filter=RunEvent.metric("custom-metric") > 10,
)
action = SendNotification.from_integration(slack_hook)

automation = api.create_automation(
     event >> action,
     name="my-automation",
     description="Send a Slack message whenever 'custom-metric' exceeds 10.",
)
```

***

### <kbd>method</kbd> `Api.create_custom_chart`

```python theme={null}
create_custom_chart(
    entity: 'str',
    name: 'str',
    display_name: 'str',
    spec_type: "Literal['vega2']",
    access: "Literal['private', 'public']",
    spec: 'str | dict'
) → str
```

Create a custom chart preset and return its id.

**Args:**

* `entity`:  The entity (user or team) that owns the chart
* `name`:  Unique identifier for the chart preset
* `display_name`:  Human-readable name shown in the UI
* `spec_type`:  Type of specification. Must be "vega2" for Vega-Lite v2 specifications.
* `access`:  Access level for the chart:
  * "private": Chart is only accessible to the entity that created it
  * "public": Chart is publicly accessible
* `spec`:  The Vega/Vega-Lite specification as a dictionary or JSON string

**Returns:**
The ID of the created chart preset in the format "entity/name"

**Raises:**

* `wandb.Error`:  If chart creation fails
* `UnsupportedError`:  If the server doesn't support custom charts

**Example:**

```python theme={null}
   import wandb

   api = wandb.Api()

   # Define a simple bar chart specification
   vega_spec = {
        "$schema": "https://vega.github.io/schema/vega-lite/v6.json",
        "mark": "bar",
        "data": {"name": "wandb"},
        "encoding": {
            "x": {"field": "${field:x}", "type": "ordinal"},
            "y": {"field": "${field:y}", "type": "quantitative"},
        },
   }

   # Create the custom chart
   chart_id = api.create_custom_chart(
        entity="my-team",
        name="my-bar-chart",
        display_name="My Custom Bar Chart",
        spec_type="vega2",
        access="private",
        spec=vega_spec,
   )

   # Use with wandb.plot_table()
   chart = wandb.plot_table(
        vega_spec_name=chart_id,
        data_table=my_table,
        fields={"x": "category", "y": "value"},
   )
```

***

### <kbd>method</kbd> `Api.create_project`

```python theme={null}
create_project(name: 'str', entity: 'str') → None
```

Create a new project.

**Args:**

* `name`:  The name of the new project.
* `entity`:  The entity of the new project.

***

### <kbd>method</kbd> `Api.create_registry`

```python theme={null}
create_registry(
    name: 'str',
    visibility: "Literal['organization', 'restricted']",
    organization: 'str | None' = None,
    description: 'str | None' = None,
    artifact_types: 'list[str] | None' = None
) → Registry
```

Create a new registry.

**Args:**

* `name`:  The name of the registry. Name must be unique within the organization.
* `visibility`:  The visibility of the registry.
* `organization`:  Anyone in the organization can view this registry. You can  edit their roles later from the settings in the UI.
* `restricted`:  Only invited members via the UI can access this registry.  Public sharing is disabled.
* `organization`:  The organization of the registry.  If no organization is set in the settings, the organization will be  fetched from the entity if the entity only belongs to one organization.
* `description`:  The description of the registry.
* `artifact_types`:  The accepted artifact types of the registry. A type is no
* `more than 128 characters and do not include characters `/`or ``:`. If not specified, all types are accepted. Allowed types added to the registry cannot be removed later.

**Returns:**
A registry object.

**Examples:**

```python theme={null}
import wandb

api = wandb.Api()
registry = api.create_registry(
   name="my-registry",
   visibility="restricted",
   organization="my-org",
   description="This is a test registry",
   artifact_types=["model"],
)
```

***

### <kbd>method</kbd> `Api.create_run`

```python theme={null}
create_run(
    run_id: 'str | None' = None,
    project: 'str | None' = None,
    entity: 'str | None' = None
) → public.Run
```

Create a new run.

**Args:**

* `run_id`:  The ID to assign to the run. If not specified, W\&B  creates a random ID.
* `project`:  The project where to log the run to. If no project is specified,  log the run to a project called "Uncategorized".
* `entity`:  The entity that owns the project. If no entity is  specified, log the run to the default entity.

**Returns:**
The newly created `Run`.

***

### <kbd>method</kbd> `Api.create_run_queue`

```python theme={null}
create_run_queue(
    name: 'str',
    type: 'public.RunQueueResourceType',
    entity: 'str | None' = None,
    prioritization_mode: 'public.RunQueuePrioritizationMode | None' = None,
    config: 'dict | None' = None,
    template_variables: 'dict | None' = None
) → public.RunQueue
```

Create a new run queue in W\&B Launch.

**Args:**

* `name`:  Name of the queue to create
* `type`:  Type of resource to be used for the queue. One of  "local-container", "local-process", "kubernetes","sagemaker",  or "gcp-vertex".
* `entity`:  Name of the entity to create the queue. If `None`, use  the configured or default entity.
* `prioritization_mode`:  Version of prioritization to use.  Either "V0" or `None`.
* `config`:  Default resource configuration to be used for the queue.  Use handlebars (eg. `{{var}}`) to specify template variables.
* `template_variables`:  A dictionary of template variable schemas to  use with the config.

**Returns:**
The newly created `RunQueue`.

**Raises:**
`ValueError` if any of the parameters are invalid `wandb.Error` on wandb API errors

***

### <kbd>method</kbd> `Api.create_team`

```python theme={null}
create_team(team: 'str', admin_username: 'str | None' = None) → Team
```

Create a new team.

**Args:**

* `team`:  The name of the team
* `admin_username`:  Username of the admin user of the team.  Defaults to the current user.

**Returns:**
A `Team` object.

***

### <kbd>method</kbd> `Api.create_user`

```python theme={null}
create_user(email: 'str', admin: 'bool | None' = False) → User
```

Create a new user.

**Args:**

* `email`:  The email address of the user.
* `admin`:  Set user as a global instance administrator.

**Returns:**
A `User` object.

***

### <kbd>method</kbd> `Api.delete_automation`

```python theme={null}
delete_automation(obj: 'Automation | str') → Literal[True]
```

Delete an automation.

**Args:**

* `obj`:  The automation to delete, or its ID.

**Returns:**
True if the automation was deleted successfully.

***

### <kbd>method</kbd> `Api.flush`

```python theme={null}
flush()
```

Flush the local cache.

The api object keeps a local cache of runs, so if the state of the run may change while executing your script you must clear the local cache with `api.flush()` to get the latest values associated with the run.

***

### <kbd>method</kbd> `Api.from_path`

```python theme={null}
from_path(path: 'str')
```

Return a run, sweep, project or report from a path.

**Args:**

* `path`:  The path to the project, run, sweep or report

**Returns:**
A `Project`, `Run`, `Sweep`, or `BetaReport` instance.

**Raises:**
`wandb.Error` if path is invalid or the object doesn't exist.

**Examples:**
In the proceeding code snippets "project", "team", "run\_id", "sweep\_id", and "report\_name" are placeholders for the project, team, run ID, sweep ID, and the name of a specific report, respectively.

```python theme={null}
import wandb

api = wandb.Api()

project = api.from_path("project")
team_project = api.from_path("team/project")
run = api.from_path("team/project/runs/run_id")
sweep = api.from_path("team/project/sweeps/sweep_id")
report = api.from_path("team/project/reports/report_name")
```

***

### <kbd>method</kbd> `Api.integrations`

```python theme={null}
integrations(
    entity: 'str | None' = None,
    per_page: 'int' = 50,
    start: 'str | None' = None
) → Iterator[Integration]
```

Return an iterator of all integrations for an entity.

**Args:**

* `entity`:  The entity (e.g. team name) for which to  fetch integrations.  If not provided, the user's default entity  will be used.
* `per_page`:  Number of integrations to fetch per page.  Defaults to 50.  Usually there is no reason to change this.

**Yields:**

* `Iterator[SlackIntegration | WebhookIntegration]`:  An iterator of any supported integrations.

***

### <kbd>method</kbd> `Api.job`

```python theme={null}
job(name: 'str | None', path: 'str | None' = None) → public.Job
```

Return a `Job` object.

**Args:**

* `name`:  The name of the job.
* `path`:  The root path to download the job artifact.

**Returns:**
A `Job` object.

***

### <kbd>method</kbd> `Api.list_jobs`

```python theme={null}
list_jobs(entity: 'str', project: 'str') → list[dict[str, Any]]
```

Return a list of jobs, if any, for the given entity and project.

**Args:**

* `entity`:  The entity for the listed jobs.
* `project`:  The project for the listed jobs.

**Returns:**
A list of matching jobs.

***

### <kbd>method</kbd> `Api.project`

```python theme={null}
project(name: 'str', entity: 'str | None' = None) → public.Project
```

Return the `Project` with the given name (and entity, if given).

**Args:**

* `name`:  The project name.
* `entity`:  Name of the entity requested.  If None, will fall back to the  default entity passed to `Api`.  If no default entity, will  raise a `ValueError`.

**Returns:**
A `Project` object.

***

### <kbd>method</kbd> `Api.projects`

```python theme={null}
projects(entity: 'str | None' = None, per_page: 'int' = 200) → public.Projects
```

Get projects for a given entity.

**Args:**

* `entity`:  Name of the entity requested.  If None, will fall back to  the default entity passed to `Api`.  If no default entity,  will raise a `ValueError`.
* `per_page`:  Sets the page size for query pagination.  Usually there is no reason to change this.

**Returns:**
A `Projects` object which is an iterable collection of `Project`objects.

***

### <kbd>method</kbd> `Api.queued_run`

```python theme={null}
queued_run(
    entity: 'str',
    project: 'str',
    queue_name: 'str',
    run_queue_item_id: 'str',
    project_queue=None,
    priority=None
)
```

Return a single queued run based on the path.

Parses paths of the form `entity/project/queue_id/run_queue_item_id`.

***

### <kbd>method</kbd> `Api.registries`

```python theme={null}
registries(
    organization: 'str | None' = None,
    filter: 'dict[str, Any] | None' = None,
    per_page: 'int' = 100,
    start: 'str | None' = None
) → Registries
```

Returns a lazy iterator of `Registry` objects.

Use the iterator to search and filter registries, collections, or artifact versions across your organization's registry.

**Args:**

* `organization`:  (str, optional) The organization of the registry to fetch.  If not specified, use the organization specified in the user's settings.
* `filter`:  (dict, optional) MongoDB-style filter to apply to each object in the lazy registry iterator.  Fields available to filter for registries are  `name`, `description`, `created_at`, `updated_at`.  Fields available to filter for collections are  `name`, `tag`, `description`, `created_at`, `updated_at`  Fields available to filter for versions are  `tag`, `alias`, `created_at`, `updated_at`, `metadata`
* `per_page`:  Sets the page size for query pagination.
* `start`:  Pagination cursor for resuming a past query, captured  from a previous paginator's `.cursor` attribute.

**Returns:**
A lazy iterator of `Registry` objects.

**Examples:**
Find all registries with the names that contain "model"

```python theme={null}
import wandb

api = wandb.Api()  # specify an org if your entity belongs to multiple orgs
api.registries(filter={"name": {"$regex": "model"}})
```

Find all collections in the registries with the name "my\_collection" and the tag "my\_tag"

```python theme={null}
api.registries().collections(filter={"name": "my_collection", "tag": "my_tag"})
```

Find all artifact versions in the registries with a collection name that contains "my\_collection" and a version that has the alias "best"

```python theme={null}
api.registries().collections(
    filter={"name": {"$regex": "my_collection"}}
).versions(filter={"alias": "best"})
```

Find all artifact versions in the registries that contain "model" and have the tag "prod" or alias "best"

```python theme={null}
api.registries(filter={"name": {"$regex": "model"}}).versions(
    filter={"$or": [{"tag": "prod"}, {"alias": "best"}]}
)
```

Pause iteration and resume later from the same position by saving the paginator's `.cursor` and passing it as `start=`:

```python theme={null}
from itertools import islice

import wandb

api = wandb.Api()

# Consume the first page of results, then save the cursor.
page_size = 10
registries = api.registries(per_page=page_size)
first_page = list(islice(registries, page_size))

saved_cursor = registries.cursor

# Later (e.g. in a new process), resume iteration from the saved cursor.
remaining_registries = api.registries(per_page=page_size, start=saved_cursor)
```

***

### <kbd>method</kbd> `Api.registry`

```python theme={null}
registry(name: 'str', organization: 'str | None' = None) → Registry
```

Return a registry given a registry name.

**Args:**

* `name`:  The name of the registry. This is without the `wandb-registry-`  prefix.
* `organization`:  The organization of the registry.  If no organization is set in the settings, the organization will be  fetched from the entity if the entity only belongs to one  organization.

**Returns:**
A registry object.

**Examples:**
Fetch and update a registry

```python theme={null}
import wandb

api = wandb.Api()
registry = api.registry(name="my-registry", organization="my-org")
registry.description = "This is an updated description"
registry.save()
```

***

### <kbd>method</kbd> `Api.reports`

```python theme={null}
reports(
    path: 'str' = '',
    name: 'str | None' = None,
    per_page: 'int' = 50
) → public.Reports
```

Get reports for a given project path.

Note: `wandb.Api.reports()` API is in beta and will likely change in future releases.

**Args:**

* `path`:  The path to the project the report resides in. Specify the  entity that created the project as a prefix followed by a  forward slash.
* `name`:  Name of the report requested.
* `per_page`:  Sets the page size for query pagination.  Usually there is no reason to change this.

**Returns:**
A `Reports` object which is an iterable collection of  `BetaReport` objects.

**Examples:**

```python theme={null}
import wandb

wandb.Api.reports("entity/project")
```

***

### <kbd>method</kbd> `Api.run`

```python theme={null}
run(path='')
```

Return a single run by parsing path in the form `entity/project/run_id`.

**Args:**

* `path`:  Path to run in the form `entity/project/run_id`.  If `api.entity` is set, this can be in the form `project/run_id`  and if `api.project` is set this can just be the run\_id.

**Returns:**
A `Run` object.

***

### <kbd>method</kbd> `Api.run_queue`

```python theme={null}
run_queue(entity: 'str', name: 'str')
```

Return the named `RunQueue` for entity.

See `Api.create_run_queue` for more information on how to create a run queue.

***

### <kbd>method</kbd> `Api.runs`

```python theme={null}
runs(
    path: 'str | None' = None,
    filters: 'dict[str, Any] | None' = None,
    order: 'str' = '+created_at',
    per_page: 'int' = 50,
    include_sweeps: 'bool' = True,
    lazy: 'bool' = True
)
```

Returns a `Runs` object, which lazily iterates over `Run` objects.

Fields you can filter by include:

* `createdAt`: The timestamp when the run was created. (in ISO 8601 format, e.g. "2023-01-01T12:00:00Z")
* `displayName`: The human-readable display name of the run. (e.g. "eager-fox-1")
* `duration`: The total runtime of the run in seconds.
* `group`: The group name used to organize related runs together.
* `host`: The hostname where the run was executed.
* `jobType`: The type of job or purpose of the run.
* `name`: The unique identifier of the run. (e.g. "a1b2cdef")
* `state`: The current state of the run.
* `tags`: The tags associated with the run.
* `username`: The username of the user who initiated the run

Additionally, you can filter by items in the run config or summary metrics. Such as `config.experiment_name`, `summary_metrics.loss`, etc.

For more complex filtering, you can use MongoDB query operators. For details, see: [https://docs.mongodb.com/manual/reference/operator/query](https://docs.mongodb.com/manual/reference/operator/query) The following operations are supported:

* `$and`
* `$or`
* `$nor`
* `$eq`
* `$ne`
* `$gt`
* `$gte`
* `$lt`
* `$lte`
* `$in`
* `$nin`
* `$exists`
* `$regex`

**Args:**

* `path`:  (str) path to project, should be in the form: "entity/project"
* `filters`:  (dict) queries for specific runs using the MongoDB query language.  You can filter by run properties such as config.key, summary\_metrics.key, state, entity, createdAt, etc.
* `For example`:  `{"config.experiment_name": "foo"}` would find runs with a config entry  of experiment name set to "foo"
* `order`:  (str) Order can be `created_at`, `heartbeat_at`, `config.*.value`, or `summary_metrics.*`.  If you prepend order with a + order is ascending (default).  If you prepend order with a - order is descending.  The default order is run.created\_at from oldest to newest.
* `per_page`:  (int) Sets the page size for query pagination.
* `include_sweeps`:  (bool) Whether to include the sweep runs in the results.
* `lazy`:  (bool) Whether to use lazy loading for faster performance.  When True (default), only essential run metadata is loaded initially.  Heavy fields like config, summaryMetrics, and systemMetrics are loaded  on-demand when accessed. Set to False for full data upfront.

**Returns:**
A `Runs` object, which is an iterable collection of `Run` objects.

**Examples:**

```python theme={null}
import wandb
from wandb.apis.public import Api

# Find runs in project where config.experiment_name has been set to "foo"
Api.runs(path="my_entity/project", filters={"config.experiment_name": "foo"})
```

```python theme={null}
# Find runs in project where config.experiment_name has been set to "foo" or "bar"
Api.runs(
    path="my_entity/project",
    filters={
         "$or": [
             {"config.experiment_name": "foo"},
             {"config.experiment_name": "bar"},
         ]
    },
)
```

```python theme={null}
# Find runs in project where config.experiment_name matches a regex
# (anchors are not supported)
Api.runs(
    path="my_entity/project",
    filters={"config.experiment_name": {"$regex": "b.*"}},
)
```

```python theme={null}
# Find runs in project where the run name matches a regex
# (anchors are not supported)
Api.runs(
    path="my_entity/project", filters={"display_name": {"$regex": "^foo.*"}}
)
```

```python theme={null}
# Find runs in project sorted by ascending loss
Api.runs(path="my_entity/project", order="+summary_metrics.loss")
```

***

### <kbd>method</kbd> `Api.slack_integrations`

```python theme={null}
slack_integrations(
    entity: 'str | None' = None,
    per_page: 'int' = 50,
    start: 'str | None' = None
) → Iterator[SlackIntegration]
```

Returns an iterator of Slack integrations for an entity.

**Args:**

* `entity`:  The entity (e.g. team name) for which to  fetch integrations.  If not provided, the user's default entity  will be used.
* `per_page`:  Number of integrations to fetch per page.  Defaults to 50.  Usually there is no reason to change this.

**Yields:**

* `Iterator[SlackIntegration]`:  An iterator of Slack integrations.

**Examples:**
Get all registered Slack integrations for the team "my-team":

```python theme={null}
import wandb

api = wandb.Api()
slack_integrations = api.slack_integrations(entity="my-team")
```

Find only Slack integrations that post to channel names starting with "team-alerts-":

```python theme={null}
slack_integrations = api.slack_integrations(entity="my-team")
team_alert_integrations = [
    ig
    for ig in slack_integrations
    if ig.channel_name.startswith("team-alerts-")
]
```

***

### <kbd>method</kbd> `Api.sweep`

```python theme={null}
sweep(path='')
```

Return a sweep by parsing path in the form `entity/project/sweep_id`.

**Args:**

* `path`:  Path to sweep in the form entity/project/sweep\_id.  If `api.entity` is set, this can be in the form  project/sweep\_id and if `api.project` is set  this can just be the sweep\_id.

**Returns:**
A `Sweep` object.

***

### <kbd>method</kbd> `Api.sync_tensorboard`

```python theme={null}
sync_tensorboard(root_dir, run_id=None, project=None, entity=None)
```

Sync a local directory containing tfevent files to wandb.

***

### <kbd>method</kbd> `Api.team`

```python theme={null}
team(team: 'str') → Team
```

Return the matching `Team` with the given name.

**Args:**

* `team`:  The name of the team.

**Returns:**
A `Team` object.

***

### <kbd>method</kbd> `Api.update_automation`

```python theme={null}
update_automation(
    obj: 'Automation',
    create_missing: 'bool' = False,
    **kwargs: 'Unpack[WriteAutomationsKwargs]'
) → Automation
```

Update an existing automation.

**Args:**

* `obj`:  The automation to update.  Must be an existing automation. create\_missing (bool):  If True, and the automation does not exist, create it. \*\*kwargs:  Any additional values to assign to the automation before  updating it.  If given, these will override any values that may  already be set on the automation:
  * `name`: The name of the automation.
  * `description`: The description of the automation.
  * `enabled`: Whether the automation is enabled.
  * `scope`: The scope of the automation.
  * `event`: The event that triggers the automation.
  * `action`: The action that is triggered by the automation.

**Returns:**
The updated automation.

**Examples:**
Disable and edit the description of an existing automation ("my-automation"):

```python theme={null}
import wandb

api = wandb.Api()

automation = api.automation(name="my-automation")
automation.enabled = False
automation.description = "Kept for reference, but no longer used."

updated_automation = api.update_automation(automation)
```

OR

```python theme={null}
import wandb

api = wandb.Api()

automation = api.automation(name="my-automation")

updated_automation = api.update_automation(
    automation,
    enabled=False,
    description="Kept for reference, but no longer used.",
)
```

***

### <kbd>method</kbd> `Api.upsert_run_queue`

```python theme={null}
upsert_run_queue(
    name: 'str',
    resource_config: 'dict',
    resource_type: 'public.RunQueueResourceType',
    entity: 'str | None' = None,
    template_variables: 'dict | None' = None,
    external_links: 'dict | None' = None,
    prioritization_mode: 'public.RunQueuePrioritizationMode | None' = None
)
```

Upsert a run queue in W\&B Launch.

**Args:**

* `name`:  Name of the queue to create
* `entity`:  Optional name of the entity to create the queue. If `None`,  use the configured or default entity.
* `resource_config`:  Optional default resource configuration to be used  for the queue. Use handlebars (eg. `{{var}}`) to specify  template variables.
* `resource_type`:  Type of resource to be used for the queue. One of  "local-container", "local-process", "kubernetes", "sagemaker",  or "gcp-vertex".
* `template_variables`:  A dictionary of template variable schemas to  be used with the config.
* `external_links`:  Optional dictionary of external links to be used  with the queue.
* `prioritization_mode`:  Optional version of prioritization to use.  Either "V0" or None

**Returns:**
The upserted `RunQueue`.

**Raises:**
ValueError if any of the parameters are invalid wandb.Error on wandb API errors

***

### <kbd>method</kbd> `Api.user`

```python theme={null}
user(username_or_email: 'str') → User | None
```

Return a user from a username or email address.

This function only works for local administrators. Use `api.viewer`  to get your own user object.

**Args:**

* `username_or_email`:  The username or email address of the user.

**Returns:**
A `User` object or None if a user is not found.

***

### <kbd>method</kbd> `Api.users`

```python theme={null}
users(username_or_email: 'str') → list[User]
```

Return all users from a partial username or email address query.

This function only works for local administrators. Use `api.viewer`  to get your own user object.

**Args:**

* `username_or_email`:  The prefix or suffix of the user you want to find.

**Returns:**
An array of `User` objects.

***

### <kbd>method</kbd> `Api.webhook_integrations`

```python theme={null}
webhook_integrations(
    entity: 'str | None' = None,
    per_page: 'int' = 50,
    start: 'str | None' = None
) → Iterator[WebhookIntegration]
```

Returns an iterator of webhook integrations for an entity.

**Args:**

* `entity`:  The entity (e.g. team name) for which to  fetch integrations.  If not provided, the user's default entity  will be used.
* `per_page`:  Number of integrations to fetch per page.  Defaults to 50.  Usually there is no reason to change this.

**Yields:**

* `Iterator[WebhookIntegration]`:  An iterator of webhook integrations.

**Examples:**
Get all registered webhook integrations for the team "my-team":

```python theme={null}
import wandb

api = wandb.Api()
webhook_integrations = api.webhook_integrations(entity="my-team")
```

Find only webhook integrations that post requests to "[https://my-fake-url.com](https://my-fake-url.com)":

```python theme={null}
webhook_integrations = api.webhook_integrations(entity="my-team")
my_webhooks = [
    ig
    for ig in webhook_integrations
    if ig.url_endpoint.startswith("https://my-fake-url.com")
]
```


# ArtifactCollection
Source: https://docs.wandb.ai/models/ref/python/public-api/artifactcollection



<GitHubLink />

## <kbd>class</kbd> `ArtifactCollection`

An artifact collection that represents a group of related artifacts.

**Args:**

* `service_api`:  Interface to the wandb-core service that performs  W\&B API calls for this collection.
* `entity`:  The entity (user or team) that owns the project.
* `project`:  The name of the project to query for artifact collections.
* `name`:  The name of the artifact collection.
* `type`:  The type of the artifact collection (e.g., "dataset", "model").
* `organization`:  Optional organization name if applicable.
* `attrs`:  Optional mapping of attributes to initialize the artifact collection.  If not provided, the object will load its attributes from W\&B upon  initialization.

### <kbd>property</kbd> ArtifactCollection.aliases

The aliases for all artifact versions contained in this collection.

**Returns:**

* `list[str]`: The aliases property value.

***

### <kbd>property</kbd> ArtifactCollection.created\_at

The creation date of the artifact collection.

**Returns:**

* `str`: The created\_at property value.

***

### <kbd>property</kbd> ArtifactCollection.description

A description of the artifact collection.

**Returns:**

* `str | None`: The description property value.

***

### <kbd>property</kbd> ArtifactCollection.entity

The entity (user or team) that owns the project.

**Returns:**

* `str`: The entity property value.

***

### <kbd>property</kbd> ArtifactCollection.id

The unique identifier of the artifact collection.

**Returns:**

* `str`: The id property value.

***

### <kbd>property</kbd> ArtifactCollection.name

The name of the artifact collection.

**Returns:**

* `str`: The name property value.

***

### <kbd>property</kbd> ArtifactCollection.project

The project that contains the artifact collection.

**Returns:**

* `str`: The project property value.

***

### <kbd>property</kbd> ArtifactCollection.tags

The tags associated with the artifact collection.

**Returns:**

* `list[str]`: The tags property value.

***

### <kbd>property</kbd> ArtifactCollection.type

Returns the type of the artifact collection.

***

### <kbd>property</kbd> ArtifactCollection.updated\_at

The date at which the artifact collection was last updated.

**Returns:**

* `str | None`: The updated\_at property value.

***

### <kbd>method</kbd> `ArtifactCollection.artifacts`

```python theme={null}
artifacts(per_page: 'int' = 50, start: 'str | None' = None) → Artifacts
```

Get all artifacts in the collection.

**Args:**

* `per_page`:  The number of artifacts to fetch per page.
* `start`:  Pagination cursor for resuming a past query, captured  from a previous paginator's `.cursor` attribute.

***

### <kbd>method</kbd> `ArtifactCollection.change_type`

```python theme={null}
change_type(new_type: 'str') → None
```

Deprecated, change type directly with `save` instead.

***

### <kbd>method</kbd> `ArtifactCollection.delete`

```python theme={null}
delete() → None
```

Delete the entire artifact collection.

***

### <kbd>method</kbd> `ArtifactCollection.is_sequence`

```python theme={null}
is_sequence() → bool
```

Return whether the artifact collection is a sequence.

***

### <kbd>method</kbd> `ArtifactCollection.save`

```python theme={null}
save() → None
```

Persist any changes made to the artifact collection.


# ArtifactCollections
Source: https://docs.wandb.ai/models/ref/python/public-api/artifactcollections



<GitHubLink />

## <kbd>class</kbd> `ArtifactCollections`

Artifact collections of a specific type in a project.

**Args:**

* `service_api`:  Interface to the wandb-core service that performs  W\&B API calls for this collection.
* `entity`:  The entity (user or team) that owns the project.
* `project`:  The name of the project to query for artifact collections.
* `type_name`:  The name of the artifact type for which to fetch collections.
* `filters`:  Optional mapping of filters to apply to the query.
* `order`:  Optional string to specify the order of the results.  If you prepend order with a + order is ascending (default).  If you prepend order with a - order is descending.
* `per_page`:  The number of artifact collections to fetch per page. Default is 50.

### <kbd>property</kbd> ArtifactCollections.cursor

An opaque cursor that marks the start of the next page to fetch.

This value may be saved and passed as `start=` to a later paginated query to resume iteration from where this paginator left off.

**Returns:**

* `str | None`: The cursor property value.

***

### <kbd>property</kbd> ArtifactCollections.more


# ArtifactFiles
Source: https://docs.wandb.ai/models/ref/python/public-api/artifactfiles



<GitHubLink />

## <kbd>class</kbd> `ArtifactFiles`

A paginator for files in an artifact.

### <kbd>property</kbd> ArtifactFiles.cursor

An opaque cursor that marks the start of the next page to fetch.

This value may be saved and passed as `start=` to a later paginated query to resume iteration from where this paginator left off.

**Returns:**

* `str | None`: The cursor property value.

***

### <kbd>property</kbd> ArtifactFiles.more

***

### <kbd>property</kbd> ArtifactFiles.path

Returns the path of the artifact.

**Returns:**

* `list[str]`: The path property value.


# Artifacts
Source: https://docs.wandb.ai/models/ref/python/public-api/artifacts



<GitHubLink />

## <kbd>class</kbd> `Artifacts`

An iterable collection of artifact versions associated with a project.

Optionally pass in filters to narrow down the results based on specific criteria.

**Args:**

* `service_api`:  Interface to the wandb-core service that performs  W\&B API calls for these artifact versions.
* `entity`:  The entity (user or team) that owns the project.
* `project`:  The name of the project to query for artifacts.
* `collection_name`:  The name of the artifact collection to query.
* `type`:  The type of the artifacts to query. Common examples include  "dataset" or "model".
* `filters`:  Optional mapping of filters to apply to the query.
* `order`:  Optional string to specify the order of the results.
* `per_page`:  The number of artifact versions to fetch per page. Default is 50.
* `tags`:  Optional string or list of strings to filter artifacts by tags.

### <kbd>property</kbd> Artifacts.cursor

An opaque cursor that marks the start of the next page to fetch.

This value may be saved and passed as `start=` to a later paginated query to resume iteration from where this paginator left off.

**Returns:**

* `str | None`: The cursor property value.

***

### <kbd>property</kbd> Artifacts.more

***


# ArtifactType
Source: https://docs.wandb.ai/models/ref/python/public-api/artifacttype



<GitHubLink />

## <kbd>class</kbd> `ArtifactType`

An artifact object that satisfies query based on the specified type.

**Args:**

* `service_api`:  Interface to the wandb-core service that performs  W\&B API calls for this artifact type.
* `entity`:  The entity (user or team) that owns the project.
* `project`:  The name of the project to query for artifact types.
* `type_name`:  The name of the artifact type.
* `attrs`:  Optional attributes to initialize the ArtifactType.  If omitted, the object will load its attributes from W\&B upon  initialization.

### <kbd>property</kbd> ArtifactType.id

The unique identifier of the artifact type.

**Returns:**

* `str`: The id property value.

***

### <kbd>property</kbd> ArtifactType.name

The name of the artifact type.

**Returns:**

* `str`: The name property value.

***

### <kbd>method</kbd> `ArtifactType.collection`

```python theme={null}
collection(name: 'str') → ArtifactCollection
```

Get a specific artifact collection by name.

**Args:**

* `name` (str):  The name of the artifact collection to retrieve.

***

### <kbd>method</kbd> `ArtifactType.collections`

```python theme={null}
collections(
    filters: 'Mapping[str, Any] | None' = None,
    order: 'str | None' = None,
    per_page: 'int' = 50,
    start: 'str | None' = None
) → ArtifactCollections
```

Get all artifact collections associated with this artifact type.

**Args:**

* `filters` (dict):  Optional mapping of filters to apply to the query.
* `order` (str):  Optional string to specify the order of the results.  If you prepend order with a + order is ascending (default).  If you prepend order with a - order is descending.  The default order is the collection ID in descending order.
* `per_page` (int):  The number of artifact collections to fetch per page.  Default is 50.
* `start`:  Pagination cursor for resuming a past query, captured  from a previous paginator's `.cursor` attribute.

***


# ArtifactTypes
Source: https://docs.wandb.ai/models/ref/python/public-api/artifacttypes



<GitHubLink />

## <kbd>class</kbd> `ArtifactTypes`

An lazy iterator of `ArtifactType` objects for a specific project.

### <kbd>property</kbd> ArtifactTypes.cursor

An opaque cursor that marks the start of the next page to fetch.

This value may be saved and passed as `start=` to a later paginated query to resume iteration from where this paginator left off.

**Returns:**

* `str | None`: The cursor property value.

***

### <kbd>property</kbd> ArtifactTypes.more


# Automations
Source: https://docs.wandb.ai/models/ref/python/public-api/automations



<GitHubLink />


# BetaReport
Source: https://docs.wandb.ai/models/ref/python/public-api/betareport



<GitHubLink />

## <kbd>class</kbd> `BetaReport`

BetaReport is a class associated with reports created in W\&B.

Provides access to report attributes (name, description, user, spec, timestamps) and methods for retrieving associated runs, sections, and for rendering the report as HTML.

**Attributes:**

* `id` (string):  Unique identifier of the report.
* `display_name` (string):  Human-readable display name of the report.
* `name` (string):  The name of the report. Use `display_name` for a more user-friendly name.
* `description` (string):  Description of the report.
* `user` (User):  Dictionary containing user info (username, email) who  created the report.
* `spec` (dict):  The spec of the report.
* `url` (string):  The URL of the report.
* `updated_at` (string):  Timestamp of last update.
* `created_at` (string):  Timestamp when the report was created.

### <kbd>method</kbd> `BetaReport.__init__`

```python theme={null}
__init__(
    service_api: 'ServiceApi',
    attrs: 'dict',
    entity: 'str | None' = None,
    project: 'str | None' = None
)
```

***

### <kbd>property</kbd> BetaReport.created\_at

***

### <kbd>property</kbd> BetaReport.description

***

### <kbd>property</kbd> BetaReport.display\_name

***

### <kbd>property</kbd> BetaReport.id

***

### <kbd>property</kbd> BetaReport.name

***

### <kbd>property</kbd> BetaReport.sections

Get the panel sections (groups) from the report.

***

### <kbd>property</kbd> BetaReport.spec

***

### <kbd>property</kbd> BetaReport.updated\_at

***

### <kbd>property</kbd> BetaReport.url

***

### <kbd>property</kbd> BetaReport.user

***

### <kbd>method</kbd> `BetaReport.runs`

```python theme={null}
runs(
    section: 'dict[str, Any]',
    per_page: 'int' = 50,
    only_selected: 'bool' = True
) → public.Runs
```

Get runs associated with a section of the report.

***

### <kbd>method</kbd> `BetaReport.to_html`

```python theme={null}
to_html(height: 'int' = 1024, hidden: 'bool' = False) → str
```

Generate HTML containing an iframe displaying this report.


# DownloadHistoryResult
Source: https://docs.wandb.ai/models/ref/python/public-api/downloadhistoryresult



<GitHubLink />

## <kbd>class</kbd> `DownloadHistoryResult`

Result of downloading a run's history exports.

**Attributes:**

* `paths`:  The paths to the downloaded history files.
* `errors`:  A dictionary mapping file paths to error messages for files that  failed to download. None if all downloads succeeded.
* `contains_live_data`:  Whether the run contains live data,  not yet exported to parquet files.

### <kbd>method</kbd> `DownloadHistoryResult.__init__`

```python theme={null}
__init__(
    paths: 'list[pathlib.Path]',
    contains_live_data: 'bool',
    errors: 'dict[pathlib.Path, str] | None' = None
) → None
```


# File
Source: https://docs.wandb.ai/models/ref/python/public-api/file



<GitHubLink />

## <kbd>class</kbd> `File`

File saved to W\&B.

Represents a single file stored in W\&B. Includes access to file metadata. Files are associated with a specific run and can include text files, model weights, datasets, visualizations, and other artifacts. You can download the file, delete the file, and access file properties.

Specify one or more attributes in a dictionary to fine a specific file logged to a specific run. You can search using the following keys:

* id (str): The ID of the run that contains the file
* name (str): Name of the file
* url (str): path to file
* direct\_url (str): path to file in the bucket
* sizeBytes (int): size of file in bytes
* md5 (str): md5 of file
* mimetype (str): mimetype of file
* updated\_at (str): timestamp of last update
* path\_uri (str): path to file in the bucket, currently only available for S3 objects and reference files

**Args:**

* `service_api`:  The service API instance to use for querying W\&B.
* `attrs` (dict):  A dictionary of attributes that define the file
* `run`:  The run object that contains the file

### <kbd>property</kbd> File.path\_uri

Returns the URI path to the file in the storage bucket.

**Returns:**

* `str`:  The S3 URI (e.g., 's3://bucket/path/to/file') if the file is stored in S3,  the direct URL if it's a reference file, or an empty string if unavailable.

**Returns:**

* `str`: The path\_uri property value.

***

### <kbd>property</kbd> File.size

Returns the size of the file in bytes.

**Returns:**

* `int`: The size property value.

***

### <kbd>method</kbd> `File.delete`

```python theme={null}
delete() → None
```

Delete the file from the W\&B server.

***

### <kbd>method</kbd> `File.download`

```python theme={null}
download(
    root: 'str' = '.',
    replace: 'bool' = False,
    exist_ok: 'bool' = False,
    api: 'Api | None' = None
) → io.TextIOWrapper
```

Downloads a file previously saved by a run from the wandb server.

**Args:**

* `root`:  Local directory to save the file. Defaults to the  current working directory (".").
* `replace`:  If `True`, download will overwrite a local file  if it exists. Defaults to `False`.
* `exist_ok`:  If `True`, will not raise ValueError if file already  exists and will not re-download unless replace=True.  Defaults to `False`.
* `api`:  If specified, the `Api` instance used to download the file.

**Raises:**
`ValueError` if file already exists, `replace=False` and `exist_ok=False`.


# Files
Source: https://docs.wandb.ai/models/ref/python/public-api/files



<GitHubLink />

## <kbd>class</kbd> `Files`

A lazy iterator over a collection of `File` objects.

Access and manage files uploaded to W\&B during a run. Handles pagination automatically when iterating through large collections of files.

**Example:**

```python theme={null}
from wandb.apis.public.files import Files
from wandb.apis.public.api import Api

# Example run object
run = Api().run("entity/project/run-id")

# Get the files for the run
files = run.files()

# Iterate over files
for file in files:
    print(file.name)
    print(file.url)
    print(file.size)

    # Download the file
    file.download(root="download_directory", replace=True)
```

### <kbd>method</kbd> `Files.__init__`

```python theme={null}
__init__(
    service_api: 'ServiceApi',
    run: 'Run',
    names: 'list[str] | None' = None,
    per_page: 'int' = 50,
    upload: 'bool' = False,
    pattern: 'str | None' = None
)
```

Initialize a lazy iterator over a collection of `File` objects.

Files are retrieved in pages from the W\&B server as needed.

**Args:**

* `service_api`:  The service API instance to use for querying W\&B.
* `run`:  The run object that contains the files
* `names` (list, optional):  A list of file names to filter the files
* `per_page` (int, optional):  The number of files to fetch per page
* `upload` (bool, optional):  If `True`, fetch the upload URL for each file
* `pattern` (str, optional):  Pattern to match when returning files from W\&B  This pattern uses mySQL's LIKE syntax,  so matching all files that end with .json would be "%.json".  If both names and pattern are provided, a ValueError will be raised.

***

### <kbd>property</kbd> Files.length

***


# IncompleteRunHistoryError
Source: https://docs.wandb.ai/models/ref/python/public-api/incompleterunhistoryerror



<GitHubLink />

## <kbd>class</kbd> `IncompleteRunHistoryError`

Raised when run history has incomplete history.

Incomplete history occurs when some data has not been exported to parquet files yet, typically because the run is still ongoing.


# Member
Source: https://docs.wandb.ai/models/ref/python/public-api/member



<GitHubLink />

## <kbd>class</kbd> `Member`

A member of a team.

### <kbd>method</kbd> `Member.__init__`

```python theme={null}
__init__(service_api: 'ServiceApi', team: 'str', attrs: 'Mapping[str, Any]')
```

**Args:**

* `service_api`:  The service API instance to use for querying W\&B.
* `team` (str):  The name of the team this member belongs to
* `attrs` (dict):  The member attributes

***

### <kbd>method</kbd> `Member.delete`

```python theme={null}
delete()
```

Remove a member from a team.

**Returns:**
Boolean indicating success


# Project
Source: https://docs.wandb.ai/models/ref/python/public-api/project



<GitHubLink />

## <kbd>class</kbd> `Project`

A project is a namespace for runs.

### <kbd>method</kbd> `Project.__init__`

```python theme={null}
__init__(
    service_api: 'ServiceApi',
    entity: 'str',
    project: 'str',
    attrs: 'Mapping[str, Any]'
) → Project
```

**Args:**

* `service_api`:  Interface to the wandb-core service that performs  W\&B API calls for this project.
* `name` (str):  The name of the project.
* `entity` (str):  The entity name that owns the project.

A single project associated with an entity.

**Args:**

* `service_api`:  The service API used to query W\&B.
* `entity`:  The entity which owns the project.
* `project`:  The name of the project to query.
* `attrs`:  The attributes of the project.

***

### <kbd>property</kbd> Project.id

***

### <kbd>property</kbd> Project.owner

Returns the project owner as a User object.

**Raises:**

* `ValueError`:  when no user information is found for the project.

**Returns:**

* `public.User`: The owner property value.

***

### <kbd>property</kbd> Project.path

Returns the path of the project. The path is a list containing the entity and project name.

**Returns:**

* `list[str]`: The path property value.

***

### <kbd>property</kbd> Project.url

Returns the URL of the project.

**Returns:**

* `str`: The url property value.

***

### <kbd>method</kbd> `Project.artifacts_types`

```python theme={null}
artifacts_types(per_page: 'int' = 50) → public.ArtifactTypes
```

Returns all artifact types associated with this project.

***

### <kbd>method</kbd> `Project.collections`

```python theme={null}
collections(
    filters: 'Mapping[str, Any] | None' = None,
    order: 'str | None' = None,
    per_page: 'int' = 50
) → public.ProjectArtifactCollections
```

Returns all artifact collections associated with this project.

**Args:**

* `filters`:  Optional mapping of filters to apply to the query.
* `order`:  Optional string to specify the order of the results.  If you prepend order with a + order is ascending (default).  If you prepend order with a - order is descending.
* `per_page`:  The number of artifact collections to fetch per page.  Default is 50.

***

### <kbd>method</kbd> `Project.sweeps`

```python theme={null}
sweeps(per_page: 'int' = 50) → Sweeps
```

Return a paginated collection of sweeps in this project.

**Args:**

* `per_page`:  The number of sweeps to fetch per request to the API.

**Returns:**
A `Sweeps` object, which is an iterable collection of `Sweep` objects.

***


# ProjectArtifactCollections
Source: https://docs.wandb.ai/models/ref/python/public-api/projectartifactcollections



<GitHubLink />

## <kbd>class</kbd> `ProjectArtifactCollections`

Artifact collections in a project.

**Args:**

* `service_api`:  Interface to the wandb-core service that performs  W\&B API calls for this collection.
* `entity`:  The entity (user or team) that owns the project.
* `project`:  The name of the project to query for artifact collections.
* `filters`:  Optional mapping of filters to apply to the query.
* `order`:  Optional string to specify the order of the results.  If you prepend order with a + order is ascending (default).  If you prepend order with a - order is descending.
* `per_page`:  The number of artifact collections to fetch per page. Default is 50.

### <kbd>property</kbd> ProjectArtifactCollections.cursor

An opaque cursor that marks the start of the next page to fetch.

This value may be saved and passed as `start=` to a later paginated query to resume iteration from where this paginator left off.

**Returns:**

* `str | None`: The cursor property value.

***

### <kbd>property</kbd> ProjectArtifactCollections.more


# Projects
Source: https://docs.wandb.ai/models/ref/python/public-api/projects



<GitHubLink />

## <kbd>class</kbd> `Projects`

An lazy iterator of `Project` objects.

An iterable interface to access projects created and saved by the entity.

### <kbd>method</kbd> `Projects.__init__`

```python theme={null}
__init__(
    service_api: 'ServiceApi',
    entity: 'str',
    per_page: 'int' = 50
) → Projects
```

**Args:**

* `service_api`:  Interface to the wandb-core service that performs  W\&B API calls for this collection.
* `entity` (str):  The entity name (username or team) to fetch projects for.
* `per_page` (int):  Number of projects to fetch per request (default is 50).

**Example:**

```python theme={null}
from wandb.apis.public.api import Api

# Find projects that belong to this entity
projects = Api().projects(entity="entity")

# Iterate over files
for project in projects:
   print(f"Project: {project.name}")
   print(f"- URL: {project.url}")
   print(f"- Created at: {project.created_at}")
   print(f"- Is benchmark: {project.is_benchmark}")
```

An iterable collection of `Project` objects.

**Args:**

* `service_api`:  The service API used to query W\&B.
* `entity`:  The entity which owns the projects.
* `per_page`:  The number of projects to fetch per request to the API.

***

### <kbd>property</kbd> Projects.cursor

An opaque cursor that marks the start of the next page to fetch.

This value may be saved and passed as `start=` to a later paginated query to resume iteration from where this paginator left off.

**Returns:**

* `str | None`: The cursor property value.

***

### <kbd>property</kbd> Projects.more


# Registry
Source: https://docs.wandb.ai/models/ref/python/public-api/registry



<GitHubLink />

## <kbd>class</kbd> `Registry`

A single registry in the Registry.

### <kbd>method</kbd> `Registry.__init__`

```python theme={null}
__init__(
    service_api: 'ServiceApi',
    organization: 'str',
    entity: 'str',
    name: 'str',
    attrs: 'RegistryFragment | None' = None
)
```

***

### <kbd>property</kbd> Registry.allow\_all\_artifact\_types

Return whether all artifact types are allowed in the registry.

If `True`, artifacts of any type can be added. If `False`, artifacts are restricted to the types listed in `artifact_types`.

**Returns:**

* `bool`: The allow\_all\_artifact\_types property value.

***

### <kbd>property</kbd> Registry.artifact\_types

Returns the artifact types allowed in the registry.

If `allow_all_artifact_types` is `True` then `artifact_types` reflects the types previously saved or currently used in the registry. If `allow_all_artifact_types` is `False` then artifacts are restricted to the types in `artifact_types`.

**Note:**

> Previously saved artifact types cannot be removed.

**Example:**

```python theme={null}
import wandb

registry = wandb.Api().create_registry()
registry.artifact_types.append("model")
registry.save()  # once saved, the artifact type `model` cannot be removed
registry.artifact_types.append("accidentally_added")
registry.artifact_types.remove(
    "accidentally_added"
)  # Types can only be removed if it has not been saved yet
```

**Returns:**

* `AddOnlyArtifactTypesList`: The artifact\_types property value.

***

### <kbd>property</kbd> Registry.created\_at

Timestamp of when the registry was created.

**Returns:**

* `str`: The created\_at property value.

***

### <kbd>property</kbd> Registry.description

Description of the registry.

**Returns:**

* `str | None`: The description property value.

***

### <kbd>property</kbd> Registry.entity

Organization entity of the registry.

**Returns:**

* `str`: The entity property value.

***

### <kbd>property</kbd> Registry.full\_name

Full name of the registry including the `wandb-registry-` prefix.

**Returns:**

* `str`: The full\_name property value.

***

### <kbd>property</kbd> Registry.id

The unique ID for this registry.

**Returns:**

* `str`: The id property value.

***

### <kbd>property</kbd> Registry.name

Name of the registry without the `wandb-registry-` prefix.

**Returns:**

* `str`: The name property value.

***

### <kbd>property</kbd> Registry.organization

Organization name of the registry.

**Returns:**

* `str`: The organization property value.

***

### <kbd>property</kbd> Registry.path

***

### <kbd>property</kbd> Registry.updated\_at

Timestamp of when the registry was last updated.

**Returns:**

* `str`: The updated\_at property value.

***

### <kbd>property</kbd> Registry.visibility

Visibility of the registry.

**Returns:**

* `Literal["organization", "restricted"]`:  The visibility level.
  * "organization": Anyone in the organization can view this registry.  You can edit their roles later from the settings in the UI.
  * "restricted": Only invited members via the UI can access this registry.  Public sharing is disabled.

**Returns:**

* `Literal['organization', 'restricted']`: The visibility property value.

***

### <kbd>method</kbd> `Registry.add_members`

```python theme={null}
add_members(*members: 'User | UserMember | Team | TeamMember | str') → Self
```

Adds users or teams to this registry.

**Args:**

* `members`:  The users or teams to add to the registry. Accepts  `User` objects, `Team` objects, or their string IDs.

**Returns:**
This registry for further method chaining, if needed.

**Raises:**

* `TypeError`:  If no members are passed as arguments.
* `ValueError`:  If unable to infer or parse the user or team IDs.

**Examples:**

```python theme={null}
import wandb

api = wandb.Api()

# Fetch an existing registry
registry = api.registry(name="my-registry", organization="my-org")

user1 = api.user(username="some-user")
user2 = api.user(username="other-user")
registry.add_members(user1, user2)

my_team = api.team(name="my-team")
registry.add_members(my_team)
```

***

### <kbd>method</kbd> `Registry.collections`

```python theme={null}
collections(
    filter: 'dict[str, Any] | None' = None,
    per_page: 'PositiveInt' = 100,
    start: 'str | None' = None
) → Collections
```

Returns the collections belonging to the registry.

***

### <kbd>classmethod</kbd> `Registry.create`

```python theme={null}
create(
    api: 'Api',
    organization: 'str',
    name: 'str',
    visibility: "Literal['organization', 'restricted']",
    description: 'str | None' = None,
    artifact_types: 'list[str] | None' = None
) → Self
```

Create a new registry.

The registry name must be unique within the organization. This function should be called using `api.create_registry()`

**Args:**

* `api`:  The W\&B API instance.
* `organization`:  The name of the organization.
* `name`:  The name of the registry (without the `wandb-registry-` prefix).
* `visibility`:  The visibility level ('organization' or 'restricted').
* `description`:  An optional description for the registry.
* `artifact_types`:  An optional list of allowed artifact types.

**Returns:**

* `Registry`:  The newly created Registry object.

**Raises:**

* `ValueError`:  If a registry with the same name already exists in the  organization or if the creation fails.

***

### <kbd>method</kbd> `Registry.delete`

```python theme={null}
delete() → None
```

Delete the registry. This is irreversible.

***

### <kbd>method</kbd> `Registry.load`

```python theme={null}
load() → None
```

Load registry attributes from the backend.

***

### <kbd>method</kbd> `Registry.members`

```python theme={null}
members() → list[UserMember | TeamMember]
```

Returns the current members (users and teams) of this registry.

***

### <kbd>method</kbd> `Registry.remove_members`

```python theme={null}
remove_members(*members: 'User | UserMember | Team | TeamMember | str') → Self
```

Removes users or teams from this registry.

**Args:**

* `members`:  The users or teams to remove from the registry. Accepts  `User` objects, `Team` objects, or their string IDs.

**Returns:**
This registry for further method chaining, if needed.

**Raises:**

* `TypeError`:  If no members are passed as arguments.
* `ValueError`:  If unable to infer or parse the user or team IDs.

**Examples:**

```python theme={null}
import wandb

api = wandb.Api()

# Fetch an existing registry
registry = api.registry(name="my-registry", organization="my-org")

user1 = api.user(username="some-user")
user2 = api.user(username="other-user")
registry.remove_members(user1, user2)

old_team = api.team(name="old-team")
registry.remove_members(old_team)
```

***

### <kbd>method</kbd> `Registry.save`

```python theme={null}
save() → None
```

Save registry attributes to the backend.

***

### <kbd>method</kbd> `Registry.team_members`

```python theme={null}
team_members() → list[TeamMember]
```

Returns the current member teams of this registry.

***

### <kbd>method</kbd> `Registry.update_member`

```python theme={null}
update_member(
    member: 'User | UserMember | Team | TeamMember | str',
    role: 'MemberRole | str'
) → Self
```

Updates the role of a member (user or team) within this registry.

**Args:**

* `member`:  The user or team to update the role of.  Accepts a `User` object, `Team` object, or their string ID.
* `role`:  The new role to assign to the member. May be one of:
  * "admin"
  * "member"
  * "viewer"
  * "restricted\_viewer" (if supported by the W\&B server)

**Returns:**
This registry for further method chaining, if needed.

**Raises:**

* `ValueError`:  If unable to infer the user or team ID.

**Examples:**
Make all users in the registry admins \`\`\`python
import wandb

api = wandb.Api()

# Fetch an existing registry

registry = api.registry(name="my-registry", organization="my-org")

for member in registry.user\_members():
registry.update\_member(member.user, role="admin")

````

---

### <kbd>method</kbd> `Registry.user_members`

```python
user_members() → list[UserMember]
````

Returns the current member users of this registry.

***

### <kbd>method</kbd> `Registry.versions`

```python theme={null}
versions(
    filter: 'dict[str, Any] | None' = None,
    per_page: 'PositiveInt' = 100,
    start: 'str | None' = None
) → Versions
```

Returns the versions belonging to the registry.


# Reports
Source: https://docs.wandb.ai/models/ref/python/public-api/reports



<GitHubLink />

## <kbd>class</kbd> `Reports`

Reports is a lazy iterator of `BetaReport` objects.

### <kbd>method</kbd> `Reports.__init__`

```python theme={null}
__init__(
    service_api: 'ServiceApi',
    project: 'Project',
    name: 'str | None' = None,
    entity: 'str | None' = None,
    per_page: 'int' = 50
)
```

**Args:**

* `service_api`:  Interface to the wandb-core service that performs  W\&B API calls for this collection.
* `project` (`wandb.sdk.internal.Project`):  The project to fetch reports from.
* `name` (str, optional):  The name of the report to filter by. If `None`,  fetches all reports.
* `entity` (str, optional):  The entity name for the project. Defaults to  the project entity.
* `per_page` (int):  Number of reports to fetch per page (default is 50).

***

### <kbd>property</kbd> Reports.length

***

### <kbd>method</kbd> `Reports.convert_objects`

```python theme={null}
convert_objects() → list[BetaReport]
```

Converts GraphQL edges to File objects.

***

### <kbd>method</kbd> `Reports.update_variables`

```python theme={null}
update_variables() → None
```

Updates the GraphQL query variables for pagination.


# Run
Source: https://docs.wandb.ai/models/ref/python/public-api/run



<GitHubLink />

## <kbd>class</kbd> `Run`

A single run associated with an entity and project.

### <kbd>method</kbd> `Run.__init__`

```python theme={null}
__init__(
    service_api: 'ServiceApi',
    entity: 'str',
    project: 'str',
    run_id: 'str',
    attrs: 'Mapping | None' = None,
    include_sweeps: 'bool' = True,
    lazy: 'bool' = True,
    api_key: 'str | None' = None
)
```

**Args:**

* `service_api`:  Interface to the wandb-core service that performs  W\&B API calls for this run.
* `entity`:  The entity associated with the run.
* `project`:  The project associated with the run.
* `run_id`:  The unique identifier for the run.
* `attrs`:  The attributes of the run.
* `include_sweeps`:  Whether to include sweeps in the run.

**Attributes:**

* `tags` (\[str]):  a list of tags associated with the run
* `url` (str):  the url of this run
* `id` (str):  unique identifier for the run (defaults to eight characters)
* `name` (str):  the name of the run
* `state` (str):  one of: running, finished, crashed, killed, preempting, preempted
* `config` (dict):  a dict of hyperparameters associated with the run
* `created_at` (str):  ISO timestamp when the run was started
* `system_metrics` (dict):  the latest system metrics recorded for the run
* `summary` (dict):  A mutable dict-like property that holds the current summary.  Calling update will persist any changes.
* `project` (str):  the project associated with the run
* `entity` (str):  the name of the entity associated with the run
* `project_internal_id` (int):  the internal id of the project
* `user` (str):  the name of the user who created the run
* `path` (str):  Unique identifier \[entity]/\[project]/\[run\_id]
* `notes` (str):  Notes about the run
* `read_only` (boolean):  Whether the run is editable
* `history_keys` (str):  History metric keys logged with `wandb.Run.log({"key": "value"})`
* `metadata` (str):  Metadata about the run from wandb-metadata.json

Initialize a Run object.

Run is always initialized by calling api.runs() where api is an instance of wandb.Api.

***

### <kbd>property</kbd> Run.config

Get run config. Auto-loads full data if in lazy mode.

**Returns:**

* `dict[str, Any]`: The config property value.

***

### <kbd>property</kbd> Run.entity

The entity associated with the run.

**Returns:**

* `str`: The entity property value.

***

### <kbd>property</kbd> Run.id

The unique identifier for the run.

**Returns:**

* `str`: The id property value.

***

### <kbd>property</kbd> Run.lastHistoryStep

Returns the last step logged in the run's history.

**Returns:**

* `int`: The lastHistoryStep property value.

***

### <kbd>property</kbd> Run.metadata

Metadata about the run from wandb-metadata.json.

Metadata includes the run's description, tags, start time, memory usage and more.

**Returns:**

* `dict[str, Any] | None`: The metadata property value.

***

### <kbd>property</kbd> Run.name

The name of the run.

**Returns:**

* `str | None`: The name property value.

***

### <kbd>property</kbd> Run.path

The path of the run. The path is a list containing the entity, project, and run\_id.

**Returns:**

* `list[str]`: The path property value.

***

### <kbd>property</kbd> Run.rawconfig

Get raw run config including internal keys. Auto-loads full data if in lazy mode.

**Returns:**

* `dict[str, Any]`: The rawconfig property value.

***

### <kbd>property</kbd> Run.state

The state of the run. Can be one of: Finished, Failed, Crashed, or Running.

**Returns:**

* `str`: The state property value.

***

### <kbd>property</kbd> Run.storage\_id

The unique storage identifier for the run.

**Returns:**

* `str`: The storage\_id property value.

***

### <kbd>property</kbd> Run.summary

Get run summary metrics. Auto-loads full data if in lazy mode.

**Returns:**

* `HTTPSummary`: The summary property value.

***

### <kbd>property</kbd> Run.summary\_metrics

Get run summary metrics. Auto-loads full data if in lazy mode.

**Returns:**

* `dict[str, Any]`: The summary\_metrics property value.

***

### <kbd>property</kbd> Run.sweep\_name

Get sweep name. Always available since sweepName is in lightweight fragment.

**Returns:**

* `str | None`: The sweep\_name property value.

***

### <kbd>property</kbd> Run.system\_metrics

Get run system metrics. Auto-loads full data if in lazy mode.

**Returns:**

* `dict[str, Any]`: The system\_metrics property value.

***

### <kbd>property</kbd> Run.url

The URL of the run.

The run URL is generated from the entity, project, and run\_id. For SaaS users, it takes the form of `https://wandb.ai/entity/project/run_id`.

**Returns:**

* `str`: The url property value.

***

### <kbd>property</kbd> Run.username

This API is deprecated. Use `entity` instead.

**Returns:**

* `str`: The username property value.

***

### <kbd>method</kbd> `Run.beta_scan_history`

```python theme={null}
beta_scan_history(
    keys: 'list[str] | None' = None,
    page_size: 'int' = 1000,
    min_step: 'int' = 0,
    max_step: 'int | None' = None,
    use_cache: 'bool' = True
) → public.HistoryScan
```

***

### <kbd>classmethod</kbd> `Run.create`

```python theme={null}
create(
    api: 'public.Api',
    run_id: 'str | None' = None,
    project: 'str | None' = None,
    entity: 'str | None' = None,
    state: "Literal['running', 'pending']" = 'running'
) → Self
```

Create a run for the given project.

For most use cases, use `wandb.init()`. `wandb.init()` provides more robust logic for creating and updating runs. `wandb.apis.public.Run.create` is intended for specific scenarios such as creating runs in a "pending" state for jobs that may be unschedulable (for example, in a Kubernetes cluster with insufficient GPUs or high contention). These pending runs can later be resumed and tracked by W\&B.

Runs created with this method have limited functionality. Calling `update()` on a run created this way may not work as expected.

**Args:**

* `api`:  The W\&B API instance.
* `run_id`:  Optional run ID. If not provided, a random ID will be generated.
* `project`:  Optional project name. Defaults to the project in API settings  or "uncategorized".
* `entity`:  Optional entity (user or team) name.
* `state`:  Initial state of the run. Use "pending" for runs that will be  resumed later, or "running" for immediate execution.

**Returns:**
A Run object representing the created run.

**Example:**
Creating a pending run for later execution

```python theme={null}
import wandb

api = wandb.Api()

run_name = "my-pending-run"

run = Run.create(
    api=api,
    project="project",
    entity="entity",
    state="pending",
    run_id=run_name,
)
```

***

### <kbd>method</kbd> `Run.delete`

```python theme={null}
delete(delete_artifacts: 'bool' = False) → None
```

Delete the given run from the wandb backend.

**Args:**

* `delete_artifacts` (bool, optional):  Whether to delete the artifacts  associated with the run.

***

### <kbd>method</kbd> `Run.download_history_exports`

```python theme={null}
download_history_exports(
    download_dir: 'pathlib.Path | str',
    require_complete_history: 'bool' = True
) → runhistory.DownloadHistoryResult
```

Download any parquet history files for the run to the provided directory.

**Args:**

* `download_dir`:  The directory to download the history files to.
* `require_complete_history`:  Whether to require the complete history to be downloaded.  If true, and the run contains data that has not been exported to parquet files yet,  an IncompleteRunHistoryError will be raised.

**Returns:**
A DownloadHistoryResult.

**Raises:**

* `IncompleteRunHistoryError`:  If require\_complete\_history is True  and the run contains data not yet exported to parquet files.
* `WandbApiFailedError`:  If the API request fails for reasons other than  incomplete history.

***

### <kbd>method</kbd> `Run.file`

```python theme={null}
file(name: 'str') → public.File
```

Return the path of a file with a given name in the artifact.

**Args:**

* `name` (str):  name of requested file.

**Returns:**
A `File` matching the name argument.

***

### <kbd>method</kbd> `Run.files`

```python theme={null}
files(
    names: 'list[str] | None' = None,
    pattern: 'str | None' = None,
    per_page: 'int' = 50
) → public.Files
```

Returns a `Files` object for all files in the run which match the given criteria.

You can specify a list of exact file names to match, or a pattern to match against. If both are provided, the pattern will be ignored.

**Args:**

* `names` (list):  names of the requested files, if empty returns all files
* `pattern` (str, optional):  Pattern to match when returning files from W\&B.  This pattern uses mySQL's LIKE syntax,  so matching all files that end with .json would be "%.json".  If both names and pattern are provided, a ValueError will be raised.
* `per_page` (int):  number of results per page.

**Returns:**
A `Files` object, which is an iterator over `File` objects.

***

### <kbd>method</kbd> `Run.history`

```python theme={null}
history(
    samples: 'int' = 500,
    keys: 'list[str] | None' = None,
    x_axis: 'str' = '_step',
    pandas: 'bool' = True,
    stream: "Literal['default', 'system']" = 'default'
) → list[dict[str, Any]] | pd.DataFrame
```

Return sampled history metrics for a run.

This is simpler and faster if you are ok with the history records being sampled.

**Args:**

* `samples `:  (int, optional) The number of samples to return
* `pandas `:  (bool, optional) Return a pandas dataframe
* `keys `:  (list, optional) Only return metrics for specific keys
* `x_axis `:  (str, optional) Use this metric as the xAxis defaults to \_step
* `stream `:  (str, optional) "default" for metrics, "system" for machine metrics

**Returns:**

* `pandas.DataFrame`:  If pandas=True returns a `pandas.DataFrame` of history  metrics.
* `list of dicts`:  If pandas=False returns a list of dicts of history metrics.

***

### <kbd>method</kbd> `Run.load`

```python theme={null}
load(force: 'bool' = False) → dict[str, Any]
```

Load run data using appropriate fragment based on lazy mode.

***

### <kbd>method</kbd> `Run.load_full_data`

```python theme={null}
load_full_data(force: 'bool' = False) → dict[str, Any]
```

Load full run data including heavy fields like config, systemMetrics, summaryMetrics.

This method is useful when you initially used lazy=True for listing runs, but need access to the full data for specific runs.

**Args:**

* `force`:  Force reload even if data is already loaded

**Returns:**
The loaded run attributes

***

### <kbd>method</kbd> `Run.log_artifact`

```python theme={null}
log_artifact(
    artifact: 'wandb.Artifact',
    aliases: 'Collection[str] | None' = None,
    tags: 'Collection[str] | None' = None
) → wandb.Artifact
```

Declare an artifact as output of a run.

**Args:**

* `artifact` (`Artifact`):  An artifact returned from  `wandb.Api().artifact(name)`.
* `aliases` (list, optional):  Aliases to apply to this artifact.
* `tags`:  (list, optional) Tags to apply to this artifact, if any.

**Returns:**
A `Artifact` object.

***

### <kbd>method</kbd> `Run.logged_artifacts`

```python theme={null}
logged_artifacts(per_page: 'int' = 100) → public.RunArtifacts
```

Fetches all artifacts logged by this run.

Retrieves all output artifacts that were logged during the run. Returns a paginated result that can be iterated over or collected into a single list.

**Args:**

* `per_page`:  Number of artifacts to fetch per API request.

**Returns:**
An iterable collection of all Artifact objects logged as outputs during this run.

**Example:**

```python theme={null}
import wandb
import tempfile

with tempfile.NamedTemporaryFile(mode="w", delete=False, suffix=".txt") as tmp:
   tmp.write("This is a test artifact")
   tmp_path = tmp.name
run = wandb.init(project="artifact-example")
artifact = wandb.Artifact("test_artifact", type="dataset")
artifact.add_file(tmp_path)
run.log_artifact(artifact)
run.finish()

api = wandb.Api()

finished_run = api.run(f"{run.entity}/{run.project}/{run.id}")

for logged_artifact in finished_run.logged_artifacts():
   print(logged_artifact.name)
```

***

### <kbd>method</kbd> `Run.save`

```python theme={null}
save() → None
```

Persist changes to the run object to the W\&B backend.

***

### <kbd>method</kbd> `Run.scan_history`

```python theme={null}
scan_history(
    keys: 'list[str] | None' = None,
    page_size: 'int' = 1000,
    min_step: 'int' = 0,
    max_step: 'int | None' = None,
    use_cache: 'bool' = True
) → public.HistoryScan
```

Returns an iterable collection of all history records for a run.

**Args:**

* `keys`:  list of metrics to read from the run's history.  if no keys are provided then all metrics will be returned.
* `page_size`:  the number of history records to read at a time.
* `min_step`:  The minimum step to start reading history from (inclusive).
* `max_step`:  The maximum step to read history up to (exclusive).
* `use_cache`:  When set to True, checks the WANDB\_CACHE\_DIR for a run history.  If the run history is not found in the cache, it will be downloaded from the server.  If set to False, the run history will be downloaded every time.

**Returns:**
A HistoryScan object, which can be iterator over to get history records.

***

### <kbd>method</kbd> `Run.to_html`

```python theme={null}
to_html(height: 'int' = 420, hidden: 'bool' = False) → str
```

Generate HTML containing an iframe displaying this run.

***

### <kbd>method</kbd> `Run.update`

```python theme={null}
update() → None
```

Persist changes to the run object to the wandb backend.

***

### <kbd>method</kbd> `Run.update_state`

```python theme={null}
update_state(state: "Literal['pending']") → bool
```

Update the state of a run.

Allows transitioning runs from 'failed' or 'crashed' to 'pending'.

**Args:**

* `state`:  The target run state. Only `"pending"` is supported.

**Returns:**
`True` if the state was successfully updated.

**Raises:**

* `wandb.Error`:  If the requested state transition is not allowed, or the server  does not support this operation.

***

### <kbd>method</kbd> `Run.upload_file`

```python theme={null}
upload_file(path: 'str', root: 'str' = '.') → public.File
```

Upload a local file to W\&B, associating it with this run.

**Args:**

* `path` (str):  Path to the file to upload. Can be absolute or relative.
* `root` (str):  The root path to save the file relative to. For example,  if you want to have the file saved in the run as "my\_dir/file.txt"  and you're currently in "my\_dir" you would set root to "../".  Defaults to current directory (".").

**Returns:**
A `File` object representing the uploaded file.

***

### <kbd>method</kbd> `Run.use_artifact`

```python theme={null}
use_artifact(
    artifact: 'wandb.Artifact',
    use_as: 'str | None' = None
) → wandb.Artifact
```

Declare an artifact as an input to a run.

**Args:**

* `artifact` (`Artifact`):  An artifact returned from  `wandb.Api().artifact(name)`
* `use_as` (string, optional):  A string identifying  how the artifact is used in the script. Used  to easily differentiate artifacts used in a  run, when using the beta wandb launch  feature's artifact swapping functionality.

**Returns:**
An `Artifact` object.

***

### <kbd>method</kbd> `Run.used_artifacts`

```python theme={null}
used_artifacts(per_page: 'int' = 100) → public.RunArtifacts
```

Fetches artifacts explicitly used by this run.

Retrieves only the input artifacts that were explicitly declared as used during the run, typically via `run.use_artifact()`. Returns a paginated result that can be iterated over or collected into a single list.

**Args:**

* `per_page`:  Number of artifacts to fetch per API request.

**Returns:**
An iterable collection of Artifact objects explicitly used as inputs in this run.

**Example:**

```python theme={null}
import wandb

run = wandb.init(project="artifact-example")
run.use_artifact("test_artifact:latest")
run.finish()

api = wandb.Api()
finished_run = api.run(f"{run.entity}/{run.project}/{run.id}")
for used_artifact in finished_run.used_artifacts():
   print(used_artifact.name)
test_artifact
```

***

### <kbd>method</kbd> `Run.wait_until_finished`

```python theme={null}
wait_until_finished() → None
```

Check the state of the run until it is finished.


# RunArtifacts
Source: https://docs.wandb.ai/models/ref/python/public-api/runartifacts



<GitHubLink />

## <kbd>class</kbd> `RunArtifacts`

An iterable collection of artifacts associated with a specific run.

### <kbd>property</kbd> RunArtifacts.cursor

An opaque cursor that marks the start of the next page to fetch.

This value may be saved and passed as `start=` to a later paginated query to resume iteration from where this paginator left off.

**Returns:**

* `str | None`: The cursor property value.

***

### <kbd>property</kbd> RunArtifacts.more


# Runs
Source: https://docs.wandb.ai/models/ref/python/public-api/runs



<GitHubLink />

## <kbd>class</kbd> `Runs`

A lazy iterator of `Run` objects associated with a project and optional filter.

Runs are retrieved in pages from the W\&B server as needed.

This is generally used indirectly using the `Api.runs` namespace.

### <kbd>method</kbd> `Runs.__init__`

```python theme={null}
__init__(
    service_api: 'ServiceApi',
    entity: 'str',
    project: 'str',
    filters: 'dict[str, Any] | None' = None,
    order: 'str' = '+created_at',
    per_page: 'int' = 50,
    include_sweeps: 'bool' = True,
    lazy: 'bool' = True,
    api_key: 'str | None' = None
)
```

**Args:**

* `service_api`:  The service API to use for requests.
* `entity`:  The entity (username or team) that owns the project.
* `project`:  The name of the project to fetch runs from.
* `filters`:  Filters to apply to the runs query.
* `order`:  Order can be `created_at`, `heartbeat_at`, `config.*.value`, or `summary_metrics.*`.  If you prepend order with a + order is ascending (default).  If you prepend order with a - order is descending.  The default order is run.created\_at from oldest to newest.
* `per_page`:  The number of runs to fetch per request (default is 50).
* `include_sweeps`:  Whether to include sweep information in the runs.  Defaults to True.

***

### <kbd>property</kbd> Runs.length

***

### <kbd>method</kbd> `Runs.histories`

```python theme={null}
histories(
    samples: 'int' = 500,
    keys: 'list[str] | None' = None,
    x_axis: 'str' = '_step',
    format: "Literal['default', 'pandas', 'polars']" = 'default',
    stream: "Literal['default', 'system']" = 'default'
) → list[dict[str, Any]] | pd.DataFrame | pl.DataFrame
```

Return sampled history metrics for all runs that fit the filters conditions.

**Args:**

* `samples`:  The number of samples to return per run
* `keys`:  Only return metrics for specific keys
* `x_axis`:  Use this metric as the xAxis defaults to \_step
* `format`:  Format to return data in, options are "default", "pandas",  "polars"
* `stream`:  "default" for metrics, "system" for machine metrics

**Returns:**

* `pandas.DataFrame`:  If `format="pandas"`, returns a `pandas.DataFrame`  of history metrics.
* `polars.DataFrame`:  If `format="polars"`, returns a `polars.DataFrame`  of history metrics.
* `list of dicts`:  If `format="default"`, returns a list of dicts  containing history metrics with a `run_id` key.

***

### <kbd>method</kbd> `Runs.upgrade_to_full`

```python theme={null}
upgrade_to_full() → None
```

Upgrade this Runs collection from lazy to full mode.

This switches to fetching full run data and upgrades any already-loaded Run objects to have full data. Uses parallel loading for better performance when upgrading multiple runs.


# Sweep
Source: https://docs.wandb.ai/models/ref/python/public-api/sweep



<GitHubLink />

## <kbd>class</kbd> `Sweep`

The set of runs associated with the sweep.

**Attributes:**

* `runs` (Runs):  List of runs
* `id` (str):  Sweep ID
* `project` (str):  The name of the project the sweep belongs to
* `config` (dict):  Dictionary containing the sweep configuration
* `state` (str):  The state of the sweep. Can be "Finished", "Failed",  "Crashed", or "Running".
* `expected_run_count` (int):  The number of expected runs for the sweep

### <kbd>method</kbd> `Sweep.__init__`

```python theme={null}
__init__(
    service_api: 'ServiceApi',
    entity: 'str',
    project: 'str',
    sweep_id: 'str',
    attrs: 'Mapping[str, Any] | None' = None
)
```

***

### <kbd>property</kbd> Sweep.config

The sweep configuration used for the sweep.

***

### <kbd>property</kbd> Sweep.entity

The entity associated with the sweep.

**Returns:**

* `str`: The entity property value.

***

### <kbd>property</kbd> Sweep.expected\_run\_count

Return the number of expected runs in the sweep or None for infinite runs.

**Returns:**

* `int | None`: The expected\_run\_count property value.

***

### <kbd>property</kbd> Sweep.name

The name of the sweep.

Returns the first name that exists in the following priority order:

1. User-edited display name 2. Name configured at creation time 3. Sweep ID

***

### <kbd>property</kbd> Sweep.order

Return the order key for the sweep.

***

### <kbd>property</kbd> Sweep.path

Returns the path of the project.

The path is a list containing the entity, project name, and sweep ID.

***

### <kbd>property</kbd> Sweep.url

The URL of the sweep.

The sweep URL is generated from the entity, project, the term "sweeps", and the sweep ID.run\_id. For SaaS users, it takes the form of `https://wandb.ai/entity/project/sweeps/sweeps_ID`.

***

### <kbd>property</kbd> Sweep.username

Deprecated. Use `Sweep.entity` instead.

**Returns:**

* `str`: The username property value.

***

### <kbd>method</kbd> `Sweep.agent`

```python theme={null}
agent(agent_id: 'str') → Agent
```

Query an agent by ID for this sweep.

**Args:**

* `agent_id`:  The ID of the agent to look up.

***

### <kbd>method</kbd> `Sweep.agents`

```python theme={null}
agents() → list[Agent]
```

Query the list of all agents for this sweep.

***

### <kbd>method</kbd> `Sweep.best_run`

```python theme={null}
best_run(order=None)
```

Return the best run sorted by the metric defined in config or the order passed in.

***

### <kbd>classmethod</kbd> `Sweep.get`

```python theme={null}
get(
    api: 'Api',
    entity: 'str | None' = None,
    project: 'str | None' = None,
    sid: 'str | None' = None,
    order: 'str | None' = None,
    query: 'str | None' = None,
    **kwargs
)
```

Execute a query against the cloud backend.

**Args:**

* `api`:  The W\&B API instance.
* `entity`:  The entity (username or team) that owns the project.
* `project`:  The name of the project to fetch sweep from.
* `sid`:  The sweep ID to query.
* `order`:  The order in which the sweep's runs are returned.
* `query`:  The query to use to execute the query.
* `**kwargs`:  Additional keyword arguments to pass to the query.

***

### <kbd>method</kbd> `Sweep.to_html`

```python theme={null}
to_html(height: 'int' = 420, hidden: 'bool' = False) → str
```

Generate HTML containing an iframe displaying this sweep.


# Sweeps
Source: https://docs.wandb.ai/models/ref/python/public-api/sweeps



<GitHubLink />

## <kbd>class</kbd> `Sweeps`

A lazy iterator over a collection of `Sweep` objects.

**Examples:**

```python theme={null}
from wandb.apis.public import Api

sweeps = Api().project(name="project_name", entity="entity").sweeps()

# Iterate over sweeps and print details
for sweep in sweeps:
    print(f"Sweep name: {sweep.name}")
    print(f"Sweep ID: {sweep.id}")
    print(f"Sweep URL: {sweep.url}")
    print("----------")
```

### <kbd>method</kbd> `Sweeps.__init__`

```python theme={null}
__init__(
    service_api: 'ServiceApi',
    entity: 'str',
    project: 'str',
    per_page: 'int' = 50
) → Sweeps
```

An iterable collection of `Sweep` objects.

**Args:**

* `service_api`:  The service API used to query W\&B.
* `entity`:  The entity which owns the sweeps.
* `project`:  The project which contains the sweeps.
* `per_page`:  The number of sweeps to fetch per request to the API.

***

### <kbd>property</kbd> Sweeps.length

***


# Team
Source: https://docs.wandb.ai/models/ref/python/public-api/team



<GitHubLink />

## <kbd>class</kbd> `Team`

A class that represents a W\&B team.

This class provides methods to manage W\&B teams, including creating teams, inviting members, and managing service accounts. It inherits from Attrs to handle team attributes.

### <kbd>method</kbd> `Team.__init__`

```python theme={null}
__init__(
    service_api: 'ServiceApi',
    name: 'str',
    attrs: 'Mapping[str, Any] | None' = None
)
```

**Args:**

* `service_api`:  The service API instance to use for querying W\&B.
* `name` (str):  The name of the team
* `attrs` (dict):  Optional dictionary of team attributes

**Note:**

> Team management requires appropriate permissions.

***

### <kbd>classmethod</kbd> `Team.create`

```python theme={null}
create(api: 'Api', team: 'str', admin_username: 'str | None' = None) → Team
```

Create a new team.

**Args:**

* `api`:  (`Api`) The api instance to use
* `team`:  (str) The name of the team
* `admin_username`:  (str) optional username of the admin user of the team, defaults to the current user.

**Returns:**
A `Team` object

***

### <kbd>method</kbd> `Team.create_service_account`

```python theme={null}
create_service_account(description: 'str') → Member | None
```

Create a service account for the team.

**Args:**

* `description`:  (str) A description for this service account

**Returns:**
The service account `Member` object, or None on failure

***

### <kbd>method</kbd> `Team.invite`

```python theme={null}
invite(username_or_email: 'str', admin: 'bool' = False) → bool
```

Invite a user to a team.

**Args:**

* `username_or_email`:  (str) The username or email address of the user  you want to invite.
* `admin`:  (bool) Whether to make this user a team admin.  Defaults to `False`.

**Returns:**
`True` on success, `False` if user was already invited or didn't exist.

***


# User
Source: https://docs.wandb.ai/models/ref/python/public-api/user



<GitHubLink />

## <kbd>class</kbd> `User`

A user on a W\&B instance.

This allows managing a user's API keys and accessing information like team memberships. The `create` class method can be used to create a new user.

**Args:**

* `service_api`:  The service API instance to use for querying W\&B.
* `attrs`:  A subset of the User type in the GraphQL schema.

### <kbd>property</kbd> User.api\_keys

Names of the user's API keys.

This property returns the names of the the API keys, *not* the secret associated with the key. The name of the key cannot be used as an API key.

The list is empty if the user has no API keys or if API keys have not been loaded.

**Returns:**

* `list[str]`: The api\_keys property value.

***

### <kbd>property</kbd> User.teams

Names of the user's teams.

This is an empty list if the user has no team memberships or if teams data was not loaded.

**Returns:**

* `list[str]`: The teams property value.

***

### <kbd>property</kbd> User.user\_api

A `wandb.Api` instance using the user's credentials.

**Returns:**

* `Api | None`: The user\_api property value.

***

### <kbd>method</kbd> `User.delete_api_key`

```python theme={null}
delete_api_key(api_key: 'str') → bool
```

Delete a user's API key.

Only the owner of the key or an admin can delete it.

**Args:**

* `api_key`:  The name of the API key to delete. Use one of  the names returned by the `api_keys` property.

**Returns:**
True on success, false on failure.

***

### <kbd>method</kbd> `User.generate_api_key`

```python theme={null}
generate_api_key(description: 'str | None' = None) → str | None
```

Generate a new API key.

**Args:**

* `description`:  A description for the new API key. This can be  used to identify the purpose of the API key.

**Returns:**
The generated API key (the full secret, not just the name), or None on failure.


# wandb docker
Source: https://docs.wandb.ai/models/ref/cli/wandb-docker



<GitHubLink />

## Usage

```bash theme={null}
docker [OPTIONS] [DOCKER_RUN_ARGS]... [DOCKER_IMAGE]
```

## Description

Run code in a Docker container with W\&B configured.

Start a Docker container, inject the WANDB\_DOCKERs environment
variable, and mount the current working directory at `/app` by
default. Inject WANDB\_API\_KEY if logged in. Override the
container entrypoint to ensure `wandb` is installed.

Pass additional arguments to insert them into `docker run` before
the image name. Use a default image if none is specified.

Use `--jupyter` to install and start JupyterLab on port 8888.
Enable the NVIDIA runtime automatically if NVIDIA Docker is
available on the host.

Requires Docker to be installed and running on the host machine.

To inject W\&B environment variables into an existing `docker run`
command without modifying the entrypoint, use `wandb docker-run`.

For example, to run the default image and mount a dataset into the container:

```bash theme={null}
wandb docker -v /mnt/dataset:/app/data
```

To run a default image and start JupyterLab:

```bash theme={null}
wandb docker -v /mnt/dataset:/app/data --jupyter
```

To run a GPU-enabled image with a training command:

```bash theme={null}
wandb docker wandb/deepo:keras-gpu --no-tty --cmd "python train.py"
```

## Arguments

| Name              | Default | Type  |
| ----------------- | ------- | ----- |
| `docker_run_args` | STR     | False |
| `docker_image`    | STR     | False |

## Options

| Flag        | Type      | Description                                                                                |
| ----------- | --------- | ------------------------------------------------------------------------------------------ |
| `--nvidia`  | BOOL Flag | Use the nvidia runtime, defaults to nvidia if nvidia-docker is present. **Default**: False |
| `--digest`  | BOOL Flag | Output the image digest and exit. **Default**: False                                       |
| `--jupyter` | BOOL Flag | Run jupyter lab in the container. **Default**: False                                       |
| `--dir`     | STR       | Which directory to mount the code in the container. **Default**: /app                      |
| `--no-dir`  | BOOL Flag | Don't mount the current directory. **Default**: False                                      |
| `--shell`   | STR       | The shell to start the container with. **Default**: /bin/bash                              |
| `--port`    | STR       | The host port to bind jupyter on. **Default**: 8888                                        |
| `--cmd`     | STR       | The command to run in the container. **Default**: None                                     |
| `--no-tty`  | BOOL Flag | Run the command without a tty. **Default**: False                                          |


# wandb docker-run
Source: https://docs.wandb.ai/models/ref/cli/wandb-docker-run



<GitHubLink />

## Usage

```bash theme={null}
docker-run [OPTIONS] [DOCKER_RUN_ARGS]...
```

## Description

Wrap `docker run` and inject W\&B environment variables automatically.

Pass all arguments through to `docker run` while injecting:

* WANDB\_API\_KEY: Inject the current API key if logged in so the
  container can authenticate with W\&B.

* WANDB\_DOCKER: Inject the resolved image ID if the image can be detected
  from the arguments so W\&B can track which Docker image produced the run.

Add `--runtime nvidia` automatically if `nvidia-docker` is detected
on the host and `--runtime` is not already set.

For example, to run the Python script train.py inside the "my-image" container:

```bash theme={null}
wandb docker-run my-image python train.py
```

## Arguments

| Name              | Default | Type  |
| ----------------- | ------- | ----- |
| `docker_run_args` | STR     | False |


# wandb enabled
Source: https://docs.wandb.ai/models/ref/cli/wandb-enabled



<GitHubLink />

## Usage

```bash theme={null}
enabled [OPTIONS]
```

## Description

Re-enable W\&B after it was deactivated with `wandb disabled`.

Set the mode to `online` to restore full W\&B functionality,
including cloud syncing and artifact storage.

Does not upload data logged while W\&B was set to `disabled`, but allows
new data to be logged and synced.

To switch between online and offline modes without fully deactivating W\&B,
use `wandb online` or `wandb offline` instead.

For example, to restore W\&B functionality after deactivating it with `wandb disabled`:

```bash theme={null}
wandb enabled
```

To run a training script with W\&B logging and syncing restored:

```bash theme={null}
python train.py # Log and sync data to W&B
```

## Options

| Flag        | Type      | Description                                                        |
| ----------- | --------- | ------------------------------------------------------------------ |
| `--service` | BOOL Flag | No effect. Accepted for backwards compatibility. **Default**: True |


# wandb init
Source: https://docs.wandb.ai/models/ref/cli/wandb-init



<GitHubLink />

## Usage

```bash theme={null}
init [OPTIONS]
```

## Description

Initialize or update W\&B configuration for the current directory.

Set a project and entity, create local W\&B settings, and
prepare the directory for experiment tracking.

For example, set up W\&B for the current directory with guided prompts
for team and project selection:

```bash theme={null}
wandb init
```

To set the default project to "foobar" and the default entity to "team-awesome" without prompts:

```bash theme={null}
wandb init --project foobar --entity team-awesome
```

To set the W\&B mode to offline:

```bash theme={null}
wandb init --mode offline
```

To reset existing W\&B configuration for the current directory:

```bash theme={null}
wandb init --reset
```

## Options

| Flag            | Type      | Description                                                                     |
| --------------- | --------- | ------------------------------------------------------------------------------- |
| `--project, -p` | STR       | Set the project to upload runs to. **Default**: None                            |
| `--entity, -e`  | STR       | Set the entity to scope the project to. **Default**: None                       |
| `--reset`       | BOOL Flag | Reset existing W\&B configuration for the directory. **Default**: False         |
| `--mode, -m`    | STR       | Set the W\&B mode. One of 'online', 'offline', or 'disabled'. **Default**: None |


# wandb job
Source: https://docs.wandb.ai/models/ref/cli/wandb-job



<GitHubLink />

## Usage

```bash theme={null}
job [OPTIONS] COMMAND [ARGS]...
```

## Description

Commands for managing and viewing W\&B jobs.

## Subcommands

| Command              | Description                                               |
| -------------------- | --------------------------------------------------------- |
| `wandb job create`   | Create a job from a source, without a wandb run.          |
| `wandb job describe` | Describe a launch job. Provide the launch job in the form |
| `wandb job list`     | List jobs in a project.                                   |


# wandb job create
Source: https://docs.wandb.ai/models/ref/cli/wandb-job/wandb-job-create



<GitHubLink />

## Usage

```bash theme={null}
create [OPTIONS] {git|code|image} PATH
```

## Description

Create a job from a source, without a wandb run.

Jobs can be of three types, git, code, or image.

git: A git source, with an entrypoint either in the path or provided
explicitly pointing to the main python executable.
code: A code path, containing a requirements.txt file.
image: A docker image.

## Arguments

| Name       | Default | Type |
| ---------- | ------- | ---- |
| `job_type` | Choice  | True |
| `path`     | STR     | True |

## Options

| Flag                  | Type | Description                                                                                                                                                                                                                        |
| --------------------- | ---- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `--project, -p`       | STR  | The project you want to list jobs from. **Default**: None                                                                                                                                                                          |
| `--entity, -e`        | STR  | The entity the jobs belong to. **Default**: None                                                                                                                                                                                   |
| `--name, -n`          | STR  | Name for the job. **Default**: None                                                                                                                                                                                                |
| `--description, -d`   | STR  | Description for the job. **Default**: None                                                                                                                                                                                         |
| `--alias, -a`         | STR  | Alias for the job. **Default**: \[]                                                                                                                                                                                                |
| `--entry-point, -E`   | STR  | Entrypoint to the script, including an executable and an entrypoint file. Required for code or repo jobs. If --build-context is provided, paths in the entrypoint command will be relative to the build context. **Default**: None |
| `--git-hash, -g`      | STR  | Commit reference to use as the source for git jobs. **Default**: None                                                                                                                                                              |
| `--runtime, -r`       | STR  | Python runtime to execute the job. **Default**: None                                                                                                                                                                               |
| `--build-context, -b` | STR  | Path to the build context from the root of the job source code. If provided, this is used as the base path for the Dockerfile and entrypoint. **Default**: None                                                                    |
| `--base-image, -B`    | STR  | Base image to use for the job. Incompatible with image jobs. **Default**: None                                                                                                                                                     |
| `--dockerfile, -D`    | STR  | Path to the Dockerfile for the job. If --build-context is provided, the Dockerfile path will be relative to the build context. **Default**: None                                                                                   |


# wandb job describe
Source: https://docs.wandb.ai/models/ref/cli/wandb-job/wandb-job-describe



<GitHubLink />

## Usage

```bash theme={null}
describe [OPTIONS] JOB
```

## Description

Describe a launch job. Provide the launch job in the form
of: entity/project/job-name:alias-or-version.

## Arguments

| Name  | Default | Type |
| ----- | ------- | ---- |
| `job` | STR     | True |


# wandb job list
Source: https://docs.wandb.ai/models/ref/cli/wandb-job/wandb-job-list



<GitHubLink />

## Usage

```bash theme={null}
list [OPTIONS]
```

## Description

List jobs in a project.

## Options

| Flag            | Type | Description                                               |
| --------------- | ---- | --------------------------------------------------------- |
| `--project, -p` | STR  | The project you want to list jobs from. **Default**: None |
| `--entity, -e`  | STR  | The entity the jobs belong to. **Default**: models        |


# wandb login
Source: https://docs.wandb.ai/models/ref/cli/wandb-login



<GitHubLink />

## Usage

```bash theme={null}
login [OPTIONS] [KEY]...
```

## Description

Authenticate your machine with W\&B.

Store an API key locally for authenticating with W\&B services.
By default, credentials are stored without server-side verification.

If no API key is provided as an argument, the command looks for
credentials in the following order:

1. The WANDB\_API\_KEY environment variable

2. The api\_key setting in a system or workspace settings file (use
   `wandb status` to see which settings file is used)

3. The .netrc file (\~/.netrc, \~/\_netrc, or the NETRC env var path)

4. An interactive prompt (if a TTY is available)

For self-hosted or dedicated cloud deployments, specify the server
URL with `--host`, or set the WANDB\_BASE\_URL environment variable.

For example, to log in interactively (prompts for API key):

```bash theme={null}
wandb login
```

To log in with an explicit API key (WANDB\_API\_KEY\_EXAMPLE):

```bash theme={null}
wandb login WANDB_API_KEY_EXAMPLE
```

To log in and verify the API key is valid:

```bash theme={null}
wandb login --verify
```

To log in to the W\&B public cloud instead of a configured self-hosted instance:

```bash theme={null}
wandb login --cloud
```

To log in to a self-hosted W\&B instance:

```bash theme={null}
wandb login --host https://my-wandb-server.example.com
```

To force a new login prompt even if already authenticated:

```bash theme={null}
wandb login --relogin
```

## Arguments

| Name  | Default | Type  |
| ----- | ------- | ----- |
| `key` | STR     | False |

## Options

| Flag                 | Type      | Description                                                                                                                                                              |
| -------------------- | --------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `--cloud`            | BOOL Flag | Log in to the W\&B public cloud ([https://api.wandb.ai](https://api.wandb.ai)). Mutually exclusive with --host. **Default**: False                                       |
| `--host, --base-url` | STR       | Log in to a specific W\&B server instance by URL (e.g. [https://my-wandb.example.com](https://my-wandb.example.com)). Mutually exclusive with --cloud. **Default**: None |
| `--relogin`          | BOOL Flag | Force a new login prompt, ignoring any existing credentials. **Default**: None                                                                                           |
| `--verify`           | BOOL Flag | Verify the API key with W\&B after storing it. If verification is successful, display the source of the credentials and the default team. **Default**: False             |


# wandb offline
Source: https://docs.wandb.ai/models/ref/cli/wandb-offline



<GitHubLink />

## Usage

```bash theme={null}
offline [OPTIONS]
```

## Description

Save data logged to W\&B locally without uploading it to the cloud.

Use `wandb online` or `wandb sync` to upload offline runs.

For example, to run a script in offline mode to log data locally without syncing
to the cloud:

```bash theme={null}
wandb offline && python train.py
```

To run a script in offline mode:

```bash theme={null}
wandb offline && python train.py
```

At a later time, sync all offline runs to the cloud:

```bash theme={null}
wandb sync --sync-all
```


# wandb online
Source: https://docs.wandb.ai/models/ref/cli/wandb-online



<GitHubLink />

## Usage

```bash theme={null}
online [OPTIONS]
```

## Description

Re-enable cloud syncing for W\&B runs.

Clear the offline mode setting so subsequent runs in this directory
sync data to the W\&B cloud.

Undoes a previous call to `wandb offline`.

For example, to re-enable cloud syncing after working offline:

```bash theme={null}
wandb online
```

To re-enable cloud syncing, then run a training script:

```bash theme={null}
wandb online && python train.py
```


# wandb pull
Source: https://docs.wandb.ai/models/ref/cli/wandb-pull



<GitHubLink />

## Usage

```bash theme={null}
pull [OPTIONS] RUN
```

## Description

Download files from a W\&B run.

Fetch all files associated with the specified run. Skip files that already
exist locally with the same content. Create subdirectories as needed to
mirror the structure of the files in W\&B.

Use the run ID to reference the run, and optionally specify the project
and entity if not included in the run argument.

For example, to download files from a run with run ID "abcd1234" in the "foobar" project
and "team-awesome" entity:

```bash theme={null}
wandb pull -p foobar -e team-awesome abcd1234
```

## Arguments

| Name  | Default | Type |
| ----- | ------- | ---- |
| `run` | STR     | True |

## Options

| Flag            | Type | Description                                                                                                                                             |
| --------------- | ---- | ------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `--project, -p` | STR  | The project containing the run to pull files from. **Default**: None                                                                                    |
| `--entity, -e`  | STR  | The entity that owns the project. Defaults to the value of the WANDB\_ENTITY environment variable or the default entity if not set. **Default**: models |


# wandb purge-cache
Source: https://docs.wandb.ai/models/ref/cli/wandb-purge-cache



<GitHubLink />

## Usage

```bash theme={null}
purge-cache [OPTIONS]
```

## Description

Purges cached logs, run history, and artifacts from the
local W\&B cache.

## Options

| Flag      | Type      | Description                                                                                                     |
| --------- | --------- | --------------------------------------------------------------------------------------------------------------- |
| `--age`   | STR       | Removes items older than the specified time period (e.g., '10s', '5m', '8h', '7d', '6M', '1y'). **Default**: 0d |
| `--force` | BOOL Flag | Do not prompt for confirmation when deleting files. **Default**: False                                          |


# wandb restore
Source: https://docs.wandb.ai/models/ref/cli/wandb-restore



<GitHubLink />

## Usage

```bash theme={null}
restore [OPTIONS] RUN
```

## Description

Restore the code, config, or Docker environement from a previous W\&B run.

Recreate the environment of a previous run so you can reproduce it.
Requires authentication with W\&B.

Restore up to three pieces of state, depending on what the original
run recorded:

1. Config (always): Write the run config to wandb/config.yaml.

2. Git (if available): Check out the original commit on a new
   wandb/run\_id branch. Fetch and apply any saved diff patch.
   If the original commit cannot be found,
   fall back to an upstream commit if the original cannot be found.
   Run this command from the same git repository as the original run. Skip
   this step with `--no-git`.

3. Docker (if available): If the run was executed inside a Docker
   container, start the same image with the original command.

If the run has no git history and no Docker image,
restore only the config.

Accept the run identifier in any of the following formats:
run\_id, project:run\_id, entity/project:run\_id, or entity/project/run\_id .

For example, to restore a run with run ID  in the default project
(stored as the WANDB\_PROJECT environment variable) and entity
(set from WANDB\_ENTITY or the authenticated user's default entity):

```bash theme={null}
wandb restore abcd1234
```

To restore a run from the "foobar" project and "team-awesome" entity with
run ID abcd1234:

```bash theme={null}
wandb restore team-awesome/foobar/abcd1234
```

To restore run abcd1234 without restoring git state. Only restore config
and Docker state:

```bash theme={null}
wandb restore --no-git abcd1234
```

To restore run abcd1234 in detached HEAD mode instead of creating a branch:

```bash theme={null}
wandb restore --no-branch abcd1234
```

To restore run abcd1234 from another team's project:

```bash theme={null}
wandb restore other-team/their-project:abcd1234
```

## Arguments

| Name  | Default | Type |
| ----- | ------- | ---- |
| `run` | STR     | True |

## Options

| Flag            | Type      | Description                                                                                    |
| --------------- | --------- | ---------------------------------------------------------------------------------------------- |
| `--no-git`      | BOOL Flag | Skip git restoration. Only restore config and Docker state. **Default**: False                 |
| `--branch`      | BOOL Flag | Create a wandb/run\_id branch or check out the commit in detached HEAD mode. **Default**: True |
| `--project, -p` | STR       | Specify the project to look up the run in. **Default**: None                                   |
| `--entity, -e`  | STR       | Specify the entity to scope the run lookup to. **Default**: None                               |


# wandb server
Source: https://docs.wandb.ai/models/ref/cli/wandb-server



<GitHubLink />

## Usage

```bash theme={null}
server [OPTIONS] COMMAND [ARGS]...
```

## Description

Commands for operating a local W\&B server

## Subcommands

| Command              | Description                                               |
| -------------------- | --------------------------------------------------------- |
| `wandb server start` | Start a local W\&B Server instance in a Docker container. |
| `wandb server stop`  | Stop a running local W\&B server.                         |


# wandb server start
Source: https://docs.wandb.ai/models/ref/cli/wandb-server/wandb-server-start



<GitHubLink />

## Usage

```bash theme={null}
start [OPTIONS]
```

## Description

Start a local W\&B Server instance in a Docker container.

Pull and run the wandb/local Docker image. Map the specified host
port to port 8080 in the container and mount a persistent Docker
volume named "wandb" to store data.

Configure the "base\_url" setting to point to the local server so
that subsequent W\&B client calls use it. Prompt for login if no
API key is found.

Requires Docker to be installed and running on the host machine.

This command starts only the W\&B Models UI. It is provided only
for local testing. In production, the W\&B Kubernetes Operator is the recommended
way to deploy W\&B Server. See
[https://docs.wandb.ai/platform/hosting/self-managed/operator](https://docs.wandb.ai/platform/hosting/self-managed/operator).

For example, to start a local W\&B server on the default port (8080):

```bash theme={null}
wandb server start
```

To start the server on port 9090:

```bash theme={null}
wandb server start -p 9090
```

To start the server in the foreground:

```bash theme={null}
wandb server start --no-daemon
```

## Options

| Flag         | Type      | Description                                                                                   |
| ------------ | --------- | --------------------------------------------------------------------------------------------- |
| `--port, -p` | STR       | The host port to bind W\&B server on. **Default**: 8080                                       |
| `--env, -e`  | STR       | Environment variables to pass to wandb/local Docker image. **Default**: \[]                   |
| `--daemon`   | BOOL Flag | Run the server in the background. Use --no-daemon to run in the foreground. **Default**: True |


# wandb server stop
Source: https://docs.wandb.ai/models/ref/cli/wandb-server/wandb-server-stop



<GitHubLink />

## Usage

```bash theme={null}
stop [OPTIONS]
```

## Description

Stop a running local W\&B server.

Stops the Docker container named `wandb-local` that was started
by `wandb server start`. Requires Docker to be installed.

For example, to stop the local W\&B server:

```bash theme={null}
wandb server stop
```


# wandb status
Source: https://docs.wandb.ai/models/ref/cli/wandb-status



<GitHubLink />

## Usage

```bash theme={null}
status [OPTIONS]
```

## Description

Display the current W\&B configuration settings.

Print all active W\&B settings as formatted JSON, including the
base URL, API key, project, entity, and other resolved values.

For example, to show current settings:

```bash theme={null}
wandb status
```

## Options

| Flag         | Type      | Description                                     |
| ------------ | --------- | ----------------------------------------------- |
| `--settings` | BOOL Flag | Display the current settings. **Default**: True |


# wandb sweep
Source: https://docs.wandb.ai/models/ref/cli/wandb-sweep



<GitHubLink />

## Usage

```bash theme={null}
sweep [OPTIONS] CONFIG_YAML_OR_SWEEP_ID
```

## Description

Create, update, or manage a hyperparameter sweep.

Provide a YAML config file to create a sweep. Define the search
strategy, parameters, and metric to optimize in the config.
Register the sweep with the W\&B server and print the sweep ID
and a command to start an agent.

Provide a sweep ID (or full path entity/project/sweep\_id) with a
state flag (`--stop`, `--cancel`, `--pause`, or `--resume`) to manage
an existing sweep.

The sweep ID is a unique identifier for the sweep, generated by
W\&B when the sweep is created. You can find the sweep ID in the W\&B App
or in the output of the `wandb sweep` command when you create a new sweep.

For example, to create a sweep using the configuration defined in
sweep\_config.yaml. Use the current user's default entity and project:

```bash theme={null}
wandb sweep sweep_config.yaml
```

To create a sweep and store the results under the "team-awesome" entity
and "foobar" project:

```bash theme={null}
wandb sweep -p foobar -e team-awesome sweep_config.yaml
```

To update sweep abcd1234 with a new configuration from sweep\_config.yaml.
This is useful for changing the parameters or search strategy of an
active sweep:

```bash theme={null}
wandb sweep --update abcd1234 sweep_config.yaml
```

To stop sweep abcd1234 under the "team-awesome" entity
and "foobar" project:

```bash theme={null}
wandb sweep --stop team-awesome/foobar/abcd1234
```

To cancel sweep abcd1234 in the current user's default entity and project:

```bash theme={null}
wandb sweep --cancel abcd1234
```

To pause sweep abcd1234 in the current user's default entity and
project. Later, resume the sweep:

```bash theme={null}
wandb sweep --pause abcd1234
```

```bash theme={null}
wandb sweep --resume abcd1234
```

To create a sweep with a local controller that uses the configuration
in sweep\_config.yaml:

```bash theme={null}
wandb sweep --controller sweep_config.yaml
```

To create a new sweep and include two previously completed runs
(run ID abcd1234 and run ID efgh5678) so their results are incorporated
into the sweep's hyperparameter search:

```bash theme={null}
wandb sweep -R abcd1234 -R efgh5678 sweep_config.yaml
```

## Arguments

| Name                      | Default | Type |
| ------------------------- | ------- | ---- |
| `config_yaml_or_sweep_id` | STR     | True |

## Options

| Flag              | Type      | Description                                                                                                   |
| ----------------- | --------- | ------------------------------------------------------------------------------------------------------------- |
| `--project, -p`   | STR       | Set the project for sweep runs. Use 'Uncategorized' if not set. **Default**: None                             |
| `--entity, -e`    | STR       | Set the entity for sweep. Use the current user's default entity if not set. **Default**: None                 |
| `--controller`    | BOOL Flag | Start a local sweep controller after creating the sweep. **Default**: False                                   |
| `--verbose`       | BOOL Flag | Display verbose output. **Default**: False                                                                    |
| `--name`          | STR       | Set a display name for the sweep. Use the sweep ID if not specified. **Default**: None                        |
| `--program`       | STR       | Override the training program specified in the sweep config. **Default**: None                                |
| `--update`        | STR       | Update an existing sweep configuration. Pass the sweep ID. **Default**: None                                  |
| `--stop`          | BOOL Flag | Stop a sweep. Let active runs finish but do not start new runs. **Default**: False                            |
| `--cancel`        | BOOL Flag | Cancel a sweep. Kill active runs and stop starting new ones. **Default**: False                               |
| `--pause`         | BOOL Flag | Pause a sweep. Temporarily stop starting new runs. **Default**: False                                         |
| `--resume`        | BOOL Flag | Resume a paused sweep. **Default**: False                                                                     |
| `--prior_run, -R` | STR       | Attach an existing run to this sweep by ID. Specify multiple times to attach multiple runs. **Default**: None |


# wandb sync
Source: https://docs.wandb.ai/models/ref/cli/wandb-sync



<GitHubLink />

## Usage

```bash theme={null}
sync [OPTIONS] [PATH]...
```

## Description

Upload existing local W\&B run data to the cloud.

Sync offline or incomplete runs from the local `wandb` directory to
the W\&B server. If PATH is provided, sync runs at that path. If no
path is given, search for a ./wandb directory, then a wandb/
subdirectory.

Run without arguments to print a summary of synced and unsynced
runs without uploading anything.

When syncing a specific path, include TensorBoard event files
by default. When using `--sync-all`, disable TensorBoard by
default (use `--sync-tensorboard` to enable it).

PATH is a .wandb file or a run directory that contains a .wandb file.
A typical path looks like:

./wandb/run-YYYYMMDD\_HHMMSS-RUN\_ID/run-RUN\_ID.wandb

where run-YYYYMMDD\_HHMMSS-RUN\_ID is the run directory and
run-RUN\_ID.wandb is the .wandb file. YYYYMMDD\_HHMMSS is the
timestamp of when the run was created and RUN\_ID is the unique ID
of the run.

For example, to show a summary of local runs and their sync status:

```bash theme={null}
wandb sync
```

To sync run ID abcd1234 that is locally saved locally in
the ./wandb/run-20170617\_000000-abcd1234 directory:

```bash theme={null}
wandb sync ./wandb/run-20170617_000000-abcd1234
```

To sync run abcd1234 by its local .wandb
filepath (./wandb/run-20170617\_000000-abcd1234/run-abcd1234.wandb):

```bash theme={null}
wandb sync ./wandb/run-20170617_000000-abcd1234/run-abcd1234.wandb
```

To sync all unsynced runs in the local wandb directory:

```bash theme={null}
wandb sync --sync-all
```

To delete local data for runs that have already been synced:

```bash theme={null}
wandb sync --clean
```

To delete synced runs older than 48 hours without a confirmation prompt:

```bash theme={null}
wandb sync --clean --clean-old-hours 48 --clean-force
```

## Arguments

| Name   | Default | Type  |
| ------ | ------- | ----- |
| `path` | PATH    | False |

## Options

| Flag                 | Type      | Description                                                                                             |
| -------------------- | --------- | ------------------------------------------------------------------------------------------------------- |
| `--id`               | STR       | Upload to an existing run ID. **Default**: None                                                         |
| `--project, -p`      | STR       | Set the project to upload the run to. **Default**: None                                                 |
| `--entity, -e`       | STR       | Set the entity to scope the project to. **Default**: None                                               |
| `--job_type`         | STR       | Set the job type to group related runs. **Default**: None                                               |
| `--sync-tensorboard` | BOOL Flag | Sync TensorBoard tfevent files. On by default for specific paths, off for --sync-all. **Default**: None |
| `--include-globs`    | STR       | Include only runs matching these glob patterns (comma-separated). **Default**: None                     |
| `--exclude-globs`    | STR       | Exclude runs matching these glob patterns (comma-separated). **Default**: None                          |
| `--include-online`   | BOOL Flag | Include runs created in online mode. **Default**: None                                                  |
| `--include-offline`  | BOOL Flag | Include runs created in offline mode. **Default**: None                                                 |
| `--include-synced`   | BOOL Flag | Include runs that are already synced. **Default**: None                                                 |
| `--mark-synced`      | BOOL Flag | Mark runs as synced after upload. **Default**: True                                                     |
| `--sync-all`         | BOOL Flag | Sync all unsynced runs in the local wandb directory. **Default**: False                                 |
| `--clean`            | BOOL Flag | Delete local data for runs that are already synced. **Default**: False                                  |
| `--clean-old-hours`  | INT       | Delete only synced runs older than this many hours (use with --clean). **Default**: 24                  |
| `--clean-force`      | BOOL Flag | Skip the confirmation prompt if --clean is specified. **Default**: False                                |
| `--show`             | INT       | Set the number of runs to show in the summary. **Default**: 5                                           |
| `--append`           | BOOL Flag | Append data to an existing run instead of creating a new run. **Default**: False                        |
| `--skip-console`     | BOOL Flag | Skip uploading console logs. **Default**: False                                                         |
| `--replace-tags`     | STR       | Rename tags during sync. Use 'old=new' pairs separated by commas. **Default**: None                     |


# wandb verify
Source: https://docs.wandb.ai/models/ref/cli/wandb-verify



<GitHubLink />

## Usage

```bash theme={null}
verify [OPTIONS]
```

## Description

Run integration checks against a self-hosted W\&B instance.

Validate that a self-hosted or dedicated cloud W\&B deployment is configured
and operating correctly. Do not run this command against the public W\&B
cloud at api.wandb.ai.

Exits with `code 1` if any critical check fails.

For example, to verify the currently configured W\&B instance:

```bash theme={null}
wandb verify --host https://my-wandb-instance.com
```

To verify a specific self-hosted instance:

```bash theme={null}
wandb verify --host https://my-wandb-server.example.com
```

## Options

| Flag     | Type | Description                                                                            |
| -------- | ---- | -------------------------------------------------------------------------------------- |
| `--host` | STR  | Target a specific W\&B instance URL. Default to configured base URL. **Default**: None |


# Query Expression Language overview
Source: https://docs.wandb.ai/models/ref/query-panel

A beta query language to select and aggregate data in W&B

Use the query expressions to select and aggregate data across runs and projects.
Learn more about [query panels](/models/app/features/panels/query-panels).

<Tip>
  See the public [Query panel examples report](https://wandb.ai/luis_team_test/weave_example_queries/reports/Query-Panel-Examples---Vmlldzo1NzIxOTY2) for interactive examples. For details about creating panels, stepping through run history, and configuration options such as **Concat** and **Paginate**, see the [Query panels overview](/models/app/features/panels/query-panels).
</Tip>

## Data Types

* [artifact](./query-panel/artifact)
* [artifactType](./query-panel/artifact-type)
* [artifactVersion](./query-panel/artifact-version)
* [audio-file](./query-panel/audio-file)
* [bokeh-file](./query-panel/bokeh-file)
* [boolean](./query-panel/boolean)
* [entity](./query-panel/entity)
* [file](./query-panel/file)
* [float](./query-panel/float)
* [html-file](./query-panel/html-file)
* [image-file](./query-panel/image-file)
* [int](./query-panel/int)
* [joined-table](./query-panel/joined-table)
* [molecule-file](./query-panel/molecule-file)
* [number](./query-panel/number)
* [object3D-file](./query-panel/object-3-d-file)
* [partitioned-table](./query-panel/partitioned-table)
* [project](./query-panel/project)
* [pytorch-model-file](./query-panel/pytorch-model-file)
* [run](./query-panel/run)
* [string](./query-panel/string)
* [table](./query-panel/table)
* [user](./query-panel/user)
* [video-file](./query-panel/video-file)


# artifact
Source: https://docs.wandb.ai/models/ref/query-panel/artifact



## Chainable Ops

### <a />`artifact-link`

Returns the url for a [artifact](/models/ref/query-panel/artifact)

| Argument   | Description                                     |
| :--------- | :---------------------------------------------- |
| `artifact` | An [artifact](/models/ref/query-panel/artifact) |

#### Return Value

The url for a [artifact](/models/ref/query-panel/artifact)

### <a />`artifact-name`

Returns the name of the [artifact](/models/ref/query-panel/artifact)

| Argument   | Description                                     |
| :--------- | :---------------------------------------------- |
| `artifact` | An [artifact](/models/ref/query-panel/artifact) |

#### Return Value

The name of the [artifact](/models/ref/query-panel/artifact)

### <a />`artifact-versions`

Returns the versions of the [artifact](/models/ref/query-panel/artifact)

| Argument   | Description                                     |
| :--------- | :---------------------------------------------- |
| `artifact` | An [artifact](/models/ref/query-panel/artifact) |

#### Return Value

The versions of the [artifact](/models/ref/query-panel/artifact)

## List Ops

### <a />`artifact-link`

Returns the url for a [artifact](/models/ref/query-panel/artifact)

| Argument   | Description                                     |
| :--------- | :---------------------------------------------- |
| `artifact` | An [artifact](/models/ref/query-panel/artifact) |

#### Return Value

The url for a [artifact](/models/ref/query-panel/artifact)

### <a />`artifact-name`

Returns the name of the [artifact](/models/ref/query-panel/artifact)

| Argument   | Description                                     |
| :--------- | :---------------------------------------------- |
| `artifact` | An [artifact](/models/ref/query-panel/artifact) |

#### Return Value

The name of the [artifact](/models/ref/query-panel/artifact)

### <a />`artifact-versions`

Returns the versions of the [artifact](/models/ref/query-panel/artifact)

| Argument   | Description                                     |
| :--------- | :---------------------------------------------- |
| `artifact` | An [artifact](/models/ref/query-panel/artifact) |

#### Return Value

The versions of the [artifact](/models/ref/query-panel/artifact)


# artifactType
Source: https://docs.wandb.ai/models/ref/query-panel/artifact-type



## Chainable Ops

### <a />`artifactType-artifactVersions`

Returns the [artifactVersions](/models/ref/query-panel/artifact-version) of all [artifacts](/models/ref/query-panel/artifact) of the [artifactType](/models/ref/query-panel/artifact-type)

| Argument       | Description                                              |
| :------------- | :------------------------------------------------------- |
| `artifactType` | An [artifactType](/models/ref/query-panel/artifact-type) |

#### Return Value

The [artifactVersions](/models/ref/query-panel/artifact-version) of all [artifacts](/models/ref/query-panel/artifact) of the [artifactType](/models/ref/query-panel/artifact-type)

### <a />`artifactType-artifacts`

Returns the [artifacts](/models/ref/query-panel/artifact) of the [artifactType](/models/ref/query-panel/artifact-type)

| Argument       | Description                                              |
| :------------- | :------------------------------------------------------- |
| `artifactType` | An [artifactType](/models/ref/query-panel/artifact-type) |

#### Return Value

The [artifacts](/models/ref/query-panel/artifact) of the [artifactType](/models/ref/query-panel/artifact-type)

### <a />`artifactType-name`

Returns the name of the [artifactType](/models/ref/query-panel/artifact-type)

| Argument       | Description                                              |
| :------------- | :------------------------------------------------------- |
| `artifactType` | An [artifactType](/models/ref/query-panel/artifact-type) |

#### Return Value

The name of the [artifactType](/models/ref/query-panel/artifact-type)

## List Ops

### <a />`artifactType-artifactVersions`

Returns the [artifactVersions](/models/ref/query-panel/artifact-version) of all [artifacts](/models/ref/query-panel/artifact) of the [artifactType](/models/ref/query-panel/artifact-type)

| Argument       | Description                                              |
| :------------- | :------------------------------------------------------- |
| `artifactType` | An [artifactType](/models/ref/query-panel/artifact-type) |

#### Return Value

The [artifactVersions](/models/ref/query-panel/artifact-version) of all [artifacts](/models/ref/query-panel/artifact) of the [artifactType](/models/ref/query-panel/artifact-type)

### <a />`artifactType-artifacts`

Returns the [artifacts](/models/ref/query-panel/artifact) of the [artifactType](/models/ref/query-panel/artifact-type)

| Argument       | Description                                              |
| :------------- | :------------------------------------------------------- |
| `artifactType` | An [artifactType](/models/ref/query-panel/artifact-type) |

#### Return Value

The [artifacts](/models/ref/query-panel/artifact) of the [artifactType](/models/ref/query-panel/artifact-type)

### <a />`artifactType-name`

Returns the name of the [artifactType](/models/ref/query-panel/artifact-type)

| Argument       | Description                                              |
| :------------- | :------------------------------------------------------- |
| `artifactType` | An [artifactType](/models/ref/query-panel/artifact-type) |

#### Return Value

The name of the [artifactType](/models/ref/query-panel/artifact-type)


# artifactVersion
Source: https://docs.wandb.ai/models/ref/query-panel/artifact-version



## Chainable Ops

### <a />`artifactVersion-aliases`

Returns the aliases for a [artifactVersion](/models/ref/query-panel/artifact-version)

| Argument          | Description                                                    |
| :---------------- | :------------------------------------------------------------- |
| `artifactVersion` | An [artifactVersion](/models/ref/query-panel/artifact-version) |

#### Return Value

The aliases for a [artifactVersion](/models/ref/query-panel/artifact-version)

### <a />`artifactVersion-createdAt`

Returns the datetime at which the [artifactVersion](/models/ref/query-panel/artifact-version) was created

| Argument          | Description                                                    |
| :---------------- | :------------------------------------------------------------- |
| `artifactVersion` | An [artifactVersion](/models/ref/query-panel/artifact-version) |

#### Return Value

The datetime at which the [artifactVersion](/models/ref/query-panel/artifact-version) was created

### <a />`artifactVersion-file`

Returns the *file* of the [artifactVersion](/models/ref/query-panel/artifact-version) for the given path

| Argument          | Description                                                    |
| :---------------- | :------------------------------------------------------------- |
| `artifactVersion` | An [artifactVersion](/models/ref/query-panel/artifact-version) |
| `path`            | The path of the *file*                                         |

#### Return Value

The *file* of the [artifactVersion](/models/ref/query-panel/artifact-version) for the given path

### <a />`artifactVersion-files`

Returns the *list* of *files* of the [artifactVersion](/models/ref/query-panel/artifact-version)

| Argument          | Description                                                    |
| :---------------- | :------------------------------------------------------------- |
| `artifactVersion` | An [artifactVersion](/models/ref/query-panel/artifact-version) |

#### Return Value

The *list* of *files* of the [artifactVersion](/models/ref/query-panel/artifact-version)

### <a />`artifactVersion-link`

Returns the url for a [artifactVersion](/models/ref/query-panel/artifact-version)

| Argument          | Description                                                    |
| :---------------- | :------------------------------------------------------------- |
| `artifactVersion` | An [artifactVersion](/models/ref/query-panel/artifact-version) |

#### Return Value

The url for a [artifactVersion](/models/ref/query-panel/artifact-version)

### <a />`artifactVersion-metadata`

Returns the [artifactVersion](/models/ref/query-panel/artifact-version) metadata dictionary

| Argument          | Description                                                    |
| :---------------- | :------------------------------------------------------------- |
| `artifactVersion` | An [artifactVersion](/models/ref/query-panel/artifact-version) |

#### Return Value

The [artifactVersion](/models/ref/query-panel/artifact-version) metadata dictionary

### <a />`artifactVersion-name`

Returns the name of the [artifactVersion](/models/ref/query-panel/artifact-version)

| Argument          | Description                                                    |
| :---------------- | :------------------------------------------------------------- |
| `artifactVersion` | An [artifactVersion](/models/ref/query-panel/artifact-version) |

#### Return Value

The name of the [artifactVersion](/models/ref/query-panel/artifact-version)

### <a />`artifactVersion-size`

Returns the size of the [artifactVersion](/models/ref/query-panel/artifact-version)

| Argument          | Description                                                    |
| :---------------- | :------------------------------------------------------------- |
| `artifactVersion` | An [artifactVersion](/models/ref/query-panel/artifact-version) |

#### Return Value

The size of the [artifactVersion](/models/ref/query-panel/artifact-version)

### <a />`artifactVersion-usedBy`

Returns the [runs](/models/ref/query-panel/run) that use the [artifactVersion](/models/ref/query-panel/artifact-version)

| Argument          | Description                                                    |
| :---------------- | :------------------------------------------------------------- |
| `artifactVersion` | An [artifactVersion](/models/ref/query-panel/artifact-version) |

#### Return Value

The [runs](/models/ref/query-panel/run) that use the [artifactVersion](/models/ref/query-panel/artifact-version)

### <a />`artifactVersion-versionId`

Returns the versionId of the [artifactVersion](/models/ref/query-panel/artifact-version)

| Argument          | Description                                                    |
| :---------------- | :------------------------------------------------------------- |
| `artifactVersion` | An [artifactVersion](/models/ref/query-panel/artifact-version) |

#### Return Value

The versionId of the [artifactVersion](/models/ref/query-panel/artifact-version)

## List Ops

### <a />`artifactVersion-aliases`

Returns the aliases for a [artifactVersion](/models/ref/query-panel/artifact-version)

| Argument          | Description                                                    |
| :---------------- | :------------------------------------------------------------- |
| `artifactVersion` | An [artifactVersion](/models/ref/query-panel/artifact-version) |

#### Return Value

The aliases for a [artifactVersion](/models/ref/query-panel/artifact-version)

### <a />`artifactVersion-createdAt`

Returns the datetime at which the [artifactVersion](/models/ref/query-panel/artifact-version) was created

| Argument          | Description                                                    |
| :---------------- | :------------------------------------------------------------- |
| `artifactVersion` | An [artifactVersion](/models/ref/query-panel/artifact-version) |

#### Return Value

The datetime at which the [artifactVersion](/models/ref/query-panel/artifact-version) was created

### <a />`artifactVersion-file`

Returns the *file* of the [artifactVersion](/models/ref/query-panel/artifact-version) for the given path

| Argument          | Description                                                    |
| :---------------- | :------------------------------------------------------------- |
| `artifactVersion` | An [artifactVersion](/models/ref/query-panel/artifact-version) |
| `path`            | The path of the *file*                                         |

#### Return Value

The *file* of the [artifactVersion](/models/ref/query-panel/artifact-version) for the given path

### <a />`artifactVersion-files`

Returns the *list* of *files* of the [artifactVersion](/models/ref/query-panel/artifact-version)

| Argument          | Description                                                    |
| :---------------- | :------------------------------------------------------------- |
| `artifactVersion` | An [artifactVersion](/models/ref/query-panel/artifact-version) |

#### Return Value

The *list* of *files* of the [artifactVersion](/models/ref/query-panel/artifact-version)

### <a />`artifactVersion-link`

Returns the url for a [artifactVersion](/models/ref/query-panel/artifact-version)

| Argument          | Description                                                    |
| :---------------- | :------------------------------------------------------------- |
| `artifactVersion` | An [artifactVersion](/models/ref/query-panel/artifact-version) |

#### Return Value

The url for a [artifactVersion](/models/ref/query-panel/artifact-version)

### <a />`artifactVersion-metadata`

Returns the [artifactVersion](/models/ref/query-panel/artifact-version) metadata dictionary

| Argument          | Description                                                    |
| :---------------- | :------------------------------------------------------------- |
| `artifactVersion` | An [artifactVersion](/models/ref/query-panel/artifact-version) |

#### Return Value

The [artifactVersion](/models/ref/query-panel/artifact-version) metadata dictionary

### <a />`artifactVersion-name`

Returns the name of the [artifactVersion](/models/ref/query-panel/artifact-version)

| Argument          | Description                                                    |
| :---------------- | :------------------------------------------------------------- |
| `artifactVersion` | An [artifactVersion](/models/ref/query-panel/artifact-version) |

#### Return Value

The name of the [artifactVersion](/models/ref/query-panel/artifact-version)

### <a />`artifactVersion-size`

Returns the size of the [artifactVersion](/models/ref/query-panel/artifact-version)

| Argument          | Description                                                    |
| :---------------- | :------------------------------------------------------------- |
| `artifactVersion` | An [artifactVersion](/models/ref/query-panel/artifact-version) |

#### Return Value

The size of the [artifactVersion](/models/ref/query-panel/artifact-version)

### <a />`artifactVersion-usedBy`

Returns the [runs](/models/ref/query-panel/run) that use the [artifactVersion](/models/ref/query-panel/artifact-version)

| Argument          | Description                                                    |
| :---------------- | :------------------------------------------------------------- |
| `artifactVersion` | An [artifactVersion](/models/ref/query-panel/artifact-version) |

#### Return Value

The [runs](/models/ref/query-panel/run) that use the [artifactVersion](/models/ref/query-panel/artifact-version)

### <a />`artifactVersion-versionId`

Returns the versionId of the [artifactVersion](/models/ref/query-panel/artifact-version)

| Argument          | Description                                                    |
| :---------------- | :------------------------------------------------------------- |
| `artifactVersion` | An [artifactVersion](/models/ref/query-panel/artifact-version) |

#### Return Value

The versionId of the [artifactVersion](/models/ref/query-panel/artifact-version)


# audio-file
Source: https://docs.wandb.ai/models/ref/query-panel/audio-file



## Chainable Ops

### <a />`asset-file`

Returns the *file* of the asset

| Argument | Description |
| :------- | :---------- |
| `asset`  | The asset   |

#### Return Value

The *file* of the asset

## List Ops

### <a />`asset-file`

Returns the *file* of the asset

| Argument | Description |
| :------- | :---------- |
| `asset`  | The asset   |

#### Return Value

The *file* of the asset


# bokeh-file
Source: https://docs.wandb.ai/models/ref/query-panel/bokeh-file



## Chainable Ops

### <a />`asset-file`

Returns the *file* of the asset

| Argument | Description |
| :------- | :---------- |
| `asset`  | The asset   |

#### Return Value

The *file* of the asset

## List Ops

### <a />`asset-file`

Returns the *file* of the asset

| Argument | Description |
| :------- | :---------- |
| `asset`  | The asset   |

#### Return Value

The *file* of the asset


# boolean
Source: https://docs.wandb.ai/models/ref/query-panel/boolean



## Chainable Ops

### <a />`and`

Returns the logical `and` of the two values

| Argument | Description         |
| :------- | :------------------ |
| `lhs`    | First binary value  |
| `rhs`    | Second binary value |

#### Return Value

The logical `and` of the two values

### <a />`or`

Returns the logical `or` of the two values

| Argument | Description         |
| :------- | :------------------ |
| `lhs`    | First binary value  |
| `rhs`    | Second binary value |

#### Return Value

The logical `or` of the two values

### <a />`boolean-not`

Returns the logical inverse of the value

| Argument | Description       |
| :------- | :---------------- |
| `bool`   | The boolean value |

#### Return Value

The logical inverse of the value

### <a />`boolean-not`

Returns the logical inverse of the value

| Argument | Description       |
| :------- | :---------------- |
| `bool`   | The boolean value |

#### Return Value

The logical inverse of the value

## List Ops

### <a />`and`

Returns the logical `and` of the two values

| Argument | Description         |
| :------- | :------------------ |
| `lhs`    | First binary value  |
| `rhs`    | Second binary value |

#### Return Value

The logical `and` of the two values

### <a />`or`

Returns the logical `or` of the two values

| Argument | Description         |
| :------- | :------------------ |
| `lhs`    | First binary value  |
| `rhs`    | Second binary value |

#### Return Value

The logical `or` of the two values

### <a />`boolean-not`

Returns the logical inverse of the value

| Argument | Description       |
| :------- | :---------------- |
| `bool`   | The boolean value |

#### Return Value

The logical inverse of the value

### <a />`boolean-not`

Returns the logical inverse of the value

| Argument | Description       |
| :------- | :---------------- |
| `bool`   | The boolean value |

#### Return Value

The logical inverse of the value


# entity
Source: https://docs.wandb.ai/models/ref/query-panel/entity



## Chainable Ops

### <a />`entity-link`

Returns the link of the [entity](/models/ref/query-panel/entity)

| Argument | Description                                 |
| :------- | :------------------------------------------ |
| `entity` | An [entity](/models/ref/query-panel/entity) |

#### Return Value

The link of the [entity](/models/ref/query-panel/entity)

### <a />`entity-name`

Returns the name of the [entity](/models/ref/query-panel/entity)

| Argument | Description                                 |
| :------- | :------------------------------------------ |
| `entity` | An [entity](/models/ref/query-panel/entity) |

#### Return Value

The name of the [entity](/models/ref/query-panel/entity)

## List Ops

### <a />`entity-link`

Returns the link of the [entity](/models/ref/query-panel/entity)

| Argument | Description                                 |
| :------- | :------------------------------------------ |
| `entity` | An [entity](/models/ref/query-panel/entity) |

#### Return Value

The link of the [entity](/models/ref/query-panel/entity)

### <a />`entity-name`

Returns the name of the [entity](/models/ref/query-panel/entity)

| Argument | Description                                 |
| :------- | :------------------------------------------ |
| `entity` | An [entity](/models/ref/query-panel/entity) |

#### Return Value

The name of the [entity](/models/ref/query-panel/entity)


# file
Source: https://docs.wandb.ai/models/ref/query-panel/file



## Chainable Ops

### <a />`file-contents`

Returns the contents of the *file*

| Argument | Description |
| :------- | :---------- |
| `file`   | A *file*    |

#### Return Value

The contents of the *file*

### <a />`file-digest`

Returns the digest of the *file*

| Argument | Description |
| :------- | :---------- |
| `file`   | A *file*    |

#### Return Value

The digest of the *file*

### <a />`file-size`

Returns the size of the *file*

| Argument | Description |
| :------- | :---------- |
| `file`   | A *file*    |

#### Return Value

The size of the *file*

### <a />`file-table`

Returns the contents of the *file* as a *table*

| Argument | Description |
| :------- | :---------- |
| `file`   | A *file*    |

#### Return Value

The contents of the *file* as a *table*

## List Ops

### <a />`file-contents`

Returns the contents of the *file*

| Argument | Description |
| :------- | :---------- |
| `file`   | A *file*    |

#### Return Value

The contents of the *file*

### <a />`file-digest`

Returns the digest of the *file*

| Argument | Description |
| :------- | :---------- |
| `file`   | A *file*    |

#### Return Value

The digest of the *file*

### <a />`file-size`

Returns the size of the *file*

| Argument | Description |
| :------- | :---------- |
| `file`   | A *file*    |

#### Return Value

The size of the *file*

### <a />`file-table`

Returns the contents of the *file* as a *table*

| Argument | Description |
| :------- | :---------- |
| `file`   | A *file*    |

#### Return Value

The contents of the *file* as a *table*


# float
Source: https://docs.wandb.ai/models/ref/query-panel/float



## Chainable Ops

### <a />`number-notEqual`

Determines inequality of two values.

| Argument | Description                  |
| :------- | :--------------------------- |
| `lhs`    | The first value to compare.  |
| `rhs`    | The second value to compare. |

#### Return Value

Whether the two values are not equal.

### <a />`number-modulo`

Divide a [number](/models/ref/query-panel/number) by another and return remainder

| Argument | Description                                           |
| :------- | :---------------------------------------------------- |
| `lhs`    | [number](/models/ref/query-panel/number) to divide    |
| `rhs`    | [number](/models/ref/query-panel/number) to divide by |

#### Return Value

Modulo of two [numbers](/models/ref/query-panel/number)

### <a />`number-mult`

Multiply two [numbers](/models/ref/query-panel/number)

| Argument | Description                                     |
| :------- | :---------------------------------------------- |
| `lhs`    | First [number](/models/ref/query-panel/number)  |
| `rhs`    | Second [number](/models/ref/query-panel/number) |

#### Return Value

Product of two [numbers](/models/ref/query-panel/number)

### <a />`number-powBinary`

Raise a [number](/models/ref/query-panel/number) to an exponent

| Argument | Description                                       |
| :------- | :------------------------------------------------ |
| `lhs`    | Base [number](/models/ref/query-panel/number)     |
| `rhs`    | Exponent [number](/models/ref/query-panel/number) |

#### Return Value

The base [numbers](/models/ref/query-panel/number) raised to nth power

### <a />`number-add`

Add two [numbers](/models/ref/query-panel/number)

| Argument | Description                                     |
| :------- | :---------------------------------------------- |
| `lhs`    | First [number](/models/ref/query-panel/number)  |
| `rhs`    | Second [number](/models/ref/query-panel/number) |

#### Return Value

Sum of two [numbers](/models/ref/query-panel/number)

### <a />`number-sub`

Subtract a [number](/models/ref/query-panel/number) from another

| Argument | Description                                               |
| :------- | :-------------------------------------------------------- |
| `lhs`    | [number](/models/ref/query-panel/number) to subtract from |
| `rhs`    | [number](/models/ref/query-panel/number) to subtract      |

#### Return Value

Difference of two [numbers](/models/ref/query-panel/number)

### <a />`number-div`

Divide a [number](/models/ref/query-panel/number) by another

| Argument | Description                                           |
| :------- | :---------------------------------------------------- |
| `lhs`    | [number](/models/ref/query-panel/number) to divide    |
| `rhs`    | [number](/models/ref/query-panel/number) to divide by |

#### Return Value

Quotient of two [numbers](/models/ref/query-panel/number)

### <a />`number-floorDiv`

Divide a [number](/models/ref/query-panel/number) by another then rounds down to the nearest whole number

| Argument | Description                                           |
| :------- | :---------------------------------------------------- |
| `lhs`    | [number](/models/ref/query-panel/number) to divide    |
| `rhs`    | [number](/models/ref/query-panel/number) to divide by |

#### Return Value

Truncated quotient of two [numbers](/models/ref/query-panel/number)

### <a />`number-less`

Check if a [number](/models/ref/query-panel/number) is less than another

| Argument | Description                                            |
| :------- | :----------------------------------------------------- |
| `lhs`    | [number](/models/ref/query-panel/number) to compare    |
| `rhs`    | [number](/models/ref/query-panel/number) to compare to |

#### Return Value

Whether the first [number](/models/ref/query-panel/number) is less than the second

### <a />`number-lessEqual`

Check if a [number](/models/ref/query-panel/number) is less than or equal to another

| Argument | Description                                            |
| :------- | :----------------------------------------------------- |
| `lhs`    | [number](/models/ref/query-panel/number) to compare    |
| `rhs`    | [number](/models/ref/query-panel/number) to compare to |

#### Return Value

Whether the first [number](/models/ref/query-panel/number) is less than or equal to the second

### <a />`number-equal`

Determines equality of two values.

| Argument | Description                  |
| :------- | :--------------------------- |
| `lhs`    | The first value to compare.  |
| `rhs`    | The second value to compare. |

#### Return Value

Whether the two values are equal.

### <a />`number-greater`

Check if a [number](/models/ref/query-panel/number) is greater than another

| Argument | Description                                            |
| :------- | :----------------------------------------------------- |
| `lhs`    | [number](/models/ref/query-panel/number) to compare    |
| `rhs`    | [number](/models/ref/query-panel/number) to compare to |

#### Return Value

Whether the first [number](/models/ref/query-panel/number) is greater than the second

### <a />`number-greaterEqual`

Check if a [number](/models/ref/query-panel/number) is greater than or equal to another

| Argument | Description                                            |
| :------- | :----------------------------------------------------- |
| `lhs`    | [number](/models/ref/query-panel/number) to compare    |
| `rhs`    | [number](/models/ref/query-panel/number) to compare to |

#### Return Value

Whether the first [number](/models/ref/query-panel/number) is greater than or equal to the second

### <a />`number-negate`

Negate a [number](/models/ref/query-panel/number)

| Argument | Description      |
| :------- | :--------------- |
| `val`    | Number to negate |

#### Return Value

A [number](/models/ref/query-panel/number)

### <a />`number-toString`

Convert a [number](/models/ref/query-panel/number) to a string

| Argument | Description       |
| :------- | :---------------- |
| `in`     | Number to convert |

#### Return Value

String representation of the [number](/models/ref/query-panel/number)

### <a />`number-toTimestamp`

Converts a [number](/models/ref/query-panel/number) to a *timestamp*. Values less than 31536000000 will be converted to seconds, values less than 31536000000000 will be converted to milliseconds, values less than 31536000000000000 will be converted to microseconds, and values less than 31536000000000000000 will be converted to nanoseconds.

| Argument | Description                      |
| :------- | :------------------------------- |
| `val`    | Number to convert to a timestamp |

#### Return Value

Timestamp

### <a />`number-abs`

Calculates the absolute value of a [number](/models/ref/query-panel/number)

| Argument | Description                                |
| :------- | :----------------------------------------- |
| `n`      | A [number](/models/ref/query-panel/number) |

#### Return Value

The absolute value of the [number](/models/ref/query-panel/number)

### <a />`number-cos`

Calculates the cosine of a [number](/models/ref/query-panel/number) in radians

| Argument | Description                        |
| :------- | :--------------------------------- |
| `n`      | Radians to calculate the cosine of |

#### Return Value

Sine of the [number](/models/ref/query-panel/number)

### <a />`gauss`

Sample from a gaussian distribution with mean and standard deviation.

| Argument | Description                                 |
| :------- | :------------------------------------------ |
| `mean`   | The mean of the distribution.               |
| `std`    | The standard deviation of the distribution. |

#### Return Value

A sample from the gaussian distribution.

### <a />`number-sin`

Calculates the sine of a [number](/models/ref/query-panel/number) in radians

| Argument | Description                      |
| :------- | :------------------------------- |
| `n`      | Radians to calculate the sine of |

#### Return Value

Sine of the [number](/models/ref/query-panel/number)

## List Ops

### <a />`number-notEqual`

Determines inequality of two values.

| Argument | Description                  |
| :------- | :--------------------------- |
| `lhs`    | The first value to compare.  |
| `rhs`    | The second value to compare. |

#### Return Value

Whether the two values are not equal.

### <a />`number-modulo`

Divide a [number](/models/ref/query-panel/number) by another and return remainder

| Argument | Description                                           |
| :------- | :---------------------------------------------------- |
| `lhs`    | [number](/models/ref/query-panel/number) to divide    |
| `rhs`    | [number](/models/ref/query-panel/number) to divide by |

#### Return Value

Modulo of two [numbers](/models/ref/query-panel/number)

### <a />`number-mult`

Multiply two [numbers](/models/ref/query-panel/number)

| Argument | Description                                     |
| :------- | :---------------------------------------------- |
| `lhs`    | First [number](/models/ref/query-panel/number)  |
| `rhs`    | Second [number](/models/ref/query-panel/number) |

#### Return Value

Product of two [numbers](/models/ref/query-panel/number)

### <a />`number-powBinary`

Raise a [number](/models/ref/query-panel/number) to an exponent

| Argument | Description                                       |
| :------- | :------------------------------------------------ |
| `lhs`    | Base [number](/models/ref/query-panel/number)     |
| `rhs`    | Exponent [number](/models/ref/query-panel/number) |

#### Return Value

The base [numbers](/models/ref/query-panel/number) raised to nth power

### <a />`number-add`

Add two [numbers](/models/ref/query-panel/number)

| Argument | Description                                     |
| :------- | :---------------------------------------------- |
| `lhs`    | First [number](/models/ref/query-panel/number)  |
| `rhs`    | Second [number](/models/ref/query-panel/number) |

#### Return Value

Sum of two [numbers](/models/ref/query-panel/number)

### <a />`number-sub`

Subtract a [number](/models/ref/query-panel/number) from another

| Argument | Description                                               |
| :------- | :-------------------------------------------------------- |
| `lhs`    | [number](/models/ref/query-panel/number) to subtract from |
| `rhs`    | [number](/models/ref/query-panel/number) to subtract      |

#### Return Value

Difference of two [numbers](/models/ref/query-panel/number)

### <a />`number-div`

Divide a [number](/models/ref/query-panel/number) by another

| Argument | Description                                           |
| :------- | :---------------------------------------------------- |
| `lhs`    | [number](/models/ref/query-panel/number) to divide    |
| `rhs`    | [number](/models/ref/query-panel/number) to divide by |

#### Return Value

Quotient of two [numbers](/models/ref/query-panel/number)

### <a />`number-floorDiv`

Divide a [number](/models/ref/query-panel/number) by another then rounds down to the nearest whole number

| Argument | Description                                           |
| :------- | :---------------------------------------------------- |
| `lhs`    | [number](/models/ref/query-panel/number) to divide    |
| `rhs`    | [number](/models/ref/query-panel/number) to divide by |

#### Return Value

Truncated quotient of two [numbers](/models/ref/query-panel/number)

### <a />`number-less`

Check if a [number](/models/ref/query-panel/number) is less than another

| Argument | Description                                            |
| :------- | :----------------------------------------------------- |
| `lhs`    | [number](/models/ref/query-panel/number) to compare    |
| `rhs`    | [number](/models/ref/query-panel/number) to compare to |

#### Return Value

Whether the first [number](/models/ref/query-panel/number) is less than the second

### <a />`number-lessEqual`

Check if a [number](/models/ref/query-panel/number) is less than or equal to another

| Argument | Description                                            |
| :------- | :----------------------------------------------------- |
| `lhs`    | [number](/models/ref/query-panel/number) to compare    |
| `rhs`    | [number](/models/ref/query-panel/number) to compare to |

#### Return Value

Whether the first [number](/models/ref/query-panel/number) is less than or equal to the second

### <a />`number-equal`

Determines equality of two values.

| Argument | Description                  |
| :------- | :--------------------------- |
| `lhs`    | The first value to compare.  |
| `rhs`    | The second value to compare. |

#### Return Value

Whether the two values are equal.

### <a />`number-greater`

Check if a [number](/models/ref/query-panel/number) is greater than another

| Argument | Description                                            |
| :------- | :----------------------------------------------------- |
| `lhs`    | [number](/models/ref/query-panel/number) to compare    |
| `rhs`    | [number](/models/ref/query-panel/number) to compare to |

#### Return Value

Whether the first [number](/models/ref/query-panel/number) is greater than the second

### <a />`number-greaterEqual`

Check if a [number](/models/ref/query-panel/number) is greater than or equal to another

| Argument | Description                                            |
| :------- | :----------------------------------------------------- |
| `lhs`    | [number](/models/ref/query-panel/number) to compare    |
| `rhs`    | [number](/models/ref/query-panel/number) to compare to |

#### Return Value

Whether the first [number](/models/ref/query-panel/number) is greater than or equal to the second

### <a />`number-negate`

Negate a [number](/models/ref/query-panel/number)

| Argument | Description      |
| :------- | :--------------- |
| `val`    | Number to negate |

#### Return Value

A [number](/models/ref/query-panel/number)

### <a />`numbers-argmax`

Finds the index of maximum [number](/models/ref/query-panel/number)

| Argument  | Description                                                                                                               |
| :-------- | :------------------------------------------------------------------------------------------------------------------------ |
| `numbers` | *list* of [numbers](/models/ref/query-panel/number) to find the index of maximum [number](/models/ref/query-panel/number) |

#### Return Value

Index of maximum [number](/models/ref/query-panel/number)

### <a />`numbers-argmin`

Finds the index of minimum [number](/models/ref/query-panel/number)

| Argument  | Description                                                                                                               |
| :-------- | :------------------------------------------------------------------------------------------------------------------------ |
| `numbers` | *list* of [numbers](/models/ref/query-panel/number) to find the index of minimum [number](/models/ref/query-panel/number) |

#### Return Value

Index of minimum [number](/models/ref/query-panel/number)

### <a />`numbers-avg`

Average of [numbers](/models/ref/query-panel/number)

| Argument  | Description                                                    |
| :-------- | :------------------------------------------------------------- |
| `numbers` | *list* of [numbers](/models/ref/query-panel/number) to average |

#### Return Value

Average of [numbers](/models/ref/query-panel/number)

### <a />`numbers-max`

Maximum number

| Argument  | Description                                                                                                      |
| :-------- | :--------------------------------------------------------------------------------------------------------------- |
| `numbers` | *list* of [numbers](/models/ref/query-panel/number) to find the maximum [number](/models/ref/query-panel/number) |

#### Return Value

Maximum [number](/models/ref/query-panel/number)

### <a />`numbers-min`

Minimum number

| Argument  | Description                                                                                                      |
| :-------- | :--------------------------------------------------------------------------------------------------------------- |
| `numbers` | *list* of [numbers](/models/ref/query-panel/number) to find the minimum [number](/models/ref/query-panel/number) |

#### Return Value

Minimum [number](/models/ref/query-panel/number)

### <a />`numbers-stddev`

Standard deviation of [numbers](/models/ref/query-panel/number)

| Argument  | Description                                                                             |
| :-------- | :-------------------------------------------------------------------------------------- |
| `numbers` | *list* of [numbers](/models/ref/query-panel/number) to calculate the standard deviation |

#### Return Value

Standard deviation of [numbers](/models/ref/query-panel/number)

### <a />`numbers-sum`

Sum of [numbers](/models/ref/query-panel/number)

| Argument  | Description                                                |
| :-------- | :--------------------------------------------------------- |
| `numbers` | *list* of [numbers](/models/ref/query-panel/number) to sum |

#### Return Value

Sum of [numbers](/models/ref/query-panel/number)

### <a />`number-toString`

Convert a [number](/models/ref/query-panel/number) to a string

| Argument | Description       |
| :------- | :---------------- |
| `in`     | Number to convert |

#### Return Value

String representation of the [number](/models/ref/query-panel/number)

### <a />`number-toTimestamp`

Converts a [number](/models/ref/query-panel/number) to a *timestamp*. Values less than 31536000000 will be converted to seconds, values less than 31536000000000 will be converted to milliseconds, values less than 31536000000000000 will be converted to microseconds, and values less than 31536000000000000000 will be converted to nanoseconds.

| Argument | Description                      |
| :------- | :------------------------------- |
| `val`    | Number to convert to a timestamp |

#### Return Value

Timestamp

### <a />`number-abs`

Calculates the absolute value of a [number](/models/ref/query-panel/number)

| Argument | Description                                |
| :------- | :----------------------------------------- |
| `n`      | A [number](/models/ref/query-panel/number) |

#### Return Value

The absolute value of the [number](/models/ref/query-panel/number)

### <a />`number-cos`

Calculates the cosine of a [number](/models/ref/query-panel/number) in radians

| Argument | Description                        |
| :------- | :--------------------------------- |
| `n`      | Radians to calculate the cosine of |

#### Return Value

Sine of the [number](/models/ref/query-panel/number)

### <a />`number-sin`

Calculates the sine of a [number](/models/ref/query-panel/number) in radians

| Argument | Description                      |
| :------- | :------------------------------- |
| `n`      | Radians to calculate the sine of |

#### Return Value

Sine of the [number](/models/ref/query-panel/number)


# html-file
Source: https://docs.wandb.ai/models/ref/query-panel/html-file



## Chainable Ops

### <a />`asset-file`

Returns the *file* of the asset

| Argument | Description |
| :------- | :---------- |
| `asset`  | The asset   |

#### Return Value

The *file* of the asset

## List Ops

### <a />`asset-file`

Returns the *file* of the asset

| Argument | Description |
| :------- | :---------- |
| `asset`  | The asset   |

#### Return Value

The *file* of the asset


# image-file
Source: https://docs.wandb.ai/models/ref/query-panel/image-file



## Chainable Ops

### <a />`asset-file`

Returns the *file* of the asset

| Argument | Description |
| :------- | :---------- |
| `asset`  | The asset   |

#### Return Value

The *file* of the asset

## List Ops

### <a />`asset-file`

Returns the *file* of the asset

| Argument | Description |
| :------- | :---------- |
| `asset`  | The asset   |

#### Return Value

The *file* of the asset


# int
Source: https://docs.wandb.ai/models/ref/query-panel/int



## Chainable Ops

### <a />`number-notEqual`

Determines inequality of two values.

| Argument | Description                  |
| :------- | :--------------------------- |
| `lhs`    | The first value to compare.  |
| `rhs`    | The second value to compare. |

#### Return Value

Whether the two values are not equal.

### <a />`number-modulo`

Divide a [number](/models/ref/query-panel/number) by another and return remainder

| Argument | Description                                           |
| :------- | :---------------------------------------------------- |
| `lhs`    | [number](/models/ref/query-panel/number) to divide    |
| `rhs`    | [number](/models/ref/query-panel/number) to divide by |

#### Return Value

Modulo of two [numbers](/models/ref/query-panel/number)

### <a />`number-mult`

Multiply two [numbers](/models/ref/query-panel/number)

| Argument | Description                                     |
| :------- | :---------------------------------------------- |
| `lhs`    | First [number](/models/ref/query-panel/number)  |
| `rhs`    | Second [number](/models/ref/query-panel/number) |

#### Return Value

Product of two [numbers](/models/ref/query-panel/number)

### <a />`number-powBinary`

Raise a [number](/models/ref/query-panel/number) to an exponent

| Argument | Description                                       |
| :------- | :------------------------------------------------ |
| `lhs`    | Base [number](/models/ref/query-panel/number)     |
| `rhs`    | Exponent [number](/models/ref/query-panel/number) |

#### Return Value

The base [numbers](/models/ref/query-panel/number) raised to nth power

### <a />`number-add`

Add two [numbers](/models/ref/query-panel/number)

| Argument | Description                                     |
| :------- | :---------------------------------------------- |
| `lhs`    | First [number](/models/ref/query-panel/number)  |
| `rhs`    | Second [number](/models/ref/query-panel/number) |

#### Return Value

Sum of two [numbers](/models/ref/query-panel/number)

### <a />`number-sub`

Subtract a [number](/models/ref/query-panel/number) from another

| Argument | Description                                               |
| :------- | :-------------------------------------------------------- |
| `lhs`    | [number](/models/ref/query-panel/number) to subtract from |
| `rhs`    | [number](/models/ref/query-panel/number) to subtract      |

#### Return Value

Difference of two [numbers](/models/ref/query-panel/number)

### <a />`number-div`

Divide a [number](/models/ref/query-panel/number) by another

| Argument | Description                                           |
| :------- | :---------------------------------------------------- |
| `lhs`    | [number](/models/ref/query-panel/number) to divide    |
| `rhs`    | [number](/models/ref/query-panel/number) to divide by |

#### Return Value

Quotient of two [numbers](/models/ref/query-panel/number)

### <a />`number-floorDiv`

Divide a [number](/models/ref/query-panel/number) by another then rounds down to the nearest whole number

| Argument | Description                                           |
| :------- | :---------------------------------------------------- |
| `lhs`    | [number](/models/ref/query-panel/number) to divide    |
| `rhs`    | [number](/models/ref/query-panel/number) to divide by |

#### Return Value

Truncated quotient of two [numbers](/models/ref/query-panel/number)

### <a />`number-less`

Check if a [number](/models/ref/query-panel/number) is less than another

| Argument | Description                                            |
| :------- | :----------------------------------------------------- |
| `lhs`    | [number](/models/ref/query-panel/number) to compare    |
| `rhs`    | [number](/models/ref/query-panel/number) to compare to |

#### Return Value

Whether the first [number](/models/ref/query-panel/number) is less than the second

### <a />`number-lessEqual`

Check if a [number](/models/ref/query-panel/number) is less than or equal to another

| Argument | Description                                            |
| :------- | :----------------------------------------------------- |
| `lhs`    | [number](/models/ref/query-panel/number) to compare    |
| `rhs`    | [number](/models/ref/query-panel/number) to compare to |

#### Return Value

Whether the first [number](/models/ref/query-panel/number) is less than or equal to the second

### <a />`number-equal`

Determines equality of two values.

| Argument | Description                  |
| :------- | :--------------------------- |
| `lhs`    | The first value to compare.  |
| `rhs`    | The second value to compare. |

#### Return Value

Whether the two values are equal.

### <a />`number-greater`

Check if a [number](/models/ref/query-panel/number) is greater than another

| Argument | Description                                            |
| :------- | :----------------------------------------------------- |
| `lhs`    | [number](/models/ref/query-panel/number) to compare    |
| `rhs`    | [number](/models/ref/query-panel/number) to compare to |

#### Return Value

Whether the first [number](/models/ref/query-panel/number) is greater than the second

### <a />`number-greaterEqual`

Check if a [number](/models/ref/query-panel/number) is greater than or equal to another

| Argument | Description                                            |
| :------- | :----------------------------------------------------- |
| `lhs`    | [number](/models/ref/query-panel/number) to compare    |
| `rhs`    | [number](/models/ref/query-panel/number) to compare to |

#### Return Value

Whether the first [number](/models/ref/query-panel/number) is greater than or equal to the second

### <a />`number-negate`

Negate a [number](/models/ref/query-panel/number)

| Argument | Description      |
| :------- | :--------------- |
| `val`    | Number to negate |

#### Return Value

A [number](/models/ref/query-panel/number)

### <a />`number-toString`

Convert a [number](/models/ref/query-panel/number) to a string

| Argument | Description       |
| :------- | :---------------- |
| `in`     | Number to convert |

#### Return Value

String representation of the [number](/models/ref/query-panel/number)

### <a />`number-toTimestamp`

Converts a [number](/models/ref/query-panel/number) to a *timestamp*. Values less than 31536000000 will be converted to seconds, values less than 31536000000000 will be converted to milliseconds, values less than 31536000000000000 will be converted to microseconds, and values less than 31536000000000000000 will be converted to nanoseconds.

| Argument | Description                      |
| :------- | :------------------------------- |
| `val`    | Number to convert to a timestamp |

#### Return Value

Timestamp

### <a />`number-abs`

Calculates the absolute value of a [number](/models/ref/query-panel/number)

| Argument | Description                                |
| :------- | :----------------------------------------- |
| `n`      | A [number](/models/ref/query-panel/number) |

#### Return Value

The absolute value of the [number](/models/ref/query-panel/number)

### <a />`number-cos`

Calculates the cosine of a [number](/models/ref/query-panel/number) in radians

| Argument | Description                        |
| :------- | :--------------------------------- |
| `n`      | Radians to calculate the cosine of |

#### Return Value

Sine of the [number](/models/ref/query-panel/number)

### <a />`gauss`

Sample from a gaussian distribution with mean and standard deviation.

| Argument | Description                                 |
| :------- | :------------------------------------------ |
| `mean`   | The mean of the distribution.               |
| `std`    | The standard deviation of the distribution. |

#### Return Value

A sample from the gaussian distribution.

### <a />`number-sin`

Calculates the sine of a [number](/models/ref/query-panel/number) in radians

| Argument | Description                      |
| :------- | :------------------------------- |
| `n`      | Radians to calculate the sine of |

#### Return Value

Sine of the [number](/models/ref/query-panel/number)

## List Ops

### <a />`number-notEqual`

Determines inequality of two values.

| Argument | Description                  |
| :------- | :--------------------------- |
| `lhs`    | The first value to compare.  |
| `rhs`    | The second value to compare. |

#### Return Value

Whether the two values are not equal.

### <a />`number-modulo`

Divide a [number](/models/ref/query-panel/number) by another and return remainder

| Argument | Description                                           |
| :------- | :---------------------------------------------------- |
| `lhs`    | [number](/models/ref/query-panel/number) to divide    |
| `rhs`    | [number](/models/ref/query-panel/number) to divide by |

#### Return Value

Modulo of two [numbers](/models/ref/query-panel/number)

### <a />`number-mult`

Multiply two [numbers](/models/ref/query-panel/number)

| Argument | Description                                     |
| :------- | :---------------------------------------------- |
| `lhs`    | First [number](/models/ref/query-panel/number)  |
| `rhs`    | Second [number](/models/ref/query-panel/number) |

#### Return Value

Product of two [numbers](/models/ref/query-panel/number)

### <a />`number-powBinary`

Raise a [number](/models/ref/query-panel/number) to an exponent

| Argument | Description                                       |
| :------- | :------------------------------------------------ |
| `lhs`    | Base [number](/models/ref/query-panel/number)     |
| `rhs`    | Exponent [number](/models/ref/query-panel/number) |

#### Return Value

The base [numbers](/models/ref/query-panel/number) raised to nth power

### <a />`number-add`

Add two [numbers](/models/ref/query-panel/number)

| Argument | Description                                     |
| :------- | :---------------------------------------------- |
| `lhs`    | First [number](/models/ref/query-panel/number)  |
| `rhs`    | Second [number](/models/ref/query-panel/number) |

#### Return Value

Sum of two [numbers](/models/ref/query-panel/number)

### <a />`number-sub`

Subtract a [number](/models/ref/query-panel/number) from another

| Argument | Description                                               |
| :------- | :-------------------------------------------------------- |
| `lhs`    | [number](/models/ref/query-panel/number) to subtract from |
| `rhs`    | [number](/models/ref/query-panel/number) to subtract      |

#### Return Value

Difference of two [numbers](/models/ref/query-panel/number)

### <a />`number-div`

Divide a [number](/models/ref/query-panel/number) by another

| Argument | Description                                           |
| :------- | :---------------------------------------------------- |
| `lhs`    | [number](/models/ref/query-panel/number) to divide    |
| `rhs`    | [number](/models/ref/query-panel/number) to divide by |

#### Return Value

Quotient of two [numbers](/models/ref/query-panel/number)

### <a />`number-floorDiv`

Divide a [number](/models/ref/query-panel/number) by another then rounds down to the nearest whole number

| Argument | Description                                           |
| :------- | :---------------------------------------------------- |
| `lhs`    | [number](/models/ref/query-panel/number) to divide    |
| `rhs`    | [number](/models/ref/query-panel/number) to divide by |

#### Return Value

Truncated quotient of two [numbers](/models/ref/query-panel/number)

### <a />`number-less`

Check if a [number](/models/ref/query-panel/number) is less than another

| Argument | Description                                            |
| :------- | :----------------------------------------------------- |
| `lhs`    | [number](/models/ref/query-panel/number) to compare    |
| `rhs`    | [number](/models/ref/query-panel/number) to compare to |

#### Return Value

Whether the first [number](/models/ref/query-panel/number) is less than the second

### <a />`number-lessEqual`

Check if a [number](/models/ref/query-panel/number) is less than or equal to another

| Argument | Description                                            |
| :------- | :----------------------------------------------------- |
| `lhs`    | [number](/models/ref/query-panel/number) to compare    |
| `rhs`    | [number](/models/ref/query-panel/number) to compare to |

#### Return Value

Whether the first [number](/models/ref/query-panel/number) is less than or equal to the second

### <a />`number-equal`

Determines equality of two values.

| Argument | Description                  |
| :------- | :--------------------------- |
| `lhs`    | The first value to compare.  |
| `rhs`    | The second value to compare. |

#### Return Value

Whether the two values are equal.

### <a />`number-greater`

Check if a [number](/models/ref/query-panel/number) is greater than another

| Argument | Description                                            |
| :------- | :----------------------------------------------------- |
| `lhs`    | [number](/models/ref/query-panel/number) to compare    |
| `rhs`    | [number](/models/ref/query-panel/number) to compare to |

#### Return Value

Whether the first [number](/models/ref/query-panel/number) is greater than the second

### <a />`number-greaterEqual`

Check if a [number](/models/ref/query-panel/number) is greater than or equal to another

| Argument | Description                                            |
| :------- | :----------------------------------------------------- |
| `lhs`    | [number](/models/ref/query-panel/number) to compare    |
| `rhs`    | [number](/models/ref/query-panel/number) to compare to |

#### Return Value

Whether the first [number](/models/ref/query-panel/number) is greater than or equal to the second

### <a />`number-negate`

Negate a [number](/models/ref/query-panel/number)

| Argument | Description      |
| :------- | :--------------- |
| `val`    | Number to negate |

#### Return Value

A [number](/models/ref/query-panel/number)

### <a />`numbers-argmax`

Finds the index of maximum [number](/models/ref/query-panel/number)

| Argument  | Description                                                                                                               |
| :-------- | :------------------------------------------------------------------------------------------------------------------------ |
| `numbers` | *list* of [numbers](/models/ref/query-panel/number) to find the index of maximum [number](/models/ref/query-panel/number) |

#### Return Value

Index of maximum [number](/models/ref/query-panel/number)

### <a />`numbers-argmin`

Finds the index of minimum [number](/models/ref/query-panel/number)

| Argument  | Description                                                                                                               |
| :-------- | :------------------------------------------------------------------------------------------------------------------------ |
| `numbers` | *list* of [numbers](/models/ref/query-panel/number) to find the index of minimum [number](/models/ref/query-panel/number) |

#### Return Value

Index of minimum [number](/models/ref/query-panel/number)

### <a />`numbers-avg`

Average of [numbers](/models/ref/query-panel/number)

| Argument  | Description                                                    |
| :-------- | :------------------------------------------------------------- |
| `numbers` | *list* of [numbers](/models/ref/query-panel/number) to average |

#### Return Value

Average of [numbers](/models/ref/query-panel/number)

### <a />`numbers-max`

Maximum number

| Argument  | Description                                                                                                      |
| :-------- | :--------------------------------------------------------------------------------------------------------------- |
| `numbers` | *list* of [numbers](/models/ref/query-panel/number) to find the maximum [number](/models/ref/query-panel/number) |

#### Return Value

Maximum [number](/models/ref/query-panel/number)

### <a />`numbers-min`

Minimum number

| Argument  | Description                                                                                                      |
| :-------- | :--------------------------------------------------------------------------------------------------------------- |
| `numbers` | *list* of [numbers](/models/ref/query-panel/number) to find the minimum [number](/models/ref/query-panel/number) |

#### Return Value

Minimum [number](/models/ref/query-panel/number)

### <a />`numbers-stddev`

Standard deviation of [numbers](/models/ref/query-panel/number)

| Argument  | Description                                                                             |
| :-------- | :-------------------------------------------------------------------------------------- |
| `numbers` | *list* of [numbers](/models/ref/query-panel/number) to calculate the standard deviation |

#### Return Value

Standard deviation of [numbers](/models/ref/query-panel/number)

### <a />`numbers-sum`

Sum of [numbers](/models/ref/query-panel/number)

| Argument  | Description                                                |
| :-------- | :--------------------------------------------------------- |
| `numbers` | *list* of [numbers](/models/ref/query-panel/number) to sum |

#### Return Value

Sum of [numbers](/models/ref/query-panel/number)

### <a />`number-toString`

Convert a [number](/models/ref/query-panel/number) to a string

| Argument | Description       |
| :------- | :---------------- |
| `in`     | Number to convert |

#### Return Value

String representation of the [number](/models/ref/query-panel/number)

### <a />`number-toTimestamp`

Converts a [number](/models/ref/query-panel/number) to a *timestamp*. Values less than 31536000000 will be converted to seconds, values less than 31536000000000 will be converted to milliseconds, values less than 31536000000000000 will be converted to microseconds, and values less than 31536000000000000000 will be converted to nanoseconds.

| Argument | Description                      |
| :------- | :------------------------------- |
| `val`    | Number to convert to a timestamp |

#### Return Value

Timestamp

### <a />`number-abs`

Calculates the absolute value of a [number](/models/ref/query-panel/number)

| Argument | Description                                |
| :------- | :----------------------------------------- |
| `n`      | A [number](/models/ref/query-panel/number) |

#### Return Value

The absolute value of the [number](/models/ref/query-panel/number)

### <a />`number-cos`

Calculates the cosine of a [number](/models/ref/query-panel/number) in radians

| Argument | Description                        |
| :------- | :--------------------------------- |
| `n`      | Radians to calculate the cosine of |

#### Return Value

Sine of the [number](/models/ref/query-panel/number)

### <a />`number-sin`

Calculates the sine of a [number](/models/ref/query-panel/number) in radians

| Argument | Description                      |
| :------- | :------------------------------- |
| `n`      | Radians to calculate the sine of |

#### Return Value

Sine of the [number](/models/ref/query-panel/number)


# joined-table
Source: https://docs.wandb.ai/models/ref/query-panel/joined-table



## Chainable Ops

### <a />`asset-file`

Returns the *file* of the asset

| Argument | Description |
| :------- | :---------- |
| `asset`  | The asset   |

#### Return Value

The *file* of the asset

### <a />`joinedtable-file`

Returns the *file* of a *joined-table*

| Argument      | Description        |
| :------------ | :----------------- |
| `joinedTable` | The *joined-table* |

#### Return Value

The  *file* of a *joined-table*

### <a />`joinedtable-rows`

Returns the rows of a *joined-table*

| Argument      | Description                                                                                    |
| :------------ | :--------------------------------------------------------------------------------------------- |
| `joinedTable` | The *joined-table*                                                                             |
| `leftOuter`   | Whether to include rows from the left table that do not have a matching row in the right table |
| `rightOuter`  | Whether to include rows from the right table that do not have a matching row in the left table |

#### Return Value

The rows of the *joined-table*

## List Ops

### <a />`asset-file`

Returns the *file* of the asset

| Argument | Description |
| :------- | :---------- |
| `asset`  | The asset   |

#### Return Value

The *file* of the asset


# molecule-file
Source: https://docs.wandb.ai/models/ref/query-panel/molecule-file



## Chainable Ops

### <a />`asset-file`

Returns the *file* of the asset

| Argument | Description |
| :------- | :---------- |
| `asset`  | The asset   |

#### Return Value

The *file* of the asset

## List Ops

### <a />`asset-file`

Returns the *file* of the asset

| Argument | Description |
| :------- | :---------- |
| `asset`  | The asset   |

#### Return Value

The *file* of the asset


# number
Source: https://docs.wandb.ai/models/ref/query-panel/number



## Chainable Ops

### <a />`number-notEqual`

Determines inequality of two values.

| Argument | Description                  |
| :------- | :--------------------------- |
| `lhs`    | The first value to compare.  |
| `rhs`    | The second value to compare. |

#### Return Value

Whether the two values are not equal.

### <a />`number-modulo`

Divide a [number](/models/ref/query-panel/number) by another and return remainder

| Argument | Description                                           |
| :------- | :---------------------------------------------------- |
| `lhs`    | [number](/models/ref/query-panel/number) to divide    |
| `rhs`    | [number](/models/ref/query-panel/number) to divide by |

#### Return Value

Modulo of two [numbers](/models/ref/query-panel/number)

### <a />`number-mult`

Multiply two [numbers](/models/ref/query-panel/number)

| Argument | Description                                     |
| :------- | :---------------------------------------------- |
| `lhs`    | First [number](/models/ref/query-panel/number)  |
| `rhs`    | Second [number](/models/ref/query-panel/number) |

#### Return Value

Product of two [numbers](/models/ref/query-panel/number)

### <a />`number-powBinary`

Raise a [number](/models/ref/query-panel/number) to an exponent

| Argument | Description                                       |
| :------- | :------------------------------------------------ |
| `lhs`    | Base [number](/models/ref/query-panel/number)     |
| `rhs`    | Exponent [number](/models/ref/query-panel/number) |

#### Return Value

The base [numbers](/models/ref/query-panel/number) raised to nth power

### <a />`number-add`

Add two [numbers](/models/ref/query-panel/number)

| Argument | Description                                     |
| :------- | :---------------------------------------------- |
| `lhs`    | First [number](/models/ref/query-panel/number)  |
| `rhs`    | Second [number](/models/ref/query-panel/number) |

#### Return Value

Sum of two [numbers](/models/ref/query-panel/number)

### <a />`number-sub`

Subtract a [number](/models/ref/query-panel/number) from another

| Argument | Description                                               |
| :------- | :-------------------------------------------------------- |
| `lhs`    | [number](/models/ref/query-panel/number) to subtract from |
| `rhs`    | [number](/models/ref/query-panel/number) to subtract      |

#### Return Value

Difference of two [numbers](/models/ref/query-panel/number)

### <a />`number-div`

Divide a [number](/models/ref/query-panel/number) by another

| Argument | Description                                           |
| :------- | :---------------------------------------------------- |
| `lhs`    | [number](/models/ref/query-panel/number) to divide    |
| `rhs`    | [number](/models/ref/query-panel/number) to divide by |

#### Return Value

Quotient of two [numbers](/models/ref/query-panel/number)

### <a />`number-floorDiv`

Divide a [number](/models/ref/query-panel/number) by another then rounds down to the nearest whole number

| Argument | Description                                           |
| :------- | :---------------------------------------------------- |
| `lhs`    | [number](/models/ref/query-panel/number) to divide    |
| `rhs`    | [number](/models/ref/query-panel/number) to divide by |

#### Return Value

Truncated quotient of two [numbers](/models/ref/query-panel/number)

### <a />`number-less`

Check if a [number](/models/ref/query-panel/number) is less than another

| Argument | Description                                            |
| :------- | :----------------------------------------------------- |
| `lhs`    | [number](/models/ref/query-panel/number) to compare    |
| `rhs`    | [number](/models/ref/query-panel/number) to compare to |

#### Return Value

Whether the first [number](/models/ref/query-panel/number) is less than the second

### <a />`number-lessEqual`

Check if a [number](/models/ref/query-panel/number) is less than or equal to another

| Argument | Description                                            |
| :------- | :----------------------------------------------------- |
| `lhs`    | [number](/models/ref/query-panel/number) to compare    |
| `rhs`    | [number](/models/ref/query-panel/number) to compare to |

#### Return Value

Whether the first [number](/models/ref/query-panel/number) is less than or equal to the second

### <a />`number-equal`

Determines equality of two values.

| Argument | Description                  |
| :------- | :--------------------------- |
| `lhs`    | The first value to compare.  |
| `rhs`    | The second value to compare. |

#### Return Value

Whether the two values are equal.

### <a />`number-greater`

Check if a [number](/models/ref/query-panel/number) is greater than another

| Argument | Description                                            |
| :------- | :----------------------------------------------------- |
| `lhs`    | [number](/models/ref/query-panel/number) to compare    |
| `rhs`    | [number](/models/ref/query-panel/number) to compare to |

#### Return Value

Whether the first [number](/models/ref/query-panel/number) is greater than the second

### <a />`number-greaterEqual`

Check if a [number](/models/ref/query-panel/number) is greater than or equal to another

| Argument | Description                                            |
| :------- | :----------------------------------------------------- |
| `lhs`    | [number](/models/ref/query-panel/number) to compare    |
| `rhs`    | [number](/models/ref/query-panel/number) to compare to |

#### Return Value

Whether the first [number](/models/ref/query-panel/number) is greater than or equal to the second

### <a />`number-negate`

Negate a [number](/models/ref/query-panel/number)

| Argument | Description      |
| :------- | :--------------- |
| `val`    | Number to negate |

#### Return Value

A [number](/models/ref/query-panel/number)

### <a />`number-toString`

Convert a [number](/models/ref/query-panel/number) to a string

| Argument | Description       |
| :------- | :---------------- |
| `in`     | Number to convert |

#### Return Value

String representation of the [number](/models/ref/query-panel/number)

### <a />`number-toTimestamp`

Converts a [number](/models/ref/query-panel/number) to a *timestamp*. Values less than 31536000000 will be converted to seconds, values less than 31536000000000 will be converted to milliseconds, values less than 31536000000000000 will be converted to microseconds, and values less than 31536000000000000000 will be converted to nanoseconds.

| Argument | Description                      |
| :------- | :------------------------------- |
| `val`    | Number to convert to a timestamp |

#### Return Value

Timestamp

### <a />`number-abs`

Calculates the absolute value of a [number](/models/ref/query-panel/number)

| Argument | Description                                |
| :------- | :----------------------------------------- |
| `n`      | A [number](/models/ref/query-panel/number) |

#### Return Value

The absolute value of the [number](/models/ref/query-panel/number)

### <a />`number-cos`

Calculates the cosine of a [number](/models/ref/query-panel/number) in radians

| Argument | Description                        |
| :------- | :--------------------------------- |
| `n`      | Radians to calculate the cosine of |

#### Return Value

Sine of the [number](/models/ref/query-panel/number)

### <a />`gauss`

Sample from a gaussian distribution with mean and standard deviation.

| Argument | Description                                 |
| :------- | :------------------------------------------ |
| `mean`   | The mean of the distribution.               |
| `std`    | The standard deviation of the distribution. |

#### Return Value

A sample from the gaussian distribution.

### <a />`number-sin`

Calculates the sine of a [number](/models/ref/query-panel/number) in radians

| Argument | Description                      |
| :------- | :------------------------------- |
| `n`      | Radians to calculate the sine of |

#### Return Value

Sine of the [number](/models/ref/query-panel/number)

## List Ops

### <a />`number-notEqual`

Determines inequality of two values.

| Argument | Description                  |
| :------- | :--------------------------- |
| `lhs`    | The first value to compare.  |
| `rhs`    | The second value to compare. |

#### Return Value

Whether the two values are not equal.

### <a />`number-modulo`

Divide a [number](/models/ref/query-panel/number) by another and return remainder

| Argument | Description                                           |
| :------- | :---------------------------------------------------- |
| `lhs`    | [number](/models/ref/query-panel/number) to divide    |
| `rhs`    | [number](/models/ref/query-panel/number) to divide by |

#### Return Value

Modulo of two [numbers](/models/ref/query-panel/number)

### <a />`number-mult`

Multiply two [numbers](/models/ref/query-panel/number)

| Argument | Description                                     |
| :------- | :---------------------------------------------- |
| `lhs`    | First [number](/models/ref/query-panel/number)  |
| `rhs`    | Second [number](/models/ref/query-panel/number) |

#### Return Value

Product of two [numbers](/models/ref/query-panel/number)

### <a />`number-powBinary`

Raise a [number](/models/ref/query-panel/number) to an exponent

| Argument | Description                                       |
| :------- | :------------------------------------------------ |
| `lhs`    | Base [number](/models/ref/query-panel/number)     |
| `rhs`    | Exponent [number](/models/ref/query-panel/number) |

#### Return Value

The base [numbers](/models/ref/query-panel/number) raised to nth power

### <a />`number-add`

Add two [numbers](/models/ref/query-panel/number)

| Argument | Description                                     |
| :------- | :---------------------------------------------- |
| `lhs`    | First [number](/models/ref/query-panel/number)  |
| `rhs`    | Second [number](/models/ref/query-panel/number) |

#### Return Value

Sum of two [numbers](/models/ref/query-panel/number)

### <a />`number-sub`

Subtract a [number](/models/ref/query-panel/number) from another

| Argument | Description                                               |
| :------- | :-------------------------------------------------------- |
| `lhs`    | [number](/models/ref/query-panel/number) to subtract from |
| `rhs`    | [number](/models/ref/query-panel/number) to subtract      |

#### Return Value

Difference of two [numbers](/models/ref/query-panel/number)

### <a />`number-div`

Divide a [number](/models/ref/query-panel/number) by another

| Argument | Description                                           |
| :------- | :---------------------------------------------------- |
| `lhs`    | [number](/models/ref/query-panel/number) to divide    |
| `rhs`    | [number](/models/ref/query-panel/number) to divide by |

#### Return Value

Quotient of two [numbers](/models/ref/query-panel/number)

### <a />`number-floorDiv`

Divide a [number](/models/ref/query-panel/number) by another then rounds down to the nearest whole number

| Argument | Description                                           |
| :------- | :---------------------------------------------------- |
| `lhs`    | [number](/models/ref/query-panel/number) to divide    |
| `rhs`    | [number](/models/ref/query-panel/number) to divide by |

#### Return Value

Truncated quotient of two [numbers](/models/ref/query-panel/number)

### <a />`number-less`

Check if a [number](/models/ref/query-panel/number) is less than another

| Argument | Description                                            |
| :------- | :----------------------------------------------------- |
| `lhs`    | [number](/models/ref/query-panel/number) to compare    |
| `rhs`    | [number](/models/ref/query-panel/number) to compare to |

#### Return Value

Whether the first [number](/models/ref/query-panel/number) is less than the second

### <a />`number-lessEqual`

Check if a [number](/models/ref/query-panel/number) is less than or equal to another

| Argument | Description                                            |
| :------- | :----------------------------------------------------- |
| `lhs`    | [number](/models/ref/query-panel/number) to compare    |
| `rhs`    | [number](/models/ref/query-panel/number) to compare to |

#### Return Value

Whether the first [number](/models/ref/query-panel/number) is less than or equal to the second

### <a />`number-equal`

Determines equality of two values.

| Argument | Description                  |
| :------- | :--------------------------- |
| `lhs`    | The first value to compare.  |
| `rhs`    | The second value to compare. |

#### Return Value

Whether the two values are equal.

### <a />`number-greater`

Check if a [number](/models/ref/query-panel/number) is greater than another

| Argument | Description                                            |
| :------- | :----------------------------------------------------- |
| `lhs`    | [number](/models/ref/query-panel/number) to compare    |
| `rhs`    | [number](/models/ref/query-panel/number) to compare to |

#### Return Value

Whether the first [number](/models/ref/query-panel/number) is greater than the second

### <a />`number-greaterEqual`

Check if a [number](/models/ref/query-panel/number) is greater than or equal to another

| Argument | Description                                            |
| :------- | :----------------------------------------------------- |
| `lhs`    | [number](/models/ref/query-panel/number) to compare    |
| `rhs`    | [number](/models/ref/query-panel/number) to compare to |

#### Return Value

Whether the first [number](/models/ref/query-panel/number) is greater than or equal to the second

### <a />`number-negate`

Negate a [number](/models/ref/query-panel/number)

| Argument | Description      |
| :------- | :--------------- |
| `val`    | Number to negate |

#### Return Value

A [number](/models/ref/query-panel/number)

### <a />`numbers-argmax`

Finds the index of maximum [number](/models/ref/query-panel/number)

| Argument  | Description                                                                                                               |
| :-------- | :------------------------------------------------------------------------------------------------------------------------ |
| `numbers` | *list* of [numbers](/models/ref/query-panel/number) to find the index of maximum [number](/models/ref/query-panel/number) |

#### Return Value

Index of maximum [number](/models/ref/query-panel/number)

### <a />`numbers-argmin`

Finds the index of minimum [number](/models/ref/query-panel/number)

| Argument  | Description                                                                                                               |
| :-------- | :------------------------------------------------------------------------------------------------------------------------ |
| `numbers` | *list* of [numbers](/models/ref/query-panel/number) to find the index of minimum [number](/models/ref/query-panel/number) |

#### Return Value

Index of minimum [number](/models/ref/query-panel/number)

### <a />`numbers-avg`

Average of [numbers](/models/ref/query-panel/number)

| Argument  | Description                                                    |
| :-------- | :------------------------------------------------------------- |
| `numbers` | *list* of [numbers](/models/ref/query-panel/number) to average |

#### Return Value

Average of [numbers](/models/ref/query-panel/number)

### <a />`numbers-max`

Maximum number

| Argument  | Description                                                                                                      |
| :-------- | :--------------------------------------------------------------------------------------------------------------- |
| `numbers` | *list* of [numbers](/models/ref/query-panel/number) to find the maximum [number](/models/ref/query-panel/number) |

#### Return Value

Maximum [number](/models/ref/query-panel/number)

### <a />`numbers-min`

Minimum number

| Argument  | Description                                                                                                      |
| :-------- | :--------------------------------------------------------------------------------------------------------------- |
| `numbers` | *list* of [numbers](/models/ref/query-panel/number) to find the minimum [number](/models/ref/query-panel/number) |

#### Return Value

Minimum [number](/models/ref/query-panel/number)

### <a />`numbers-stddev`

Standard deviation of [numbers](/models/ref/query-panel/number)

| Argument  | Description                                                                             |
| :-------- | :-------------------------------------------------------------------------------------- |
| `numbers` | *list* of [numbers](/models/ref/query-panel/number) to calculate the standard deviation |

#### Return Value

Standard deviation of [numbers](/models/ref/query-panel/number)

### <a />`numbers-sum`

Sum of [numbers](/models/ref/query-panel/number)

| Argument  | Description                                                |
| :-------- | :--------------------------------------------------------- |
| `numbers` | *list* of [numbers](/models/ref/query-panel/number) to sum |

#### Return Value

Sum of [numbers](/models/ref/query-panel/number)

### <a />`number-toString`

Convert a [number](/models/ref/query-panel/number) to a string

| Argument | Description       |
| :------- | :---------------- |
| `in`     | Number to convert |

#### Return Value

String representation of the [number](/models/ref/query-panel/number)

### <a />`number-toTimestamp`

Converts a [number](/models/ref/query-panel/number) to a *timestamp*. Values less than 31536000000 will be converted to seconds, values less than 31536000000000 will be converted to milliseconds, values less than 31536000000000000 will be converted to microseconds, and values less than 31536000000000000000 will be converted to nanoseconds.

| Argument | Description                      |
| :------- | :------------------------------- |
| `val`    | Number to convert to a timestamp |

#### Return Value

Timestamp

### <a />`number-abs`

Calculates the absolute value of a [number](/models/ref/query-panel/number)

| Argument | Description                                |
| :------- | :----------------------------------------- |
| `n`      | A [number](/models/ref/query-panel/number) |

#### Return Value

The absolute value of the [number](/models/ref/query-panel/number)

### <a />`number-cos`

Calculates the cosine of a [number](/models/ref/query-panel/number) in radians

| Argument | Description                        |
| :------- | :--------------------------------- |
| `n`      | Radians to calculate the cosine of |

#### Return Value

Sine of the [number](/models/ref/query-panel/number)

### <a />`number-sin`

Calculates the sine of a [number](/models/ref/query-panel/number) in radians

| Argument | Description                      |
| :------- | :------------------------------- |
| `n`      | Radians to calculate the sine of |

#### Return Value

Sine of the [number](/models/ref/query-panel/number)


# object3D-file
Source: https://docs.wandb.ai/models/ref/query-panel/object-3-d-file



## Chainable Ops

### <a />`asset-file`

Returns the *file* of the asset

| Argument | Description |
| :------- | :---------- |
| `asset`  | The asset   |

#### Return Value

The *file* of the asset

## List Ops

### <a />`asset-file`

Returns the *file* of the asset

| Argument | Description |
| :------- | :---------- |
| `asset`  | The asset   |

#### Return Value

The *file* of the asset


# partitioned-table
Source: https://docs.wandb.ai/models/ref/query-panel/partitioned-table



## Chainable Ops

### <a />`asset-file`

Returns the *file* of the asset

| Argument | Description |
| :------- | :---------- |
| `asset`  | The asset   |

#### Return Value

The *file* of the asset

### <a />`partitionedtable-file`

Returns the *file* of a *partitioned-table*

| Argument           | Description             |
| :----------------- | :---------------------- |
| `partitionedTable` | The *partitioned-table* |

#### Return Value

*file* of the *partitioned-table*

### <a />`partitionedtable-rows`

Returns the rows of a *partitioned-table*

| Argument           | Description                              |
| :----------------- | :--------------------------------------- |
| `partitionedTable` | The *partitioned-table* to get rows from |

#### Return Value

Rows of the *partitioned-table*

## List Ops

### <a />`asset-file`

Returns the *file* of the asset

| Argument | Description |
| :------- | :---------- |
| `asset`  | The asset   |

#### Return Value

The *file* of the asset


# project
Source: https://docs.wandb.ai/models/ref/query-panel/project



## Chainable Ops

### <a />`project-artifact`

Returns the [artifact](/models/ref/query-panel/artifact) for a given name within a [project](/models/ref/query-panel/project)

| Argument       | Description                                                  |
| :------------- | :----------------------------------------------------------- |
| `project`      | A [project](/models/ref/query-panel/project)                 |
| `artifactName` | The name of the [artifact](/models/ref/query-panel/artifact) |

#### Return Value

The [artifact](/models/ref/query-panel/artifact) for a given name within a [project](/models/ref/query-panel/project)

### <a />`project-artifactType`

Returns the [artifactType](/models/ref/query-panel/artifact-type) for a given name within a [project](/models/ref/query-panel/project)

| Argument       | Description                                                           |
| :------------- | :-------------------------------------------------------------------- |
| `project`      | A [project](/models/ref/query-panel/project)                          |
| `artifactType` | The name of the [artifactType](/models/ref/query-panel/artifact-type) |

#### Return Value

The [artifactType](/models/ref/query-panel/artifact-type) for a given name within a [project](/models/ref/query-panel/project)

### <a />`project-artifactTypes`

Returns the [artifactTypes](/models/ref/query-panel/artifact-type) for a [project](/models/ref/query-panel/project)

| Argument  | Description                                  |
| :-------- | :------------------------------------------- |
| `project` | A [project](/models/ref/query-panel/project) |

#### Return Value

The [artifactTypes](/models/ref/query-panel/artifact-type) for a [project](/models/ref/query-panel/project)

### <a />`project-artifactVersion`

Returns the [artifactVersion](/models/ref/query-panel/artifact-version) for a given name and version within a [project](/models/ref/query-panel/project)

| Argument               | Description                                                                          |
| :--------------------- | :----------------------------------------------------------------------------------- |
| `project`              | A [project](/models/ref/query-panel/project)                                         |
| `artifactName`         | The name of the [artifactVersion](/models/ref/query-panel/artifact-version)          |
| `artifactVersionAlias` | The version alias of the [artifactVersion](/models/ref/query-panel/artifact-version) |

#### Return Value

The [artifactVersion](/models/ref/query-panel/artifact-version) for a given name and version within a [project](/models/ref/query-panel/project)

### <a />`project-createdAt`

Returns the creation time of the [project](/models/ref/query-panel/project)

| Argument  | Description                                  |
| :-------- | :------------------------------------------- |
| `project` | A [project](/models/ref/query-panel/project) |

#### Return Value

The creation time of the [project](/models/ref/query-panel/project)

### <a />`project-name`

Returns the name of the [project](/models/ref/query-panel/project)

| Argument  | Description                                  |
| :-------- | :------------------------------------------- |
| `project` | A [project](/models/ref/query-panel/project) |

#### Return Value

The name of the [project](/models/ref/query-panel/project)

### <a />`project-runs`

Returns the [runs](/models/ref/query-panel/run) from a [project](/models/ref/query-panel/project)

| Argument  | Description                                  |
| :-------- | :------------------------------------------- |
| `project` | A [project](/models/ref/query-panel/project) |

#### Return Value

The [runs](/models/ref/query-panel/run) from a [project](/models/ref/query-panel/project)

### <a />`project-updatedAt`

Returns the update time of the [project](/models/ref/query-panel/project)

| Argument  | Description                                  |
| :-------- | :------------------------------------------- |
| `project` | A [project](/models/ref/query-panel/project) |

#### Return Value

The update time of the [project](/models/ref/query-panel/project)

## List Ops

### <a />`project-artifact`

Returns the [artifact](/models/ref/query-panel/artifact) for a given name within a [project](/models/ref/query-panel/project)

| Argument       | Description                                                  |
| :------------- | :----------------------------------------------------------- |
| `project`      | A [project](/models/ref/query-panel/project)                 |
| `artifactName` | The name of the [artifact](/models/ref/query-panel/artifact) |

#### Return Value

The [artifact](/models/ref/query-panel/artifact) for a given name within a [project](/models/ref/query-panel/project)

### <a />`project-artifactType`

Returns the [artifactType](/models/ref/query-panel/artifact-type) for a given name within a [project](/models/ref/query-panel/project)

| Argument       | Description                                                           |
| :------------- | :-------------------------------------------------------------------- |
| `project`      | A [project](/models/ref/query-panel/project)                          |
| `artifactType` | The name of the [artifactType](/models/ref/query-panel/artifact-type) |

#### Return Value

The [artifactType](/models/ref/query-panel/artifact-type) for a given name within a [project](/models/ref/query-panel/project)

### <a />`project-artifactTypes`

Returns the [artifactTypes](/models/ref/query-panel/artifact-type) for a [project](/models/ref/query-panel/project)

| Argument  | Description                                  |
| :-------- | :------------------------------------------- |
| `project` | A [project](/models/ref/query-panel/project) |

#### Return Value

The [artifactTypes](/models/ref/query-panel/artifact-type) for a [project](/models/ref/query-panel/project)

### <a />`project-artifactVersion`

Returns the [artifactVersion](/models/ref/query-panel/artifact-version) for a given name and version within a [project](/models/ref/query-panel/project)

| Argument               | Description                                                                          |
| :--------------------- | :----------------------------------------------------------------------------------- |
| `project`              | A [project](/models/ref/query-panel/project)                                         |
| `artifactName`         | The name of the [artifactVersion](/models/ref/query-panel/artifact-version)          |
| `artifactVersionAlias` | The version alias of the [artifactVersion](/models/ref/query-panel/artifact-version) |

#### Return Value

The [artifactVersion](/models/ref/query-panel/artifact-version) for a given name and version within a [project](/models/ref/query-panel/project)

### <a />`project-createdAt`

Returns the creation time of the [project](/models/ref/query-panel/project)

| Argument  | Description                                  |
| :-------- | :------------------------------------------- |
| `project` | A [project](/models/ref/query-panel/project) |

#### Return Value

The creation time of the [project](/models/ref/query-panel/project)

### <a />`project-name`

Returns the name of the [project](/models/ref/query-panel/project)

| Argument  | Description                                  |
| :-------- | :------------------------------------------- |
| `project` | A [project](/models/ref/query-panel/project) |

#### Return Value

The name of the [project](/models/ref/query-panel/project)

### <a />`project-runs`

Returns the [runs](/models/ref/query-panel/run) from a [project](/models/ref/query-panel/project)

| Argument  | Description                                  |
| :-------- | :------------------------------------------- |
| `project` | A [project](/models/ref/query-panel/project) |

#### Return Value

The [runs](/models/ref/query-panel/run) from a [project](/models/ref/query-panel/project)

### <a />`project-updatedAt`

Returns the update time of the [project](/models/ref/query-panel/project)

| Argument  | Description                                  |
| :-------- | :------------------------------------------- |
| `project` | A [project](/models/ref/query-panel/project) |

#### Return Value

The update time of the [project](/models/ref/query-panel/project)


# pytorch-model-file
Source: https://docs.wandb.ai/models/ref/query-panel/pytorch-model-file



## Chainable Ops

### <a />`asset-file`

Returns the *file* of the asset

| Argument | Description |
| :------- | :---------- |
| `asset`  | The asset   |

#### Return Value

The *file* of the asset

## List Ops

### <a />`asset-file`

Returns the *file* of the asset

| Argument | Description |
| :------- | :---------- |
| `asset`  | The asset   |

#### Return Value

The *file* of the asset


# run
Source: https://docs.wandb.ai/models/ref/query-panel/run



## Chainable Ops

### <a />`run-config`

Returns the config *typedDict* of the [run](/models/ref/query-panel/run)

| Argument | Description                          |
| :------- | :----------------------------------- |
| `run`    | A [run](/models/ref/query-panel/run) |

#### Return Value

The config *typedDict* of the [run](/models/ref/query-panel/run)

### <a />`run-createdAt`

Returns the created at datetime of the [run](/models/ref/query-panel/run)

| Argument | Description                          |
| :------- | :----------------------------------- |
| `run`    | A [run](/models/ref/query-panel/run) |

#### Return Value

The created at datetime of the [run](/models/ref/query-panel/run)

### <a />`run-heartbeatAt`

Returns the last heartbeat datetime of the [run](/models/ref/query-panel/run)

| Argument | Description                          |
| :------- | :----------------------------------- |
| `run`    | A [run](/models/ref/query-panel/run) |

#### Return Value

The last heartbeat datetime of the [run](/models/ref/query-panel/run)

### <a />`run-history`

Returns the log history of the [run](/models/ref/query-panel/run)

| Argument | Description                          |
| :------- | :----------------------------------- |
| `run`    | A [run](/models/ref/query-panel/run) |

#### Return Value

The log history of the [run](/models/ref/query-panel/run)

### <a />`run-jobType`

Returns the job type of the [run](/models/ref/query-panel/run)

| Argument | Description                          |
| :------- | :----------------------------------- |
| `run`    | A [run](/models/ref/query-panel/run) |

#### Return Value

The job type of the [run](/models/ref/query-panel/run)

### <a />`run-loggedArtifactVersion`

Returns the [artifactVersion](/models/ref/query-panel/artifact-version) logged by the [run](/models/ref/query-panel/run) for a given name and alias

| Argument              | Description                                                                       |
| :-------------------- | :-------------------------------------------------------------------------------- |
| `run`                 | A [run](/models/ref/query-panel/run)                                              |
| `artifactVersionName` | The name:alias of the [artifactVersion](/models/ref/query-panel/artifact-version) |

#### Return Value

The [artifactVersion](/models/ref/query-panel/artifact-version) logged by the [run](/models/ref/query-panel/run) for a given name and alias

### <a />`run-loggedArtifactVersions`

Returns all of the [artifactVersions](/models/ref/query-panel/artifact-version) logged by the [run](/models/ref/query-panel/run)

| Argument | Description                          |
| :------- | :----------------------------------- |
| `run`    | A [run](/models/ref/query-panel/run) |

#### Return Value

The [artifactVersions](/models/ref/query-panel/artifact-version) logged by the [run](/models/ref/query-panel/run)

### <a />`run-name`

Returns the name of the [run](/models/ref/query-panel/run)

| Argument | Description                          |
| :------- | :----------------------------------- |
| `run`    | A [run](/models/ref/query-panel/run) |

#### Return Value

The name of the [run](/models/ref/query-panel/run)

### <a />`run-runtime`

Returns the runtime in seconds of the [run](/models/ref/query-panel/run)

| Argument | Description                          |
| :------- | :----------------------------------- |
| `run`    | A [run](/models/ref/query-panel/run) |

#### Return Value

The runtime in seconds of the [run](/models/ref/query-panel/run)

### <a />`run-summary`

Returns the summary *typedDict* of the [run](/models/ref/query-panel/run)

| Argument | Description                          |
| :------- | :----------------------------------- |
| `run`    | A [run](/models/ref/query-panel/run) |

#### Return Value

The summary *typedDict* of the [run](/models/ref/query-panel/run)

### <a />`run-usedArtifactVersions`

Returns all of the [artifactVersions](/models/ref/query-panel/artifact-version) used by the [run](/models/ref/query-panel/run)

| Argument | Description                          |
| :------- | :----------------------------------- |
| `run`    | A [run](/models/ref/query-panel/run) |

#### Return Value

The [artifactVersions](/models/ref/query-panel/artifact-version) used by the [run](/models/ref/query-panel/run)

### <a />`run-user`

Returns the [user](/models/ref/query-panel/user) of the [run](/models/ref/query-panel/run)

| Argument | Description                          |
| :------- | :----------------------------------- |
| `run`    | A [run](/models/ref/query-panel/run) |

#### Return Value

The [user](/models/ref/query-panel/user) of the [run](/models/ref/query-panel/run)

## List Ops

### <a />`run-config`

Returns the config *typedDict* of the [run](/models/ref/query-panel/run)

| Argument | Description                          |
| :------- | :----------------------------------- |
| `run`    | A [run](/models/ref/query-panel/run) |

#### Return Value

The config *typedDict* of the [run](/models/ref/query-panel/run)

### <a />`run-createdAt`

Returns the created at datetime of the [run](/models/ref/query-panel/run)

| Argument | Description                          |
| :------- | :----------------------------------- |
| `run`    | A [run](/models/ref/query-panel/run) |

#### Return Value

The created at datetime of the [run](/models/ref/query-panel/run)

### <a />`run-heartbeatAt`

Returns the last heartbeat datetime of the [run](/models/ref/query-panel/run)

| Argument | Description                          |
| :------- | :----------------------------------- |
| `run`    | A [run](/models/ref/query-panel/run) |

#### Return Value

The last heartbeat datetime of the [run](/models/ref/query-panel/run)

### <a />`run-history`

Returns the log history of the [run](/models/ref/query-panel/run)

| Argument | Description                          |
| :------- | :----------------------------------- |
| `run`    | A [run](/models/ref/query-panel/run) |

#### Return Value

The log history of the [run](/models/ref/query-panel/run)

### <a />`run-jobType`

Returns the job type of the [run](/models/ref/query-panel/run)

| Argument | Description                          |
| :------- | :----------------------------------- |
| `run`    | A [run](/models/ref/query-panel/run) |

#### Return Value

The job type of the [run](/models/ref/query-panel/run)

### <a />`run-loggedArtifactVersion`

Returns the [artifactVersion](/models/ref/query-panel/artifact-version) logged by the [run](/models/ref/query-panel/run) for a given name and alias

| Argument              | Description                                                                       |
| :-------------------- | :-------------------------------------------------------------------------------- |
| `run`                 | A [run](/models/ref/query-panel/run)                                              |
| `artifactVersionName` | The name:alias of the [artifactVersion](/models/ref/query-panel/artifact-version) |

#### Return Value

The [artifactVersion](/models/ref/query-panel/artifact-version) logged by the [run](/models/ref/query-panel/run) for a given name and alias

### <a />`run-loggedArtifactVersions`

Returns all of the [artifactVersions](/models/ref/query-panel/artifact-version) logged by the [run](/models/ref/query-panel/run)

| Argument | Description                          |
| :------- | :----------------------------------- |
| `run`    | A [run](/models/ref/query-panel/run) |

#### Return Value

The [artifactVersions](/models/ref/query-panel/artifact-version) logged by the [run](/models/ref/query-panel/run)

### <a />`run-name`

Returns the name of the [run](/models/ref/query-panel/run)

| Argument | Description                          |
| :------- | :----------------------------------- |
| `run`    | A [run](/models/ref/query-panel/run) |

#### Return Value

The name of the [run](/models/ref/query-panel/run)

### <a />`run-runtime`

Returns the runtime in seconds of the [run](/models/ref/query-panel/run)

| Argument | Description                          |
| :------- | :----------------------------------- |
| `run`    | A [run](/models/ref/query-panel/run) |

#### Return Value

The runtime in seconds of the [run](/models/ref/query-panel/run)

### <a />`run-summary`

Returns the summary *typedDict* of the [run](/models/ref/query-panel/run)

| Argument | Description                          |
| :------- | :----------------------------------- |
| `run`    | A [run](/models/ref/query-panel/run) |

#### Return Value

The summary *typedDict* of the [run](/models/ref/query-panel/run)

### <a />`run-usedArtifactVersions`

Returns all of the [artifactVersions](/models/ref/query-panel/artifact-version) used by the [run](/models/ref/query-panel/run)

| Argument | Description                          |
| :------- | :----------------------------------- |
| `run`    | A [run](/models/ref/query-panel/run) |

#### Return Value

The [artifactVersions](/models/ref/query-panel/artifact-version) used by the [run](/models/ref/query-panel/run)

### <a />`run-user`

Returns the [user](/models/ref/query-panel/user) of the [run](/models/ref/query-panel/run)

| Argument | Description                          |
| :------- | :----------------------------------- |
| `run`    | A [run](/models/ref/query-panel/run) |

#### Return Value

The [user](/models/ref/query-panel/user) of the [run](/models/ref/query-panel/run)


# string
Source: https://docs.wandb.ai/models/ref/query-panel/string



## Chainable Ops

### <a />`string-notEqual`

Determines inequality of two values.

| Argument | Description                  |
| :------- | :--------------------------- |
| `lhs`    | The first value to compare.  |
| `rhs`    | The second value to compare. |

#### Return Value

Whether the two values are not equal.

### <a />`string-add`

Concatenates two [strings](/models/ref/query-panel/string)

| Argument | Description                                         |
| :------- | :-------------------------------------------------- |
| `lhs`    | The first [string](/models/ref/query-panel/string)  |
| `rhs`    | The second [string](/models/ref/query-panel/string) |

#### Return Value

The concatenated [string](/models/ref/query-panel/string)

### <a />`string-equal`

Determines equality of two values.

| Argument | Description                  |
| :------- | :--------------------------- |
| `lhs`    | The first value to compare.  |
| `rhs`    | The second value to compare. |

#### Return Value

Whether the two values are equal.

### <a />`string-append`

Appends a suffix to a [string](/models/ref/query-panel/string)

| Argument | Description                                               |
| :------- | :-------------------------------------------------------- |
| `str`    | The [string](/models/ref/query-panel/string) to append to |
| `suffix` | The suffix to append                                      |

#### Return Value

The [string](/models/ref/query-panel/string) with the suffix appended

### <a />`string-contains`

Checks if a [string](/models/ref/query-panel/string) contains a substring

| Argument | Description                                           |
| :------- | :---------------------------------------------------- |
| `str`    | The [string](/models/ref/query-panel/string) to check |
| `sub`    | The substring to check for                            |

#### Return Value

Whether the [string](/models/ref/query-panel/string) contains the substring

### <a />`string-endsWith`

Checks if a [string](/models/ref/query-panel/string) ends with a suffix

| Argument | Description                                           |
| :------- | :---------------------------------------------------- |
| `str`    | The [string](/models/ref/query-panel/string) to check |
| `suffix` | The suffix to check for                               |

#### Return Value

Whether the [string](/models/ref/query-panel/string) ends with the suffix

### <a />`string-findAll`

Finds all occurrences of a substring in a [string](/models/ref/query-panel/string)

| Argument | Description                                                                          |
| :------- | :----------------------------------------------------------------------------------- |
| `str`    | The [string](/models/ref/query-panel/string) to find occurrences of the substring in |
| `sub`    | The substring to find                                                                |

#### Return Value

The *list* of indices of the substring in the [string](/models/ref/query-panel/string)

### <a />`string-isAlnum`

Checks if a [string](/models/ref/query-panel/string) is alphanumeric

| Argument | Description                                           |
| :------- | :---------------------------------------------------- |
| `str`    | The [string](/models/ref/query-panel/string) to check |

#### Return Value

Whether the [string](/models/ref/query-panel/string) is alphanumeric

### <a />`string-isAlpha`

Checks if a [string](/models/ref/query-panel/string) is alphabetic

| Argument | Description                                           |
| :------- | :---------------------------------------------------- |
| `str`    | The [string](/models/ref/query-panel/string) to check |

#### Return Value

Whether the [string](/models/ref/query-panel/string) is alphabetic

### <a />`string-isNumeric`

Checks if a [string](/models/ref/query-panel/string) is numeric

| Argument | Description                                           |
| :------- | :---------------------------------------------------- |
| `str`    | The [string](/models/ref/query-panel/string) to check |

#### Return Value

Whether the [string](/models/ref/query-panel/string) is numeric

### <a />`string-lStrip`

Strip leading whitespace

| Argument | Description                                            |
| :------- | :----------------------------------------------------- |
| `str`    | The [string](/models/ref/query-panel/string) to strip. |

#### Return Value

The stripped [string](/models/ref/query-panel/string).

### <a />`string-len`

Returns the length of a [string](/models/ref/query-panel/string)

| Argument | Description                                           |
| :------- | :---------------------------------------------------- |
| `str`    | The [string](/models/ref/query-panel/string) to check |

#### Return Value

The length of the [string](/models/ref/query-panel/string)

### <a />`string-lower`

Converts a [string](/models/ref/query-panel/string) to lowercase

| Argument | Description                                                          |
| :------- | :------------------------------------------------------------------- |
| `str`    | The [string](/models/ref/query-panel/string) to convert to lowercase |

#### Return Value

The lowercase [string](/models/ref/query-panel/string)

### <a />`string-partition`

Partitions a [string](/models/ref/query-panel/string) into a *list* of the [strings](/models/ref/query-panel/string)

| Argument | Description                                           |
| :------- | :---------------------------------------------------- |
| `str`    | The [string](/models/ref/query-panel/string) to split |
| `sep`    | The separator to split on                             |

#### Return Value

A *list* of [strings](/models/ref/query-panel/string): the [string](/models/ref/query-panel/string) before the separator, the separator, and the [string](/models/ref/query-panel/string) after the separator

### <a />`string-prepend`

Prepends a prefix to a [string](/models/ref/query-panel/string)

| Argument | Description                                                |
| :------- | :--------------------------------------------------------- |
| `str`    | The [string](/models/ref/query-panel/string) to prepend to |
| `prefix` | The prefix to prepend                                      |

#### Return Value

The [string](/models/ref/query-panel/string) with the prefix prepended

### <a />`string-rStrip`

Strip trailing whitespace

| Argument | Description                                            |
| :------- | :----------------------------------------------------- |
| `str`    | The [string](/models/ref/query-panel/string) to strip. |

#### Return Value

The stripped [string](/models/ref/query-panel/string).

### <a />`string-replace`

Replaces all occurrences of a substring in a [string](/models/ref/query-panel/string)

| Argument | Description                                                         |
| :------- | :------------------------------------------------------------------ |
| `str`    | The [string](/models/ref/query-panel/string) to replace contents of |
| `sub`    | The substring to replace                                            |
| `newSub` | The substring to replace the old substring with                     |

#### Return Value

The [string](/models/ref/query-panel/string) with the replacements

### <a />`string-slice`

Slices a [string](/models/ref/query-panel/string) into a substring based on beginning and end indices

| Argument | Description                                           |
| :------- | :---------------------------------------------------- |
| `str`    | The [string](/models/ref/query-panel/string) to slice |
| `begin`  | The beginning index of the substring                  |
| `end`    | The ending index of the substring                     |

#### Return Value

The substring

### <a />`string-split`

Splits a [string](/models/ref/query-panel/string) into a *list* of [strings](/models/ref/query-panel/string)

| Argument | Description                                           |
| :------- | :---------------------------------------------------- |
| `str`    | The [string](/models/ref/query-panel/string) to split |
| `sep`    | The separator to split on                             |

#### Return Value

The *list* of [strings](/models/ref/query-panel/string)

### <a />`string-startsWith`

Checks if a [string](/models/ref/query-panel/string) starts with a prefix

| Argument | Description                                           |
| :------- | :---------------------------------------------------- |
| `str`    | The [string](/models/ref/query-panel/string) to check |
| `prefix` | The prefix to check for                               |

#### Return Value

Whether the [string](/models/ref/query-panel/string) starts with the prefix

### <a />`string-strip`

Strip whitespace from both ends of a [string](/models/ref/query-panel/string).

| Argument | Description                                            |
| :------- | :----------------------------------------------------- |
| `str`    | The [string](/models/ref/query-panel/string) to strip. |

#### Return Value

The stripped [string](/models/ref/query-panel/string).

### <a />`string-upper`

Converts a [string](/models/ref/query-panel/string) to uppercase

| Argument | Description                                                          |
| :------- | :------------------------------------------------------------------- |
| `str`    | The [string](/models/ref/query-panel/string) to convert to uppercase |

#### Return Value

The uppercase [string](/models/ref/query-panel/string)

### <a />`string-levenshtein`

Calculates the Levenshtein distance between two [strings](/models/ref/query-panel/string).

| Argument | Description                                          |
| :------- | :--------------------------------------------------- |
| `str1`   | The first [string](/models/ref/query-panel/string).  |
| `str2`   | The second [string](/models/ref/query-panel/string). |

#### Return Value

The Levenshtein distance between the two [strings](/models/ref/query-panel/string).

### <a />`local-sqlconnection`

hello

| Argument | Description |
| :------- | :---------- |
| `conf`   | hello       |
| `size`   | hello       |

#### Return Value

hello

## List Ops

### <a />`string-notEqual`

Determines inequality of two values.

| Argument | Description                  |
| :------- | :--------------------------- |
| `lhs`    | The first value to compare.  |
| `rhs`    | The second value to compare. |

#### Return Value

Whether the two values are not equal.

### <a />`string-add`

Concatenates two [strings](/models/ref/query-panel/string)

| Argument | Description                                         |
| :------- | :-------------------------------------------------- |
| `lhs`    | The first [string](/models/ref/query-panel/string)  |
| `rhs`    | The second [string](/models/ref/query-panel/string) |

#### Return Value

The concatenated [string](/models/ref/query-panel/string)

### <a />`string-equal`

Determines equality of two values.

| Argument | Description                  |
| :------- | :--------------------------- |
| `lhs`    | The first value to compare.  |
| `rhs`    | The second value to compare. |

#### Return Value

Whether the two values are equal.

### <a />`string-append`

Appends a suffix to a [string](/models/ref/query-panel/string)

| Argument | Description                                               |
| :------- | :-------------------------------------------------------- |
| `str`    | The [string](/models/ref/query-panel/string) to append to |
| `suffix` | The suffix to append                                      |

#### Return Value

The [string](/models/ref/query-panel/string) with the suffix appended

### <a />`string-contains`

Checks if a [string](/models/ref/query-panel/string) contains a substring

| Argument | Description                                           |
| :------- | :---------------------------------------------------- |
| `str`    | The [string](/models/ref/query-panel/string) to check |
| `sub`    | The substring to check for                            |

#### Return Value

Whether the [string](/models/ref/query-panel/string) contains the substring

### <a />`string-endsWith`

Checks if a [string](/models/ref/query-panel/string) ends with a suffix

| Argument | Description                                           |
| :------- | :---------------------------------------------------- |
| `str`    | The [string](/models/ref/query-panel/string) to check |
| `suffix` | The suffix to check for                               |

#### Return Value

Whether the [string](/models/ref/query-panel/string) ends with the suffix

### <a />`string-findAll`

Finds all occurrences of a substring in a [string](/models/ref/query-panel/string)

| Argument | Description                                                                          |
| :------- | :----------------------------------------------------------------------------------- |
| `str`    | The [string](/models/ref/query-panel/string) to find occurrences of the substring in |
| `sub`    | The substring to find                                                                |

#### Return Value

The *list* of indices of the substring in the [string](/models/ref/query-panel/string)

### <a />`string-isAlnum`

Checks if a [string](/models/ref/query-panel/string) is alphanumeric

| Argument | Description                                           |
| :------- | :---------------------------------------------------- |
| `str`    | The [string](/models/ref/query-panel/string) to check |

#### Return Value

Whether the [string](/models/ref/query-panel/string) is alphanumeric

### <a />`string-isAlpha`

Checks if a [string](/models/ref/query-panel/string) is alphabetic

| Argument | Description                                           |
| :------- | :---------------------------------------------------- |
| `str`    | The [string](/models/ref/query-panel/string) to check |

#### Return Value

Whether the [string](/models/ref/query-panel/string) is alphabetic

### <a />`string-isNumeric`

Checks if a [string](/models/ref/query-panel/string) is numeric

| Argument | Description                                           |
| :------- | :---------------------------------------------------- |
| `str`    | The [string](/models/ref/query-panel/string) to check |

#### Return Value

Whether the [string](/models/ref/query-panel/string) is numeric

### <a />`string-lStrip`

Strip leading whitespace

| Argument | Description                                            |
| :------- | :----------------------------------------------------- |
| `str`    | The [string](/models/ref/query-panel/string) to strip. |

#### Return Value

The stripped [string](/models/ref/query-panel/string).

### <a />`string-len`

Returns the length of a [string](/models/ref/query-panel/string)

| Argument | Description                                           |
| :------- | :---------------------------------------------------- |
| `str`    | The [string](/models/ref/query-panel/string) to check |

#### Return Value

The length of the [string](/models/ref/query-panel/string)

### <a />`string-lower`

Converts a [string](/models/ref/query-panel/string) to lowercase

| Argument | Description                                                          |
| :------- | :------------------------------------------------------------------- |
| `str`    | The [string](/models/ref/query-panel/string) to convert to lowercase |

#### Return Value

The lowercase [string](/models/ref/query-panel/string)

### <a />`string-partition`

Partitions a [string](/models/ref/query-panel/string) into a *list* of the [strings](/models/ref/query-panel/string)

| Argument | Description                                           |
| :------- | :---------------------------------------------------- |
| `str`    | The [string](/models/ref/query-panel/string) to split |
| `sep`    | The separator to split on                             |

#### Return Value

A *list* of [strings](/models/ref/query-panel/string): the [string](/models/ref/query-panel/string) before the separator, the separator, and the [string](/models/ref/query-panel/string) after the separator

### <a />`string-prepend`

Prepends a prefix to a [string](/models/ref/query-panel/string)

| Argument | Description                                                |
| :------- | :--------------------------------------------------------- |
| `str`    | The [string](/models/ref/query-panel/string) to prepend to |
| `prefix` | The prefix to prepend                                      |

#### Return Value

The [string](/models/ref/query-panel/string) with the prefix prepended

### <a />`string-rStrip`

Strip trailing whitespace

| Argument | Description                                            |
| :------- | :----------------------------------------------------- |
| `str`    | The [string](/models/ref/query-panel/string) to strip. |

#### Return Value

The stripped [string](/models/ref/query-panel/string).

### <a />`string-replace`

Replaces all occurrences of a substring in a [string](/models/ref/query-panel/string)

| Argument | Description                                                         |
| :------- | :------------------------------------------------------------------ |
| `str`    | The [string](/models/ref/query-panel/string) to replace contents of |
| `sub`    | The substring to replace                                            |
| `newSub` | The substring to replace the old substring with                     |

#### Return Value

The [string](/models/ref/query-panel/string) with the replacements

### <a />`string-slice`

Slices a [string](/models/ref/query-panel/string) into a substring based on beginning and end indices

| Argument | Description                                           |
| :------- | :---------------------------------------------------- |
| `str`    | The [string](/models/ref/query-panel/string) to slice |
| `begin`  | The beginning index of the substring                  |
| `end`    | The ending index of the substring                     |

#### Return Value

The substring

### <a />`string-split`

Splits a [string](/models/ref/query-panel/string) into a *list* of [strings](/models/ref/query-panel/string)

| Argument | Description                                           |
| :------- | :---------------------------------------------------- |
| `str`    | The [string](/models/ref/query-panel/string) to split |
| `sep`    | The separator to split on                             |

#### Return Value

The *list* of [strings](/models/ref/query-panel/string)

### <a />`string-startsWith`

Checks if a [string](/models/ref/query-panel/string) starts with a prefix

| Argument | Description                                           |
| :------- | :---------------------------------------------------- |
| `str`    | The [string](/models/ref/query-panel/string) to check |
| `prefix` | The prefix to check for                               |

#### Return Value

Whether the [string](/models/ref/query-panel/string) starts with the prefix

### <a />`string-strip`

Strip whitespace from both ends of a [string](/models/ref/query-panel/string).

| Argument | Description                                            |
| :------- | :----------------------------------------------------- |
| `str`    | The [string](/models/ref/query-panel/string) to strip. |

#### Return Value

The stripped [string](/models/ref/query-panel/string).

### <a />`string-upper`

Converts a [string](/models/ref/query-panel/string) to uppercase

| Argument | Description                                                          |
| :------- | :------------------------------------------------------------------- |
| `str`    | The [string](/models/ref/query-panel/string) to convert to uppercase |

#### Return Value

The uppercase [string](/models/ref/query-panel/string)

### <a />`string-levenshtein`

Calculates the Levenshtein distance between two [strings](/models/ref/query-panel/string).

| Argument | Description                                          |
| :------- | :--------------------------------------------------- |
| `str1`   | The first [string](/models/ref/query-panel/string).  |
| `str2`   | The second [string](/models/ref/query-panel/string). |

#### Return Value

The Levenshtein distance between the two [strings](/models/ref/query-panel/string).


# table
Source: https://docs.wandb.ai/models/ref/query-panel/table



## Chainable Ops

### <a />`asset-file`

Returns the *file* of the asset

| Argument | Description |
| :------- | :---------- |
| `asset`  | The asset   |

#### Return Value

The *file* of the asset

### <a />`table-rows`

Returns the rows of a *table*

| Argument | Description |
| :------- | :---------- |
| `table`  | A *table*   |

#### Return Value

The rows of the *table*

## List Ops

### <a />`asset-file`

Returns the *file* of the asset

| Argument | Description |
| :------- | :---------- |
| `asset`  | The asset   |

#### Return Value

The *file* of the asset

### <a />`table-rows`

Returns the rows of a *table*

| Argument | Description |
| :------- | :---------- |
| `table`  | A *table*   |

#### Return Value

The rows of the *table*


# user
Source: https://docs.wandb.ai/models/ref/query-panel/user



## Chainable Ops

### <a />`user-username`

Returns the username of the [user](/models/ref/query-panel/user)

| Argument | Description                            |
| :------- | :------------------------------------- |
| `user`   | A [user](/models/ref/query-panel/user) |

#### Return Value

The username of the [user](/models/ref/query-panel/user)

## List Ops

### <a />`user-username`

Returns the username of the [user](/models/ref/query-panel/user)

| Argument | Description                            |
| :------- | :------------------------------------- |
| `user`   | A [user](/models/ref/query-panel/user) |

#### Return Value

The username of the [user](/models/ref/query-panel/user)


# video-file
Source: https://docs.wandb.ai/models/ref/query-panel/video-file



## Chainable Ops

### <a />`asset-file`

Returns the *file* of the asset

| Argument | Description |
| :------- | :---------- |
| `asset`  | The asset   |

#### Return Value

The *file* of the asset

## List Ops

### <a />`asset-file`

Returns the *file* of the asset

| Argument | Description |
| :------- | :---------- |
| `asset`  | The asset   |

#### Return Value

The *file* of the asset


# Reports
Source: https://docs.wandb.ai/models/ref/wandb_workspaces/reports



<GitHubLink />

<Note>
  W\&B Report and Workspace API is in Public Preview.
</Note>

Python library for programmatically working with W\&B Reports API.

```python theme={null}
import wandb_workspaces.reports.v2 as wr

report = wr.Report(
     entity="entity",
     project="project",
     title="An amazing title",
     description="A descriptive description.",
)

blocks = [
     wr.PanelGrid(
         panels=[
             wr.LinePlot(x="time", y="velocity"),
             wr.ScatterPlot(x="time", y="acceleration"),
         ]
     )
]

report.blocks = blocks
report.save()
```

## <kbd>class</kbd> `BarPlot`

A panel object that shows a 2D bar plot.

**Attributes:**

* `title` (Optional\[str]):  The text that appears at the top of the plot.
* `metrics` (LList\[MetricType]):  orientation Literal\["v", "h"]: The orientation of the bar plot.  Set to either vertical ("v") or horizontal ("h"). Defaults to horizontal ("h").
* `range_x` (Tuple\[float | None, float | None]):  Tuple that specifies the range of the x-axis.
* `title_x` (Optional\[str]):  The label of the x-axis.
* `title_y` (Optional\[str]):  The label of the y-axis.
* `groupby` (Optional\[str]):  Group runs based on a metric logged to your W\&B project that the  report pulls information from.
* `groupby_aggfunc` (Optional\[GroupAgg]):  Aggregate runs with specified  function. Options include "mean", "min", "max", "median", "sum", "samples", or `None`.
* `groupby_rangefunc` (Optional\[GroupArea]):   Group runs based on a range. Options  include "minmax", "stddev", "stderr", "none", "samples", or `None`.
* `max_runs_to_show` (Optional\[int]):  The maximum number of runs to show on the plot.
* `max_bars_to_show` (Optional\[int]):  The maximum number of bars to show on the bar plot.
* `custom_expressions` (Optional\[LList\[str]]):  A list of custom expressions to be used in the bar plot.
* `legend_template` (Optional\[str]):  The template for the legend.
* `font_size` ( Optional\[FontSize]):  The size of the line plot's font.  Options include "small", "medium", "large", "auto", or `None`.
* `line_titles` (Optional\[dict]):  The titles of the lines. The keys are the line names and the values are the titles.
* `line_colors` (Optional\[dict]):  The colors of the lines. The keys are the line names and the values are the colors.
* `aggregate` (Optional\[bool]):  If set to `True`, aggregate the data.

***

## <kbd>class</kbd> `BlockQuote`

A block of quoted text.

**Attributes:**

* `text` (str):  The text of the block quote.

***

## <kbd>class</kbd> `CalloutBlock`

A block of callout text.

**Attributes:**

* `text` (str):  The callout text.

***

## <kbd>class</kbd> `CheckedList`

A list of items with checkboxes. Add one or more `CheckedListItem` within `CheckedList`.

**Attributes:**

* `items` (LList\[CheckedListItem]):  A list of one or more `CheckedListItem` objects.

***

## <kbd>class</kbd> `CheckedListItem`

A list item with a checkbox. Add one or more `CheckedListItem` within `CheckedList`.

**Attributes:**

* `text` (str):  The text of the list item.
* `checked` (bool):  Whether the checkbox is checked. By default, set to `False`.

***

## <kbd>class</kbd> `CodeBlock`

A block of code.

**Attributes:**

* `code` (str):  The code in the block.
* `language` (Optional\[Language]):  The language of the code. Language specified  is used for syntax highlighting. By default, set to "python". Options include  'javascript', 'python', 'css', 'json', 'html', 'markdown', 'yaml'.

***

## <kbd>class</kbd> `CodeComparer`

A panel object that compares the code between two different runs.

**Attributes:**

* `diff` (Literal\['split', 'unified']):  How to display code differences.  Options include "split" and "unified".

***

## <kbd>class</kbd> `Config`

Metrics logged to a run's config object. Config objects are commonly logged using `run.config[name] = ...` or passing a config as a dictionary of key-value pairs, where the key is the name of the metric and the value is the value of that metric.

**Attributes:**

* `name` (str):  The name of the metric.

***

## <kbd>class</kbd> `CustomChart`

A panel that shows a custom chart. The chart is defined by a weave query.

**Attributes:**

* `query` (dict):  The query that defines the custom chart. The key is the name of the field, and the value is the query.
* `chart_name` (str):  The title of the custom chart.
* `chart_fields` (dict):  Key-value pairs that define the axis of the  plot. Where the key is the label, and the value is the metric.
* `chart_strings` (dict):  Key-value pairs that define the strings in the chart.

***

### <kbd>classmethod</kbd> `from_table`

```python theme={null}
from_table(
    table_name: str,
    chart_fields: dict = None,
    chart_strings: dict = None
)
```

Create a custom chart from a table.

**Arguments:**

* `table_name` (str):  The name of the table.
* `chart_fields` (dict):  The fields to display in the chart.
* `chart_strings` (dict):  The strings to display in the chart.

***

## <kbd>class</kbd> `Gallery`

A block that renders a gallery of reports and URLs.

**Attributes:**

* `items` (List\[Union\[`GalleryReport`, `GalleryURL`]]):  A list of  `GalleryReport` and `GalleryURL` objects.

***

## <kbd>class</kbd> `GalleryReport`

A reference to a report in the gallery.

**Attributes:**

* `report_id` (str):  The ID of the report.

***

## <kbd>class</kbd> `GalleryURL`

A URL to an external resource.

**Attributes:**

* `url` (str):  The URL of the resource.
* `title` (Optional\[str]):  The title of the resource.
* `description` (Optional\[str]):  The description of the resource.
* `image_url` (Optional\[str]):  The URL of an image to display.

***

## <kbd>class</kbd> `GradientPoint`

A point in a gradient.

**Attributes:**

* `color`:  The color of the point.
* `offset`:  The position of the point in the gradient. The value should be between 0 and 100.

***

## <kbd>class</kbd> `H1`

An H1 heading with the text specified.

**Attributes:**

* `text` (str):  The text of the heading.
* `collapsed_blocks` (Optional\[LList\["BlockTypes"]]):  The blocks to show when the heading is collapsed.

***

## <kbd>class</kbd> `H2`

An H2 heading with the text specified.

**Attributes:**

* `text` (str):  The text of the heading.
* `collapsed_blocks` (Optional\[LList\["BlockTypes"]]):  One or more blocks to  show when the heading is collapsed.

***

## <kbd>class</kbd> `H3`

An H3 heading with the text specified.

**Attributes:**

* `text` (str):  The text of the heading.
* `collapsed_blocks` (Optional\[LList\["BlockTypes"]]):  One or more blocks to  show when the heading is collapsed.

***

## <kbd>class</kbd> `Heading`

***

## <kbd>class</kbd> `HorizontalRule`

HTML horizontal line.

***

## <kbd>class</kbd> `Image`

A block that renders an image.

**Attributes:**

* `url` (str):  The URL of the image.
* `caption` (str):  The caption of the image.  Caption appears underneath the image.

***

## <kbd>class</kbd> `InlineCode`

Inline code. Does not add newline character after code.

**Attributes:**

* `text` (str):  The code you want to appear in the report.

***

## <kbd>class</kbd> `InlineLatex`

Inline LaTeX markdown. Does not add newline character after the LaTeX markdown.

**Attributes:**

* `text` (str):  LaTeX markdown you want to appear in the report.

***

## <kbd>class</kbd> `LatexBlock`

A block of LaTeX text.

**Attributes:**

* `text` (str):  The LaTeX text.

***

## <kbd>class</kbd> `Layout`

The layout of a panel in a report. Adjusts the size and position of the panel.

**Attributes:**

* `x` (int):  The x position of the panel.
* `y` (int):  The y position of the panel.
* `w` (int):  The width of the panel.
* `h` (int):  The height of the panel.

***

## <kbd>class</kbd> `LinePlot`

A panel object with 2D line plots.

**Attributes:**

* `title` (Optional\[str]):  The text that appears at the top of the plot.
* `x` (Optional\[MetricType]):  The name of a metric logged to your W\&B project that the  report pulls information from. The metric specified is used for the x-axis.
* `y` (LList\[MetricType]):  One or more metrics logged to your W\&B project that the report pulls  information from. The metric specified is used for the y-axis.
* `range_x` (Tuple\[float | `None`, float | `None`]):  Tuple that specifies the range of the x-axis.
* `range_y` (Tuple\[float | `None`, float | `None`]):  Tuple that specifies the range of the y-axis.
* `log_x` (Optional\[bool]):  Plots the x-coordinates using a base-10 logarithmic scale.
* `log_y` (Optional\[bool]):  Plots the y-coordinates using a base-10 logarithmic scale.
* `title_x` (Optional\[str]):  The label of the x-axis.
* `title_y` (Optional\[str]):  The label of the y-axis.
* `ignore_outliers` (Optional\[bool]):  If set to `True`, do not plot outliers.
* `groupby` (Optional\[str]):  Group runs based on a metric logged to your W\&B project that the  report pulls information from.
* `groupby_aggfunc` (Optional\[GroupAgg]):  Aggregate runs with specified  function. Options include "mean", "min", "max", "median", "sum", "samples", or `None`.
* `groupby_rangefunc` (Optional\[GroupArea]):   Group runs based on a range. Options  include "minmax", "stddev", "stderr", "none", "samples", or `None`.
* `smoothing_factor` (Optional\[float]):  The smoothing factor to apply to the  smoothing type. Accepted values range between 0 and 1.
* `smoothing_type Optional[SmoothingType]`:  Apply a filter based on the specified  distribution. Options include "exponentialTimeWeighted", "exponential",  "gaussian", "average", or "none".
* `smoothing_show_original` (Optional\[bool]):    If set to `True`, show the original data.
* `max_runs_to_show` (Optional\[int]):  The maximum number of runs to show on the line plot.
* `custom_expressions` (Optional\[LList\[str]]):  Custom expressions to apply to the data.
* `plot_type Optional[LinePlotStyle]`:  The type of line plot to generate.  Options include "line", "stacked-area", or "pct-area".
* `font_size Optional[FontSize]`:  The size of the line plot's font.  Options include "small", "medium", "large", "auto", or `None`.
* `legend_position Optional[LegendPosition]`:  Where to place the legend.  Options include "north", "south", "east", "west", or `None`.
* `legend_template` (Optional\[str]):  The template for the legend.
* `aggregate` (Optional\[bool]):  If set to `True`, aggregate the data.
* `xaxis_expression` (Optional\[str]):  The expression for the x-axis.
* `xaxis_format` (Optional\[str]):  The format for the x-axis. This option  appears if you define a custom metric. For example,  you can specify 'datetime' to format the x-axis as a date and time.
* `legend_fields` (Optional\[LList\[str]]):  The fields to include in the legend.
* `metric_regex` (Optional\[str]):  Regular expression pattern to match y-axis metrics.  The backend will use this pattern to select matching metrics.
* `point_visualization_method` (Optional\[PointVizMethod]):  The method used to aggregate  points when there are too many to display. Options include "bucketing-gorilla" (buckets  data points and shows min, max, avg per bucket to preserve outliers and spikes) or  "sampling" (randomly samples points for faster rendering but may miss outliers).
* `line_titles` (Optional\[Dict\[str, str]]):  Per-series display titles. Keys use the format ":".
* `line_colors` (Optional\[Dict\[str, Any]]):  Per-series colors. Keys use the format ":".
* `line_widths` (Optional\[Dict\[str, float]]):  Per-series line widths. Keys use the format ":".
* `line_marks` (Optional\[Dict\[str, Mark]]):  Per-series dash styles. Keys use the format ":".
* `Valid values`:  "solid", "dashed", "dotted", "dotdash", "dotdotdash", "points".

***

## <kbd>class</kbd> `Link`

A link to a URL.

**Attributes:**

* `text` (Union\[str, TextWithInlineComments]):  The text of the link.
* `url` (str):  The URL the link points to.

***

## <kbd>class</kbd> `MarkdownBlock`

A block of markdown text. Useful if you want to write text that uses common markdown syntax.

**Attributes:**

* `text` (str):  The markdown text.

***

## <kbd>class</kbd> `MarkdownPanel`

A panel that renders markdown.

**Attributes:**

* `markdown` (str):  The text you want to appear in the markdown panel.

***

## <kbd>class</kbd> `MediaBrowser`

A panel that displays media files in a gallery or grid layout.

**Attributes:**

* `title` (Optional\[str]):  The title of the panel.
* `num_columns` (Optional\[int]):  The number of columns in the grid.
* `media_keys` (LList\[str]):  A list of media keys that correspond to the media files.
* `mode` (Optional\[Literal\["gallery", "grid"]]):  The display mode for the panel.  If not specified, will be inferred from the axes provided. Required if both  gallery\_axis and grid axes are specified.
* `gallery_axis` (Optional\[Literal\["step", "index", "run"]]):  The field to use for the axis in gallery mode.
* `grid_x_axis` (Optional\[Literal\["step", "index", "run"]]):  The field to use for the x-axis in grid mode.
* `grid_y_axis` (Optional\[Literal\["step", "index", "run"]]):  The field to use for the y-axis in grid mode.

***

## <kbd>class</kbd> `Metric`

A metric to display in a report that is logged in your project.

**Attributes:**

* `name` (str):  The name of the metric.

***

## <kbd>class</kbd> `OrderBy`

A metric to order by.

**Attributes:**

* `name` (str):  The name of the metric.
* `ascending` (bool):  Whether to sort in ascending order.  By default set to `False`.

***

## <kbd>class</kbd> `OrderedList`

A list of items in a numbered list.

**Attributes:**

* `items` (LList\[TextLikeField]):  A list of one or more `OrderedListItem` objects.  Each item can be a string or a list of TextLike objects.

***

## <kbd>class</kbd> `OrderedListItem`

A list item in an ordered list.

**Attributes:**

* `text` (str):  The text of the list item.

***

## <kbd>class</kbd> `P`

A paragraph of text.

**Attributes:**

* `text` (str):  The text of the paragraph.

***

## <kbd>class</kbd> `Panel`

A panel that displays a visualization in a panel grid.

**Attributes:**

* `layout` (Layout):  A `Layout` object.

***

## <kbd>class</kbd> `PanelGrid`

A grid that consists of runsets and panels. Add runsets and panels with `Runset` and `Panel` objects, respectively.

Available panels include: `LinePlot`, `ScatterPlot`, `BarPlot`, `ScalarChart`, `CodeComparer`, `ParallelCoordinatesPlot`, `ParameterImportancePlot`, `RunComparer`, `MediaBrowser`, `MarkdownPanel`, `CustomChart`, `WeavePanel`, `WeavePanelSummaryTable`, `WeavePanelArtifactVersionedFile`.

**Attributes:**

* `runsets` (LList\["Runset"]):  A list of one or more `Runset` objects.
* `hide_run_sets` (bool):  Whether to hide the run sets of the panel grid for report viewers.
* `panels` (LList\["PanelTypes"]):  A list of one or more `Panel` objects.
* `active_runset` (Optional\[int]):  The index of the active runset tab. By default, it is set to 0.
* `custom_run_colors` (dict):  Key-value pairs where the key is the name of a  run and the value is a color specified by a hexadecimal value.

***

## <kbd>class</kbd> `ParallelCoordinatesPlot`

A panel object that shows a parallel coordinates plot.

**Attributes:**

* `columns` (LList\[ParallelCoordinatesPlotColumn]):  A list of one  or more `ParallelCoordinatesPlotColumn` objects.
* `title` (Optional\[str]):  The text that appears at the top of the plot.
* `gradient` (Optional\[LList\[GradientPoint]]):  A list of gradient points.
* `font_size` (Optional\[FontSize]):  The size of the line plot's font.  Options include "small", "medium", "large", "auto", or `None`.

***

## <kbd>class</kbd> `ParallelCoordinatesPlotColumn`

A column within a parallel coordinates plot.  The order of `metric`s specified determine the order of the parallel axis (x-axis) in the parallel coordinates plot.

**Attributes:**

* `metric` (str | Config | SummaryMetric):  The name of the  metric logged to your W\&B project that the report pulls information from.
* `display_name` (Optional\[str]):  The name of the metric
* `inverted` (Optional\[bool]):  Whether to invert the metric.
* `log` (Optional\[bool]):  Whether to apply a log transformation to the metric.

***

## <kbd>class</kbd> `ParameterImportancePlot`

A panel that shows how important each hyperparameter is in predicting the chosen metric.

**Attributes:**

* `with_respect_to` (str):  The metric you want to compare the  parameter importance against. Common metrics might include the loss, accuracy,  and so forth. The metric you specify must be logged within the project  that the report pulls information from.

***

## <kbd>class</kbd> `Report`

An object that represents a W\&B Report. Use the returned object's `blocks` attribute to customize your report. Report objects do not automatically save. Use the `save()` method to persists changes.

**Attributes:**

* `project` (str):  The name of the W\&B project you want to load in.  The project specified appears in the report's URL.
* `entity` (str):  The W\&B entity that owns the report.  The entity appears in the report's URL.
* `title` (str):  The title of the report. The title  appears at the top of the report as an H1 heading.
* `description` (str):  A description of the report.  The description appears underneath the report's title.
* `blocks` (LList\[BlockTypes]):  A list of one or more HTML tags,  plots, grids, runsets, and more.
* `width` (Literal\['readable', 'fixed', 'fluid']):  The width of the report. Options include 'readable', 'fixed', 'fluid'.

***

### <kbd>property</kbd> url

The URL where the report is hosted. The report URL consists of `https://wandb.ai/{entity}/{project_name}/reports/`. Where `{entity}` and `{project_name}` consists of the entity that the report belongs to and the name of the project, respectively.

***

### <kbd>method</kbd> `delete`

```python theme={null}
delete() → bool
```

Delete this report from W\&B.

This will also delete any draft views that reference this report.

**Returns:**

* `bool`:  `True` if the delete operation was acknowledged as successful by the backend, `False` otherwise.

***

### <kbd>method</kbd> `disable_share_link`

```python theme={null}
disable_share_link() → bool
```

Disable the "anyone with link can view" share link for this report.

Revokes all active public access tokens for the report.

**Returns:**

* `bool`:  True if link(s) were revoked, False if no active link existed.

***

### <kbd>method</kbd> `enable_share_link`

```python theme={null}
enable_share_link() → str
```

Enable "anyone with link can view" sharing for this report.

Creates a public access token for the report. If a share link already exists, returns the existing link.

**Returns:**

* `str`:  The magic link URL that can be shared.

***

### <kbd>classmethod</kbd> `from_url`

```python theme={null}
from_url(url: str, as_model: bool = False)
```

Load in the report into current environment. Pass in the URL where the report is hosted.

**Arguments:**

* `url` (str):  The URL where the report is hosted.
* `as_model` (bool):  If True, return the model object instead of the Report object.  By default, set to `False`.

***

### <kbd>method</kbd> `get_share_url`

```python theme={null}
get_share_url() → Optional[str]
```

Fetch the magic link URL for this report, or None if sharing is not enabled.

Makes a network call to check for active access tokens. Returns the report URL with an access token appended, allowing anyone with the link to view the report even if the project is private.

***

### <kbd>method</kbd> `save`

```python theme={null}
save(draft: bool = False, clone: bool = False)
```

Persists changes made to a report object.

***

### <kbd>method</kbd> `to_html`

```python theme={null}
to_html(height: int = 1024, hidden: bool = False) → str
```

Generate HTML containing an iframe displaying this report. Commonly used to within a Python notebook.

**Arguments:**

* `height` (int):  Height of the iframe.
* `hidden` (bool):  If True, hide the iframe. Default set to `False`.

***

## <kbd>class</kbd> `RunComparer`

A panel that compares metrics across different runs from the project the report pulls information from.

**Attributes:**

* `diff_only` (Optional\[Literal\["split", `True`]]):  Display only the  difference across runs in a project. You can toggle this feature on and off in the W\&B Report UI.

***

## <kbd>class</kbd> `RunSettings`

Configure the display appearance and visibility of an individual run in a report panel grid.

Use `RunSettings` to customize how a specific run is rendered within a `Runset`, including its line color and visibility in charts.

**Attributes:**

* `color`:  Display color for the run. Accepts hex format (`#ff0000`), a CSS color name (red), or an RGB string ("rgb(255,0,0)").
* `disabled`:  If True, hide the run from charts. This is equivalent to clicking the closed-eye icon in the UI.

***

## <kbd>class</kbd> `Runset`

A set of runs to display in a panel grid.

**Attributes:**

* `entity` (str):  An entity that owns or has the correct  permissions to the project where the runs are stored.
* `project` (str):  The name of the project were the runs are stored.
* `name` (str):  The name of the run set. Set to `Run set` by default.
* `query` (str):  A query string to filter runs.
* `filters` (Union\[str, LList\[expr.FilterExpr]]):  Filters to apply to runs. Can be:
  * A string expression: e.g., "Config('lr') = 0.001 and State = 'finished'"
* `Supports operators`:  `=`, `==`, `!=`, `<`, `>`, `<=`, `>=`, `in`, `not in`, `and`, `or`
  * A list of FilterExpr objects: e.g., \[expr.Config('lr') == 0.001]
* `groupby` (LList\[str]):  A list of metric names to group by. Supported formats are:
  * "group" or "run.group" to group by a run attribute
  * "config.param" to group by a config parameter
  * "summary.metric" to group by a summary metric
* `order` (LList\[OrderBy]):  A list of `OrderBy` objects to order by.
* `custom_run_colors` (dict):  A dictionary mapping run IDs (or `RunsetGroup` keys for  grouped runs) to colors. For simple per-run colors, prefer using `run_settings`.
* `run_settings` (Dict\[str, RunSettings]):  Per-run display settings keyed by run ID.  Use this to set colors and/or hide individual runs. See `RunSettings`.
* `pinned_columns` (LList\[str]):  Column accessors to pin to the left of
* `the runs table (e.g. `\["summary`: accuracy", "config:lr.value"]`). Cols also in `hidden_columns` are stripped on save.
* `visible_columns` (LList\[str]):  Column accessors to force-show. Use with  `lock_columns=True` to curate the visible set. On reload, this  also includes cols set only via `column_order`.
* `hidden_columns` (LList\[str]):  Column accessors to hide. Takes  precedence over `pinned_columns`, `visible_columns`, and  `column_order`. On reload, may include server-seeded entries
* `like `run`: name`.
* `column_order` (LList\[str]):  Left-to-right column order. Under  `lock_columns=True`, entries not in `pinned_columns` or  `visible_columns` are dropped.
* `column_widths` (Dict\[str, int]):  Per-column pixel widths.
* `lock_columns` (Optional\[bool]):  Opt-in lock for the runs-table column  set. `True` shows only `pinned_columns` and `visible_columns`;  `False` or `None` (default) is permissive. Matches the lock icon  in the runs-table toolbar.

Column format:

* Columns are addressed as `"<section>: <name>"`:
  * `run:<attr>`: built-in run properties (e.g. `run:state`)
  * `config:<key>.value`: config keys (note the `.value` suffix)
  * `summary:<key>`: summary metrics
  * `tags:__ALL__`: run tags

Column limit: The FE runs-table displays at most 500 columns. When the visible set (pinned + visible + column\_order, minus hidden) reaches or exceeds 500, the SDK emits a warning on save; hidden columns don't count. The SDK does not truncate — the full set is sent and the frontend enforces the limit on render.

**Example:**

```python theme={null}
   # Using string filters
   wr.Runset(
        entity="my-entity",
        project="my-project",
        filters="Config('learning_rate') = 0.001 and State = 'finished'"
   )

   # Using FilterExpr list
   wr.Runset(
        entity="my-entity",
        project="my-project",
        filters=[expr.Config("learning_rate") == 0.001]
   )
```

***

### <kbd>method</kbd> `convert_filterexpr_list_to_string`

```python theme={null}
convert_filterexpr_list_to_string()
```

Convert FilterExpr list to string expression.

***

### <kbd>method</kbd> `merge_custom_run_colors_into_run_settings`

```python theme={null}
merge_custom_run_colors_into_run_settings()
```

Merge plain-string-keyed custom\_run\_colors into run\_settings for backward compat.

***

## <kbd>class</kbd> `RunsetGroup`

UI element that shows a group of runsets.

**Attributes:**

* `runset_name` (str):  The name of the runset.
* `keys` (Tuple\[RunsetGroupKey, ...]):  The keys to group by.  Pass in one or more `RunsetGroupKey`  objects to group by.

***

## <kbd>class</kbd> `RunsetGroupKey`

Groups runsets by a metric type and value. Part of a `RunsetGroup`. Specify the metric type and value to group by as key-value pairs.

**Attributes:**

* `key` (Type\[str] | Type\[Config] | Type\[SummaryMetric] | Type\[Metric]):  The metric type to group by.
* `value` (str):  The value of the metric to group by.

***

## <kbd>class</kbd> `ScalarChart`

A panel object that shows a scalar chart.

**Attributes:**

* `title` (Optional\[str]):  The text that appears at the top of the plot.
* `metric` (MetricType):  The name of a metric logged to your W\&B project that the  report pulls information from.
* `groupby_aggfunc` (Optional\[GroupAgg]):  Aggregate runs with specified  function. Options include "mean", "min", "max", "median", "sum", "samples", or `None`.
* `groupby_rangefunc` (Optional\[GroupArea]):   Group runs based on a range. Options  include "minmax", "stddev", "stderr", "none", "samples", or `None`.
* `custom_expressions` (Optional\[LList\[str]]):  A list of custom expressions to be used in the scalar chart.
* `legend_template` (Optional\[str]):  The template for the legend.
* `font_size Optional[FontSize]`:  The size of the line plot's font.  Options include "small", "medium", "large", "auto", or `None`.

***

## <kbd>class</kbd> `ScatterPlot`

A panel object that shows a 2D or 3D scatter plot.

**Arguments:**

* `title` (Optional\[str]):  The text that appears at the top of the plot.
* `x Optional[SummaryOrConfigOnlyMetric]`:  The name of a metric logged to your W\&B project that the  report pulls information from. The metric specified is used for the x-axis.
* `y Optional[SummaryOrConfigOnlyMetric]`:   One or more metrics logged to your W\&B project that the report pulls  information from. Metrics specified are plotted within the y-axis. z Optional\[SummaryOrConfigOnlyMetric]:
* `range_x` (Tuple\[float | `None`, float | `None`]):  Tuple that specifies the range of the x-axis.
* `range_y` (Tuple\[float | `None`, float | `None`]):  Tuple that specifies the range of the y-axis.
* `range_z` (Tuple\[float | `None`, float | `None`]):  Tuple that specifies the range of the z-axis.
* `log_x` (Optional\[bool]):  Plots the x-coordinates using a base-10 logarithmic scale.
* `log_y` (Optional\[bool]):  Plots the y-coordinates using a base-10 logarithmic scale.
* `log_z` (Optional\[bool]):  Plots the z-coordinates using a base-10 logarithmic scale.
* `running_ymin` (Optional\[bool]):   Apply a moving average or rolling mean.
* `running_ymax` (Optional\[bool]):  Apply a moving average or rolling mean.
* `running_ymean` (Optional\[bool]):  Apply a moving average or rolling mean.
* `legend_template` (Optional\[str]):   A string that specifies the format of the legend.
* `gradient` (Optional\[LList\[GradientPoint]]):   A list of gradient points that specify the color gradient of the plot.
* `font_size` (Optional\[FontSize]):  The size of the line plot's font.  Options include "small", "medium", "large", "auto", or `None`.
* `regression` (Optional\[bool]):  If `True`, a regression line is plotted on the scatter plot.

***

## <kbd>class</kbd> `SoundCloud`

A block that renders a SoundCloud player.

**Attributes:**

* `html` (str):  The HTML code to embed the SoundCloud player.

***

## <kbd>class</kbd> `Spotify`

A block that renders a Spotify player.

**Attributes:**

* `spotify_id` (str):  The Spotify ID of the track or playlist.

***

## <kbd>class</kbd> `SummaryMetric`

A summary metric to display in a report.

**Attributes:**

* `name` (str):  The name of the metric.

***

## <kbd>class</kbd> `TableOfContents`

A block that contains a list of sections and subsections using H1, H2, and H3 HTML blocks specified in a report.

***

## <kbd>class</kbd> `TextWithInlineComments`

A block of text with inline comments.

**Attributes:**

* `text` (str):  The text of the block.

***

## <kbd>class</kbd> `Twitter`

A block that displays a Twitter feed.

**Attributes:**

* `html` (str):  The HTML code to display the Twitter feed.

***

## <kbd>class</kbd> `UnorderedList`

A list of items in a bulleted list.

**Attributes:**

* `items` (LList\[TextLikeField]):  A list of one or more `UnorderedListItem` objects.  Each item can be a string or a list of TextLike objects.

***

## <kbd>class</kbd> `UnorderedListItem`

A list item in an unordered list.

**Attributes:**

* `text` (str):  The text of the list item.

***

## <kbd>class</kbd> `Video`

A block that renders a video.

**Attributes:**

* `url` (str):  The URL of the video.

***

## <kbd>class</kbd> `WeaveBlockArtifact`

A block that shows an artifact logged to W\&B. The query takes the form of

```python theme={null}
project('entity', 'project').artifact('artifact-name')
```

The term "Weave" in the API name does not refer to the W\&B Weave toolkit used for tracking and evaluating LLM.

**Attributes:**

* `entity` (str):  The entity that owns or has the appropriate  permissions to the project where the artifact is stored.
* `project` (str):  The project where the artifact is stored.
* `artifact` (str):  The name of the artifact to retrieve.
* `tab Literal["overview", "metadata", "usage", "files", "lineage"]`:  The  tab to display in the artifact panel.

***

## <kbd>class</kbd> `WeaveBlockArtifactVersionedFile`

A block that shows a versioned file logged to a W\&B artifact. The query takes the form of

```python theme={null}
project('entity', 'project').artifactVersion('name', 'version').file('file-name')
```

The term "Weave" in the API name does not refer to the W\&B Weave toolkit used for tracking and evaluating LLM.

**Attributes:**

* `entity` (str):  The entity that owns or has the  appropriate permissions to the project where the artifact is stored.
* `project` (str):  The project where the artifact is stored.
* `artifact` (str):  The name of the artifact to retrieve.
* `version` (str):  The version of the artifact to retrieve.
* `file` (str):  The name of the file stored in the artifact to retrieve.

***

## <kbd>class</kbd> `WeaveBlockSummaryTable`

A block that shows a W\&B Table, pandas DataFrame, plot, or other value logged to W\&B. The query takes the form of

```python theme={null}
project('entity', 'project').runs.summary['value']
```

The term "Weave" in the API name does not refer to the W\&B Weave toolkit used for tracking and evaluating LLM.

**Attributes:**

* `entity` (str):  The entity that owns or has the  appropriate permissions to the project where the values are logged.
* `project` (str):  The project where the value is logged in.
* `table_name` (str):  The name of the table, DataFrame, plot, or value.

***

## <kbd>class</kbd> `WeavePanel`

An empty query panel that can be used to display custom content using queries.

The term "Weave" in the API name does not refer to the W\&B Weave toolkit used for tracking and evaluating LLM.

***

## <kbd>class</kbd> `WeavePanelArtifact`

A panel that shows an artifact logged to W\&B.

The term "Weave" in the API name does not refer to the W\&B Weave toolkit used for tracking and evaluating LLM.

**Attributes:**

* `artifact` (str):  The name of the artifact to retrieve.
* `tab Literal["overview", "metadata", "usage", "files", "lineage"]`:  The tab to display in the artifact panel.

***

## <kbd>class</kbd> `WeavePanelArtifactVersionedFile`

A panel that shows a versioned file logged to a W\&B artifact.

```python theme={null}
project('entity', 'project').artifactVersion('name', 'version').file('file-name')
```

The term "Weave" in the API name does not refer to the W\&B Weave toolkit used for tracking and evaluating LLM.

**Attributes:**

* `artifact` (str):  The name of the artifact to retrieve.
* `version` (str):  The version of the artifact to retrieve.
* `file` (str):  The name of the file stored in the artifact to retrieve.

***

## <kbd>class</kbd> `WeavePanelSummaryTable`

A panel that shows a W\&B Table, pandas DataFrame, plot, or other value logged to W\&B. The query takes the form of

```python theme={null}
runs.summary['value']
```

The term "Weave" in the API name does not refer to the W\&B Weave toolkit used for tracking and evaluating LLM.

**Attributes:**

* `table_name` (str):  The name of the table, DataFrame, plot, or value.
* `layout` (Layout):  The layout configuration for the panel, including position and size.  Inherited from the Panel base class. Use to adjust width (w) and height (h).

**Example:**
Create a report with a summary table panel that displays a table with custom dimensions:

```python theme={null}
    import wandb_workspaces.reports as wr

    report = wr.Report(
         project="my-project",
         entity="my-entity",
         title="Summary Table Report"
    )

    report.blocks = [
         wr.PanelGrid(
             runsets=[wr.Runset(project="my-project")],
             panels=[
                 wr.WeavePanelSummaryTable(
                     table_name="my-table-name",
                     layout=wr.Layout(w=24, h=20)
                 )
             ]
         )
    ]

    report.save()
```

The layout parameters control panel dimensions:

* w (width): Width in grid units (default: 8, max: 24)
* h (height): Height in grid units (default: 6)
* x (x-position): Horizontal position in grid (default: 0)
* y (y-position): Vertical position in grid (default: 0)


# Workspaces
Source: https://docs.wandb.ai/models/ref/wandb_workspaces/workspaces



<GitHubLink />

<Note>
  W\&B Report and Workspace API is in Public Preview.
</Note>

Python library for programmatically working with W\&B Workspace API.

```python theme={null}
# How to import
import wandb_workspaces.workspaces as ws
import wandb_workspaces.reports.v2 as wr

# Example of creating a workspace
ws.Workspace(
     name="Example W&B Workspace",
     entity="entity", # entity that owns the workspace
     project="project", # project that the workspace is associated with
     sections=[
         ws.Section(
             name="Validation Metrics",
             panels=[
                 wr.LinePlot(x="Step", y=["val_loss"]),
                 wr.BarPlot(metrics=["val_accuracy"]),
                 wr.ScalarChart(metric="f1_score", groupby_aggfunc="mean"),
             ],
             is_open=True,
         ),
     ],
)
workspace.save()
```

## <kbd>class</kbd> `RunSettings`

Settings for a run in a runset (left hand bar).

**Attributes:**

* `color` (str):  The color of the run in the UI.  Can be hex (#ff0000), css color (red), or rgb (rgb(255, 0, 0))
* `disabled` (bool):  Whether the run is deactivated (eye closed in the UI). Default is set to `False`.

***

## <kbd>class</kbd> `RunsetSettings`

Settings for the runset (the left bar containing runs) in a workspace.

**Attributes:**

* `query` (str):  A query to filter the runset (can be a regex expr, see next param).
* `regex_query` (bool):  Controls whether the query (above) is a regex expr. Default is set to `False`.
* `filters` (Union\[str, LList\[FilterExpr], Or, And]):  Filters for the runset.
  * As a list of FilterExpr: filters are AND'd together.
  * As a string: Python-like expressions, e.g., "Config('lr') = 0.001 and State = 'finished'"
* `Supports operators`:  `=`, `==`, `!=`, `<`, `>`, `<=`, `>=`, `in`, `not in`, `and`, `or`
  * As an Or/And combinator: allows OR logic and nested groups. e.g., Or(And(Config("lr") == 0.01, Metric("State") == "finished"), Config("lr") == 0.1)
* `groupby` (LList\[expr.MetricType]):  A list of metrics to group by in the runset. Set to  `Metric`, `Summary`, `Config`, `Tags`, or `KeysInfo`.
* `order` (LList\[expr.Ordering]):  A list of metrics and ordering to apply to the runset.
* `run_settings` (Dict\[str, RunSettings]):  A dictionary of run settings, where the key  is the run's ID and the value is a RunSettings object.
* `pinned_columns` (LList\[str]):  List of column names to pin.
* `Column names use format`:  "run:displayName", "summary:metric", "config:param".
* `run`: displayName is automatically added if not present.
* `Example`:  \["summary:accuracy", "summary:loss"]
* `baseline_run` (Optional\[str]):  W\&B run slug of the baseline run (the  value of `wandb.Run.id`, e.g. `"1mbku38n"` — also the last path  segment of a run URL). Used for delta columns and comparison  styling. When set, the baseline run is automatically added to  `pinned_runs` to match the W\&B app's behavior. Must refer to a  run in the workspace's own project — cross-project pins are not  supported by this SDK yet (see the W\&B app's "Add cross-project  runs" drawer for the UI equivalent).
* `pinned_runs` (LList\[str]):  Ordered list of W\&B run slugs to keep  visible in the run selector and always fetched for plots. Pass  slug strings (`wandb.Run.id`), not GraphQL IDs. The W\&B app  caps this at 20 entries.

**Example:**

```python theme={null}
   # Using string filters (new)
   RunsetSettings(
        filters="Config('learning_rate') = 0.001 and State = 'finished'",
        pinned_columns=["summary:accuracy", "summary:loss"],
        baseline_run="1mbku38n",  # wandb.Run.id (the slug from the URL)
        pinned_runs=["1mbku38n", "2u1g3j1c"],
   )

   # Using FilterExpr list (original way)
   RunsetSettings(
        filters=[expr.Config("learning_rate") == 0.001],
        pinned_columns=["summary:accuracy", "summary:loss"],
   )
```

***

### <kbd>method</kbd> `convert_filterexpr_list_to_string`

```python theme={null}
convert_filterexpr_list_to_string()
```

Convert FilterExpr list or Or/And to string expression.

***

### <kbd>method</kbd> `ensure_baseline_pinned`

```python theme={null}
ensure_baseline_pinned()
```

Mirror the W\&B app's invariant: a baseline run is always also pinned.

The frontend's `setBaselineRun` action handler routes through `setRunPinnedAndBaseline`, so any spec produced by the UI has the baseline ID present in `pinnedRunIds`. We enforce the same here so SDK-produced specs match.

***

### <kbd>method</kbd> `validate_and_setup_columns`

```python theme={null}
validate_and_setup_columns()
```

Ensure run:displayName is present and set up internal column fields.

***

## <kbd>class</kbd> `Section`

Represents a section in a workspace.

**Attributes:**

* `name` (str):  The name/title of the section.
* `panels` (LList\[PanelTypes]):  An ordered list of panels in the section. By default, first is top-left and last is bottom-right.
* `is_open` (bool):  Whether the section is open or closed. Default is closed.
* `pinned` (bool):  Whether the section is pinned. Pinned sections appear at the top of the workspace. Default is False.
* `layout_settings` (SectionLayoutSettings):  Settings for panel layout in the section.
* `panel_settings`:  Panel-level settings applied to all panels in the section, similar to `WorkspaceSettings` for a `Section`.

***

## <kbd>class</kbd> `SectionLayoutSettings`

Panel layout settings for a section, typically seen at the top right of the section of the W\&B App Workspace UI.

**Attributes:**

* `columns` (int):  The number of columns in the layout. Default is 3.
* `rows` (int):  The number of rows in the layout. Default is 2.

***

## <kbd>class</kbd> `SectionPanelSettings`

Panel settings for a section, similar to `WorkspaceSettings` for a section.

Settings applied here can be overrided by more granular Panel settings in this priority: Section \< Panel.

**Attributes:**

* `x_axis` (str):  X-axis metric name setting. By default, set to "Step".
* `x_min Optional[float]`:  Minimum value for the x-axis.
* `x_max Optional[float]`:  Maximum value for the x-axis.
* `smoothing_type` (Literal\['exponentialTimeWeighted', 'exponential', 'gaussian', 'average', 'none']):  Smoothing  type applied to all panels.
* `smoothing_weight` (int):  Smoothing weight applied to all panels.

***

## <kbd>class</kbd> `Workspace`

Represents a W\&B workspace, including sections, settings, and config for run sets.

**Attributes:**

* `entity` (str):  The entity this workspace will be saved to (usually user or team name).
* `project` (str):  The project this workspace will be saved to.
* `name`:  The name of the workspace.
* `sections` (LList\[Section]):  An ordered list of sections in the workspace.  The first section is at the top of the workspace.
* `settings` (WorkspaceSettings):  Settings for the workspace, typically seen at  the top of the workspace in the UI.
* `runset_settings` (RunsetSettings):  Settings for the runset  (the left bar containing runs) in a workspace.
* `auto_generate_panels` (bool):  Whether to automatically generate panels for all keys logged in this project.  Recommended if you would like all available data to be visualized by default.  This can only be set during workspace creation and cannot be modified afterward.

***

### <kbd>property</kbd> auto\_generate\_panels

***

### <kbd>property</kbd> url

The URL to the workspace in the W\&B app.

***

### <kbd>classmethod</kbd> `from_url`

```python theme={null}
from_url(url: str)
```

Get a workspace from a URL.

***

### <kbd>method</kbd> `save`

```python theme={null}
save()
```

Save the current workspace to W\&B.

**Returns:**

* `Workspace`:  The updated workspace with the saved internal name and ID.

***

### <kbd>method</kbd> `save_as_new_view`

```python theme={null}
save_as_new_view()
```

Save the current workspace as a new view to W\&B.

**Returns:**

* `Workspace`:  The updated workspace with the saved internal name and ID.

***

## <kbd>class</kbd> `WorkspaceSettings`

Settings for the workspace, typically seen at the top of the workspace in the UI.

This object includes settings for the x-axis, smoothing, outliers, panels, tooltips, runs, and panel query bar.

Settings applied here can be overrided by more granular Section and Panel settings in this priority: Workspace \< Section \< Panel

**Attributes:**

* `x_axis` (str):  X-axis metric name setting.
* `x_min` (Optional\[float]):  Minimum value for the x-axis.
* `x_max` (Optional\[float]):  Maximum value for the x-axis.
* `smoothing_type` (Literal\['exponentialTimeWeighted', 'exponential', 'gaussian', 'average', 'none']):  Smoothing  type applied to all panels.
* `smoothing_weight` (int):  Smoothing weight applied to all panels.
* `ignore_outliers` (bool):  Ignore outliers in all panels.
* `sort_panels_alphabetically` (bool):  Sorts panels in all sections alphabetically.
* `group_by_prefix` (Literal\["first", "last"]):  Group panels by the first or up to last  prefix (first or last). Default is set to `last`.
* `remove_legends_from_panels` (bool):  Remove legends from all panels.
* `tooltip_number_of_runs` (Literal\["default", "all", "none"]):  The number of runs to show in the tooltip.
* `tooltip_color_run_names` (bool):  Whether to color run names in the tooltip to  match the runset (True) or not (False). Default is set to `True`.
* `max_runs` (int):  The maximum number of runs to show per panel (this will be the first 10 runs in the runset).
* `point_visualization_method` (Literal\["line", "point", "line\_point"]):  The visualization method for points.
* `panel_search_query` (str):  The query for the panel search bar (can be a regex expression).
* `auto_expand_panel_search_results` (bool):  Whether to auto expand the panel search results.


# Overview
Source: https://docs.wandb.ai/weave/reference

API documentation and SDK references for Weights & Biases Weave

Welcome to the Weights & Biases Weave reference documentation. Here you'll find API documentation and SDK references to help you integrate Weave into your applications.

<CardGroup>
  <Card title="Python SDK" icon="python" href="/weave/reference/python-sdk">
    Reference for the Weave Python SDK, including core functionality, tracing, and server interfaces.
  </Card>

  <Card title="TypeScript SDK" icon="js" href="/weave/reference/typescript-sdk">
    Reference for the Weave TypeScript SDK, covering classes, functions, and interfaces.
  </Card>

  <Card title="Service API" icon="code" href="/weave/reference/service-api">
    REST API reference for direct integration with Weave services, including the available endpoints.
  </Card>
</CardGroup>


# weave
Source: https://docs.wandb.ai/weave/reference/python-sdk

Python SDK reference for weave

# API Overview

***

<SourceLink />

## <kbd>class</kbd> `Agent`

**Pydantic Fields:**

* `name`: `str | None`
* `description`: `str | None`
* `ref`: `trace.refs.ObjectRef | None`
* `model_name`: `<class 'str'>`
* `temperature`: `<class 'float'>`
* `system_message`: `<class 'str'>`
* `tools`: `list[typing.Any]`

<SourceLink />

### <kbd>method</kbd> `step`

```python theme={null}
step(state: AgentState) → AgentState
```

Run a step of the agent.

**Args:**

* <b>`state`</b>: The current state of the environment.
* <b>`action`</b>: The action to take.
  **Returns:**
  The new state of the environment.

***

<SourceLink />

## <kbd>class</kbd> `AgentState`

**Pydantic Fields:**

* `name`: `str | None`
* `description`: `str | None`
* `ref`: `trace.refs.ObjectRef | None`
* `history`: `list[typing.Any]`

***

<SourceLink />

## <kbd>class</kbd> `AnnotationSpec`

**Pydantic Fields:**

* `name`: `str | None`
* `description`: `str | None`
* `field_schema`: `dict[str, typing.Any]`
* `unique_among_creators`: `<class 'bool'>`
* `op_scope`: `list[str] | None`

<SourceLink />

### <kbd>classmethod</kbd> `preprocess_field_schema`

```python theme={null}
preprocess_field_schema(data: dict[str, Any]) → dict[str, Any]
```

***

<SourceLink />

### <kbd>classmethod</kbd> `validate_field_schema`

```python theme={null}
validate_field_schema(schema: dict[str, Any]) → dict[str, Any]
```

***

<SourceLink />

### <kbd>method</kbd> `value_is_valid`

```python theme={null}
value_is_valid(payload: Any) → bool
```

Validates a payload against this annotation spec's schema.

**Args:**

* <b>`payload`</b>: The data to validate against the schema
  **Returns:**

* <b>`bool`</b>:  True if validation succeeds, False otherwise

***

<SourceLink />

## <kbd>class</kbd> `Audio`

A class representing audio data in a supported format (wav or mp3).

This class handles audio data storage and provides methods for loading from different sources and exporting to files.

**Attributes:**

* <b>`format`</b>:  The audio format (currently supports 'wav' or 'mp3')
* <b>`data`</b>:  The raw audio data as bytes

**Args:**

* <b>`data`</b>: The audio data (bytes or base64 encoded string)

* <b>`format`</b>: The audio format ('wav' or 'mp3')

* <b>`validate_base64`</b>: Whether to attempt base64 decoding of the input data
  **Raises:**

* <b>`ValueError`</b>:  If audio data is empty or format is not supported

<SourceLink />

### <kbd>method</kbd> `__init__`

```python theme={null}
__init__(
    data: 'bytes',
    format: 'SUPPORTED_FORMATS_TYPE',
    validate_base64: 'bool' = True
) → None
```

***

<SourceLink />

### <kbd>method</kbd> `export`

```python theme={null}
export(path: 'str | bytes | Path | PathLike') → None
```

Export audio data to a file.

**Args:**

***

<SourceLink />

### <kbd>classmethod</kbd> `from_data`

```python theme={null}
from_data(data: 'str | bytes', format: 'str') → Self
```

Create an Audio object from raw data and specified format.

* <b>`path`</b>: Path where the audio file should be written
  **Args:**

* <b>`data`</b>: Audio data as bytes or base64 encoded string

* <b>`format`</b>: Audio format ('wav' or 'mp3')
  **Returns:**

* <b>`Audio`</b>:  A new Audio instance

**Raises:**

* <b>`ValueError`</b>:  If format is not supported

***

<SourceLink />

### <kbd>classmethod</kbd> `from_path`

```python theme={null}
from_path(path: 'str | bytes | Path | PathLike') → Self
```

Create an Audio object from a file path.

**Args:**

* <b>`path`</b>: Path to an audio file (must have .wav or .mp3 extension)
  **Returns:**

* <b>`Audio`</b>:  A new Audio instance loaded from the file

**Raises:**

* <b>`ValueError`</b>:  If file doesn't exist or has unsupported extension

***

<SourceLink />

## <kbd>class</kbd> `ClassifierMonitor`

A monitor that merges multiple scorers into a single classifier.

Classifier monitors combine prompts from multiple LLMAsAJudgeScorers targeting the same model into a single scoring call.

**Pydantic Fields:**

* `name`: `str | None`
* `description`: `str | None`
* `ref`: `trace.refs.ObjectRef | None`
* `sampling_rate`: `<class 'float'>`
* `scorers`: `list[flow.scorer.Scorer]`
* `op_names`: `list[typing.Union[typing.Literal['genai.turn_ended'], str]]`
* `query`: `trace_server.interface.query.Query | None`
* `is_traced`: `<class 'bool'>`
* `active`: `<class 'bool'>`
* `scorer_debounce_config`: `flow.monitor.ScorerDebounceConfig | None`
* `prompt_header`: `str | None`
* `prompt_footer`: `str | None`

<SourceLink />

### <kbd>method</kbd> `activate`

```python theme={null}
activate() → ObjectRef
```

Activates the monitor.

**Returns:**
The ref to the monitor.

***

<SourceLink />

### <kbd>method</kbd> `deactivate`

```python theme={null}
deactivate() → ObjectRef
```

Deactivates the monitor.

**Returns:**
The ref to the monitor.

***

<SourceLink />

### <kbd>classmethod</kbd> `from_obj`

```python theme={null}
from_obj(obj: WeaveObject) → Self
```

***

<SourceLink />

### <kbd>method</kbd> `get_prompt_footer`

```python theme={null}
get_prompt_footer() → str
```

Text to append after the merged classifier prompts.

***

<SourceLink />

### <kbd>method</kbd> `get_prompt_header`

```python theme={null}
get_prompt_header(op_name: str) → str
```

Text to prepend before the merged classifier prompts.

***

<SourceLink />

## <kbd>class</kbd> `Content`

A class to represent content from various sources, resolving them to a unified byte-oriented representation with associated metadata.

This class must be instantiated using one of its classmethods:

* from\_path()
* from\_bytes()
* from\_text()
* from\_url()
* from\_base64()
* from\_data\_url()

<SourceLink />

### <kbd>method</kbd> `__init__`

```python theme={null}
__init__(*args: 'Any', **kwargs: 'Any') → None
```

Direct initialization is disabled. Please use a classmethod like `Content.from_path()` to create an instance.

**Pydantic Fields:**

* `data`: `<class 'bytes'>`
* `size`: `<class 'int'>`
* `mimetype`: `<class 'str'>`
* `digest`: `<class 'str'>`
* `filename`: `<class 'str'>`
* `content_type`: `typing.Literal['bytes', 'text', 'base64', 'file', 'url', 'data_url', 'data_url:base64', 'data_url:encoding', 'data_url:encoding:base64']`
* `input_type`: `<class 'str'>`
* `encoding`: `<class 'str'>`
* `metadata`: `dict[str, typing.Any] | None`
* `extension`: `str | None`

***

#### <kbd>property</kbd> art

#### <kbd>property</kbd> ref

***

<SourceLink />

### <kbd>method</kbd> `as_string`

```python theme={null}
as_string() → str
```

Display the data as a string. Bytes are decoded using the `encoding` attribute If base64, the data will be re-encoded to base64 bytes then decoded to an ASCII string

**Returns:**
str.

***

<SourceLink />

### <kbd>classmethod</kbd> `from_base64`

```python theme={null}
from_base64(
    b64_data: 'str | bytes',
    extension: 'str | None' = None,
    mimetype: 'str | None' = None,
    metadata: 'dict[str, Any] | None' = None
) → Self
```

Initializes Content from a base64 encoded string or bytes.

***

<SourceLink />

### <kbd>classmethod</kbd> `from_bytes`

```python theme={null}
from_bytes(
    data: 'bytes',
    extension: 'str | None' = None,
    mimetype: 'str | None' = None,
    metadata: 'dict[str, Any] | None' = None,
    encoding: 'str' = 'utf-8'
) → Self
```

Initializes Content from raw bytes.

***

<SourceLink />

### <kbd>classmethod</kbd> `from_data_url`

```python theme={null}
from_data_url(url: 'str', metadata: 'dict[str, Any] | None' = None) → Self
```

Initializes Content from a data URL.

***

<SourceLink />

### <kbd>classmethod</kbd> `from_path`

```python theme={null}
from_path(
    path: 'str | Path',
    encoding: 'str' = 'utf-8',
    mimetype: 'str | None' = None,
    metadata: 'dict[str, Any] | None' = None
) → Self
```

Initializes Content from a local file path.

***

<SourceLink />

### <kbd>classmethod</kbd> `from_text`

```python theme={null}
from_text(
    text: 'str',
    extension: 'str | None' = None,
    mimetype: 'str | None' = None,
    metadata: 'dict[str, Any] | None' = None,
    encoding: 'str' = 'utf-8'
) → Self
```

Initializes Content from a string of text.

***

<SourceLink />

### <kbd>classmethod</kbd> `from_url`

```python theme={null}
from_url(
    url: 'str',
    headers: 'dict[str, Any] | None' = None,
    timeout: 'int | None' = 30,
    metadata: 'dict[str, Any] | None' = None
) → Self
```

Initializes Content by fetching bytes from an HTTP(S) URL.

Downloads the content, infers mimetype/extension from headers, URL path, and data, and constructs a Content object from the resulting bytes.

***

<SourceLink />

### <kbd>classmethod</kbd> `model_validate`

```python theme={null}
model_validate(
    obj: 'Any',
    strict: 'bool | None' = None,
    from_attributes: 'bool | None' = None,
    context: 'dict[str, Any] | None' = None,
    extra: 'str | None' = None,
    by_alias: 'bool | None' = None,
    by_name: 'bool | None' = None
) → Self
```

Override model\_validate to handle Content reconstruction from dict.

***

<SourceLink />

### <kbd>classmethod</kbd> `model_validate_json`

```python theme={null}
model_validate_json(
    json_data: 'str | bytes | bytearray',
    strict: 'bool | None' = None,
    context: 'dict[str, Any] | None' = None,
    extra: 'str | None' = None,
    by_alias: 'bool | None' = None,
    by_name: 'bool | None' = None
) → Self
```

Override model\_validate\_json to handle Content reconstruction from JSON.

***

<SourceLink />

### <kbd>method</kbd> `open`

```python theme={null}
open() → bool
```

Open the file using the operating system's default application.

This method uses the platform-specific mechanism to open the file with the default application associated with the file's type.

**Returns:**

* <b>`bool`</b>:  True if the file was successfully opened, False otherwise.

***

<SourceLink />

### <kbd>method</kbd> `save`

```python theme={null}
save(dest: 'str | Path') → None
```

Copy the file to the specified destination path. Updates the filename and the path of the content to reflect the last saved copy.

**Args:**

***

<SourceLink />

### <kbd>method</kbd> `serialize_data`

```python theme={null}
serialize_data(data: 'bytes') → str
```

When dumping model in json mode

***

<SourceLink />

### <kbd>method</kbd> `to_data_url`

```python theme={null}
to_data_url(use_base64: 'bool' = True) → str
```

Constructs a data URL from the content.

* <b>`dest`</b>: Destination path where the file will be copied to (string or pathlib.Path)  The destination path can be a file or a directory.  If dest has no file extension (e.g. .txt), destination will be considered a directory.
  **Args:**

* <b>`use_base64`</b>: If True, the data will be base64 encoded.  Otherwise, it will be percent-encoded. Defaults to True.
  **Returns:**
  A data URL string.

***

<SourceLink />

## <kbd>class</kbd> `Dataset`

Dataset object with easy saving and automatic versioning.

**Examples:**

```python theme={null}
# Create a dataset
dataset = Dataset(name='grammar', rows=[
     {'id': '0', 'sentence': "He no likes ice cream.", 'correction': "He doesn't like ice cream."},
     {'id': '1', 'sentence': "She goed to the store.", 'correction': "She went to the store."},
     {'id': '2', 'sentence': "They plays video games all day.", 'correction': "They play video games all day."}
])

# Publish the dataset
weave.publish(dataset)

# Retrieve the dataset
dataset_ref = weave.ref('grammar').get()

# Access a specific example
example_label = dataset_ref.rows[2]['sentence']
```

**Pydantic Fields:**

* `name`: `str | None`
* `description`: `str | None`
* `ref`: `trace.refs.ObjectRef | None`
* `rows`: `trace.table.Table | trace.vals.WeaveTable`

<SourceLink />

### <kbd>method</kbd> `add_rows`

```python theme={null}
add_rows(rows: Iterable[dict]) → Dataset
```

Create a new dataset version by appending rows to the existing dataset.

This is useful for adding examples to large datasets without having to load the entire dataset into memory.

**Args:**

* <b>`rows`</b>: The rows to add to the dataset.
  **Returns:**
  The updated dataset.

***

<SourceLink />

### <kbd>classmethod</kbd> `convert_to_table`

```python theme={null}
convert_to_table(rows: Any) → Table | WeaveTable
```

***

<SourceLink />

### <kbd>classmethod</kbd> `from_calls`

```python theme={null}
from_calls(calls: Iterable[Call]) → Self
```

***

<SourceLink />

### <kbd>classmethod</kbd> `from_hf`

```python theme={null}
from_hf(hf_dataset: 'HFDataset | HFDatasetDict') → Self
```

***

<SourceLink />

### <kbd>classmethod</kbd> `from_obj`

```python theme={null}
from_obj(obj: WeaveObject) → Self
```

***

<SourceLink />

### <kbd>classmethod</kbd> `from_pandas`

```python theme={null}
from_pandas(df: 'DataFrame') → Self
```

***

<SourceLink />

### <kbd>method</kbd> `select`

```python theme={null}
select(indices: Iterable[int]) → Self
```

Select rows from the dataset based on the provided indices.

**Args:**

* <b>`indices`</b>: An iterable of integer indices specifying which rows to select.
  **Returns:**
  A new Dataset object containing only the selected rows.

***

<SourceLink />

### <kbd>method</kbd> `to_hf`

```python theme={null}
to_hf() → HFDataset
```

***

<SourceLink />

### <kbd>method</kbd> `to_pandas`

```python theme={null}
to_pandas() → DataFrame
```

***

<SourceLink />

## <kbd>class</kbd> `EasyPrompt`

<SourceLink />

### <kbd>method</kbd> `__init__`

```python theme={null}
__init__(
    content: str | dict | list | None = None,
    role: str | None = None,
    dedent: bool = False,
    **kwargs: Any
) → None
```

**Pydantic Fields:**

* `name`: `str | None`
* `description`: `str | None`
* `ref`: `trace.refs.ObjectRef | None`
* `data`: `<class 'list'>`
* `config`: `<class 'dict'>`
* `requirements`: `<class 'dict'>`

***

#### <kbd>property</kbd> as\_str

Join all messages into a single string.

***

#### <kbd>property</kbd> is\_bound

***

#### <kbd>property</kbd> messages

#### <kbd>property</kbd> placeholders

***

#### <kbd>property</kbd> system\_message

Join all messages into a system prompt message.

***

#### <kbd>property</kbd> system\_prompt

Join all messages into a system prompt object.

***

#### <kbd>property</kbd> unbound\_placeholders

***

<SourceLink />

### <kbd>method</kbd> `append`

```python theme={null}
append(item: Any, role: str | None = None, dedent: bool = False) → None
```

***

<SourceLink />

### <kbd>method</kbd> `as_dict`

```python theme={null}
as_dict() → dict[str, Any]
```

***

<SourceLink />

### <kbd>method</kbd> `as_pydantic_dict`

```python theme={null}
as_pydantic_dict() → dict[str, Any]
```

***

<SourceLink />

### <kbd>method</kbd> `bind`

```python theme={null}
bind(*args: Any, **kwargs: Any) → Prompt
```

***

<SourceLink />

### <kbd>method</kbd> `bind_rows`

```python theme={null}
bind_rows(dataset: list[dict] | Any) → list['Prompt']
```

***

<SourceLink />

### <kbd>method</kbd> `config_table`

```python theme={null}
config_table(title: str | None = None) → Table
```

***

<SourceLink />

### <kbd>method</kbd> `configure`

```python theme={null}
configure(config: dict | None = None, **kwargs: Any) → Prompt
```

***

<SourceLink />

### <kbd>method</kbd> `dump`

```python theme={null}
dump(fp: <class 'IO'>) → None
```

***

<SourceLink />

### <kbd>method</kbd> `dump_file`

```python theme={null}
dump_file(filepath: str | Path) → None
```

***

<SourceLink />

### <kbd>method</kbd> `format`

```python theme={null}
format(**kwargs: Any) → Any
```

***

<SourceLink />

### <kbd>classmethod</kbd> `from_obj`

```python theme={null}
from_obj(obj: WeaveObject) → Self
```

***

<SourceLink />

### <kbd>classmethod</kbd> `load`

```python theme={null}
load(fp: <class 'IO'>) → Self
```

***

<SourceLink />

### <kbd>classmethod</kbd> `load_file`

```python theme={null}
load_file(filepath: str | Path) → Self
```

***

<SourceLink />

### <kbd>method</kbd> `messages_table`

```python theme={null}
messages_table(title: str | None = None) → Table
```

***

<SourceLink />

### <kbd>method</kbd> `print`

```python theme={null}
print() → str
```

***

<SourceLink />

### <kbd>method</kbd> `publish`

```python theme={null}
publish(
    name: str | None = None,
    tags: list[str] | None = None,
    aliases: list[str] | None = None
) → ObjectRef
```

***

<SourceLink />

### <kbd>method</kbd> `require`

```python theme={null}
require(param_name: str, **kwargs: Any) → Prompt
```

***

<SourceLink />

### <kbd>method</kbd> `run`

```python theme={null}
run() → Any
```

***

<SourceLink />

### <kbd>method</kbd> `validate_requirement`

```python theme={null}
validate_requirement(key: str, value: Any) → list
```

***

<SourceLink />

### <kbd>method</kbd> `validate_requirements`

```python theme={null}
validate_requirements(values: dict[str, Any]) → list
```

***

<SourceLink />

### <kbd>method</kbd> `values_table`

```python theme={null}
values_table(title: str | None = None) → Table
```

***

<SourceLink />

## <kbd>class</kbd> `Evaluation`

Sets up an evaluation which includes a set of scorers and a dataset.

Calling evaluation.evaluate(model) will pass in rows from a dataset into a model matching  the names of the columns of the dataset to the argument names in model.predict.

Then it will call all of the scorers and save the results in weave.

If you want to preprocess the rows from the dataset you can pass in a function to preprocess\_model\_input.

**Examples:**

```python theme={null}
# Collect your examples
examples = [
     {"question": "What is the capital of France?", "expected": "Paris"},
     {"question": "Who wrote 'To Kill a Mockingbird'?", "expected": "Harper Lee"},
     {"question": "What is the square root of 64?", "expected": "8"},
]

# Define any custom scoring function
@weave.op
def match_score1(expected: str, model_output: dict) -> dict:
     # Here is where you'd define the logic to score the model output
     return {'match': expected == model_output['generated_text']}

@weave.op
def function_to_evaluate(question: str):
     # here's where you would add your LLM call and return the output
     return  {'generated_text': 'Paris'}

# Score your examples using scoring functions
evaluation = Evaluation(
     dataset=examples, scorers=[match_score1]
)

# Start tracking the evaluation
weave.init('intro-example')
# Run the evaluation
asyncio.run(evaluation.evaluate(function_to_evaluate))
```

**Pydantic Fields:**

* `name`: `str | None`
* `description`: `str | None`
* `ref`: `trace.refs.ObjectRef | None`
* `dataset`: `<class 'dataset.dataset.Dataset'>`
* `scorers`: `list[typing.Annotated[trace.op_protocol.Op | flow.scorer.Scorer, BeforeValidator(func=<function cast_to_scorer at 0x7f1615b1b600>, json_schema_input_type=PydanticUndefined)]] | None`
* `preprocess_model_input`: `collections.abc.Callable[[dict], dict] | None`
* `trials`: `<class 'int'>`
* `metadata`: `dict[str, typing.Any] | None`
* `evaluation_name`: `str | collections.abc.Callable[trace.call.Call, str] | None`

<SourceLink />

### <kbd>method</kbd> `evaluate`

```python theme={null}
evaluate(model: Op | Model) → dict
```

***

<SourceLink />

### <kbd>classmethod</kbd> `from_obj`

```python theme={null}
from_obj(obj: WeaveObject) → Self
```

***

<SourceLink />

### <kbd>method</kbd> `get_eval_results`

```python theme={null}
get_eval_results(model: Op | Model) → EvaluationResults
```

***

<SourceLink />

### <kbd>method</kbd> `get_evaluate_calls`

```python theme={null}
get_evaluate_calls() → PaginatedIterator[CallSchema, WeaveObject]
```

Retrieve all evaluation calls that used this Evaluation object.

Note that this returns a CallsIter instead of a single call because it's possible to have multiple evaluation calls for a single evaluation (e.g. if you run the same evaluation multiple times).

**Returns:**

* <b>`CallsIter`</b>:  An iterator over Call objects representing evaluation runs.

**Raises:**

* <b>`ValueError`</b>:  If the evaluation has no ref (hasn't been saved/run yet).

**Examples:**

```python theme={null}
evaluation = Evaluation(dataset=examples, scorers=[scorer])
await evaluation.evaluate(model)  # Run evaluation first
calls = evaluation.get_evaluate_calls()
for call in calls:
     print(f"Evaluation run: {call.id} at {call.started_at}")
```

***

<SourceLink />

### <kbd>method</kbd> `get_score_calls`

```python theme={null}
get_score_calls() → dict[str, list[Call]]
```

Retrieve scorer calls for each evaluation run, grouped by trace ID.

**Returns:**

* <b>`dict[str, list[Call]]`</b>:  A dictionary mapping trace IDs to lists of scorer Call objects.  Each trace ID represents one evaluation run, and the list contains all scorer  calls executed during that run.

**Examples:**

```python theme={null}
evaluation = Evaluation(dataset=examples, scorers=[accuracy_scorer, f1_scorer])
await evaluation.evaluate(model)
score_calls = evaluation.get_score_calls()
for trace_id, calls in score_calls.items():
     print(f"Trace {trace_id}: {len(calls)} scorer calls")
     for call in calls:
         scorer_name = call.summary.get("weave", {}).get("trace_name")
         print(f"  Scorer: {scorer_name}, Output: {call.output}")
```

***

<SourceLink />

### <kbd>method</kbd> `get_scores`

```python theme={null}
get_scores() → dict[str, dict[str, list[Any]]]
```

Extract and organize scorer outputs from evaluation runs.

**Returns:**

* <b>`dict[str, dict[str, list[Any]]]`</b>:  A nested dictionary structure where:
  * First level keys are trace IDs (evaluation runs)
  * Second level keys are scorer names
  * Values are lists of scorer outputs for that run and scorer

**Examples:**

```python theme={null}
evaluation = Evaluation(dataset=examples, scorers=[accuracy_scorer, f1_scorer])
await evaluation.evaluate(model)
scores = evaluation.get_scores()
# Access scores by trace and scorer
for trace_id, trace_scores in scores.items():
         print(f"Evaluation run {trace_id}:")
         for scorer_name, outputs in trace_scores.items():
             print(f"  {scorer_name}: {outputs}")
```

Expected output:

```
{
     "trace_123": {
     "accuracy_scorer": [{"accuracy": 0.85}],
     "f1_scorer": [{"f1": 0.78}]
     }
}
```

***

<SourceLink />

### <kbd>method</kbd> `model_post_init`

```python theme={null}
model_post_init(context: Any) → None
```

***

<SourceLink />

### <kbd>method</kbd> `predict_and_score`

```python theme={null}
predict_and_score(model: Op | Model, example: dict) → dict
```

***

<SourceLink />

### <kbd>method</kbd> `summarize`

```python theme={null}
summarize(eval_table: EvaluationResults) → dict
```

***

<SourceLink />

## <kbd>class</kbd> `EvaluationLogger`

This class provides an imperative interface for logging evaluations.

An evaluation is started automatically when the first prediction is logged using the `log_prediction` method, and finished when the `log_summary` method is called.

Each time you log a prediction, you will get back a `ScoreLogger` object. You can use this object to log scores and metadata for that specific prediction. For more information, see the `ScoreLogger` class.

Basic usage - log predictions with inputs and outputs directly:

```python theme={null}
ev = EvaluationLogger()

# Log predictions with known inputs/outputs
pred = ev.log_prediction(inputs={'q': 'Hello'}, outputs={'a': 'Hi there!'})
pred.log_score("correctness", 0.9)

# Finish the evaluation
ev.log_summary({"avg_score": 0.9})
```

Advanced usage - use context manager for dynamic outputs and nested operations:

```python theme={null}
ev = EvaluationLogger()

# Use context manager when you need to capture nested operations
with ev.log_prediction(inputs={'q': 'Hello'}) as pred:
     # Any operations here (like LLM calls) automatically become
     # children of the predict call
     response = your_llm_call(...)
     pred.output = response.content
     pred.log_score("correctness", 0.9)

# Finish the evaluation
ev.log_summary({"avg_score": 0.9})
```

<SourceLink />

### <kbd>method</kbd> `__init__`

```python theme={null}
__init__(
    name: 'str | None' = None,
    model: 'Model | dict | str | None' = None,
    dataset: 'Dataset | list[dict] | str | None' = None,
    eval_attributes: 'dict[str, Any] | None' = None,
    scorers: 'list[str] | None' = None
) → None
```

***

#### <kbd>property</kbd> attributes

***

#### <kbd>property</kbd> ui\_url

***

<SourceLink />

### <kbd>method</kbd> `fail`

```python theme={null}
fail(exception: 'BaseException') → None
```

Convenience method to fail the evaluation with an exception.

***

<SourceLink />

### <kbd>method</kbd> `finish`

```python theme={null}
finish(exception: 'BaseException | None' = None) → None
```

Clean up the evaluation resources explicitly without logging a summary.

Ensures all prediction calls and the main evaluation call are finalized. This is automatically called if the logger is used as a context manager.

***

<SourceLink />

### <kbd>method</kbd> `log_example`

```python theme={null}
log_example(
    inputs: 'dict[str, Any]',
    output: 'Any',
    scores: 'dict[str, ScoreType]'
) → None
```

Log a complete example with inputs, output, and scores.

This is a convenience method that combines log\_prediction and log\_score for when you have all the data upfront.

**Args:**

* <b>`inputs`</b>: The input data for the prediction
* <b>`output`</b>: The output value
* <b>`scores`</b>: Dictionary mapping scorer names to score values
  **Example:**

```python theme={null}
ev = EvaluationLogger()
ev.log_example(
    inputs={'q': 'What is 2+2?'},
    output='4',
    scores={'correctness': 1.0, 'fluency': 0.9}
)
```

***

<SourceLink />

### <kbd>method</kbd> `log_prediction`

```python theme={null}
log_prediction(inputs: 'dict[str, Any]', output: 'Any' = None) → ScoreLogger
```

Log a prediction to the Evaluation.

Returns a ScoreLogger that can be used directly or as a context manager.

**Args:**

* <b>`inputs`</b>: The input data for the prediction
* <b>`output`</b>: The output value. Defaults to None. Can be set later using pred.output.
  **Returns:**
  ScoreLogger for logging scores and optionally finishing the prediction.

Example (direct):

* <b>`pred = ev.log_prediction({'q'`</b>:  '...'}, output="answer") pred.log\_score("correctness", 0.9) pred.finish()

Example (context manager):

* <b>`with ev.log_prediction({'q'`</b>:  '...'}) as pred:  response = model(...)  pred.output = response  pred.log\_score("correctness", 0.9) # Automatically calls finish() on exit

***

<SourceLink />

### <kbd>method</kbd> `log_summary`

```python theme={null}
log_summary(summary: 'dict | None' = None, auto_summarize: 'bool' = True) → None
```

Log a summary dict to the Evaluation.

This will calculate the summary, call the summarize op, and then finalize the evaluation, meaning no more predictions or scores can be logged.

***

<SourceLink />

### <kbd>method</kbd> `set_view`

```python theme={null}
set_view(
    name: 'str',
    content: 'Content | str',
    extension: 'str | None' = None,
    mimetype: 'str | None' = None,
    metadata: 'dict[str, Any] | None' = None,
    encoding: 'str' = 'utf-8'
) → None
```

Attach a view to the evaluation's main call summary under `weave.views`.

Saves the provided content as an object in the project and writes its reference URI under `summary.weave.views.<name>` for the evaluation's `evaluate` call. String inputs are wrapped as text content using `Content.from_text` with the provided extension or mimetype.

**Args:**

* <b>`name`</b>: The view name to display, used as the key under `summary.weave.views`.
* <b>`content`</b>: A `weave.Content` instance or string to serialize.
* <b>`extension`</b>: Optional file extension for string content inputs.
* <b>`mimetype`</b>: Optional MIME type for string content inputs.
* <b>`metadata`</b>: Optional metadata attached to newly created `Content`.
* <b>`encoding`</b>: Text encoding for string content inputs.
  **Returns:**
  None

**Examples:**
` import weave`

> > > ev = weave.EvaluationLogger()
> > > ev.set\_view("report", "# Report", extension="md")

***

<SourceLink />

## <kbd>class</kbd> `File`

A class representing a file with path, mimetype, and size information.

<SourceLink />

### <kbd>method</kbd> `__init__`

```python theme={null}
__init__(path: 'str | Path', mimetype: 'str | None' = None)
```

Initialize a File object.

**Args:**

***

#### <kbd>property</kbd> filename

Get the filename of the file.

* <b>`path`</b>: Path to the file (string or pathlib.Path)

* <b>`mimetype`</b>: Optional MIME type of the file - will be inferred from extension if not provided
  **Returns:**

* <b>`str`</b>:  The name of the file without the directory path.

***

<SourceLink />

### <kbd>method</kbd> `open`

```python theme={null}
open() → bool
```

Open the file using the operating system's default application.

This method uses the platform-specific mechanism to open the file with the default application associated with the file's type.

**Returns:**

* <b>`bool`</b>:  True if the file was successfully opened, False otherwise.

***

<SourceLink />

### <kbd>method</kbd> `save`

```python theme={null}
save(dest: 'str | Path') → None
```

Copy the file to the specified destination path.

**Args:**

***

<SourceLink />

## <kbd>class</kbd> `LLM`

One LLM API call. Maps to a chat OTel span.

* <b>`dest`</b>: Destination path where the file will be copied to (string or pathlib.Path)  The destination path can be a file or a directory.
  **Pydantic Fields:**

* `model`: `<class 'str'>`

* `provider_name`: `<class 'str'>`

* `response_id`: `<class 'str'>`

* `response_model`: `<class 'str'>`

* `output_type`: `<class 'str'>`

* `system_instructions`: `list[str]`

* `usage`: `<class 'session.types.Usage'>`

* `reasoning`: `<class 'session.types.Reasoning'>`

* `finish_reasons`: `list[str]`

* `input_messages`: `list[session.types.Message]`

* `output_messages`: `list[session.types.Message]`

* `media_attachments`: `list[session.types.MediaAttachment]`

* `request_temperature`: `float | None`

* `request_max_tokens`: `int | None`

* `request_top_p`: `float | None`

* `request_frequency_penalty`: `float | None`

* `request_presence_penalty`: `float | None`

* `request_seed`: `int | None`

* `request_stop_sequences`: `list[str]`

* `request_choice_count`: `int | None`

* `started_at`: `datetime.datetime | None`

* `ended_at`: `datetime.datetime | None`

<SourceLink />

### <kbd>method</kbd> `attach_media`

```python theme={null}
attach_media(
    content: 'bytes | str' = '',
    uri: 'str' = '',
    file_id: 'str' = '',
    mime_type: 'str' = '',
    modality: 'str' = ''
) → LLM
```

Attach media to this LLM call.

Exactly one of content, uri, or file\_id must be provided. Modality is inferred from mime\_type when not set explicitly.

***

<SourceLink />

### <kbd>method</kbd> `attach_media_url`

```python theme={null}
attach_media_url(url: 'str', modality: 'str' = '') → LLM
```

Attach a media URL to this LLM call.

Convenience over `attach_media` for the common case where the caller has a URL string from an upstream message and doesn't want to inspect it. `data:` URLs are parsed into `mime_type` + inline content (kind=blob); plain URIs become `kind=uri`. Empty URLs are ignored. Returns `self` for chaining.

***

<SourceLink />

### <kbd>method</kbd> `end`

```python theme={null}
end() → None
```

***

<SourceLink />

### <kbd>method</kbd> `model_post_init`

```python theme={null}
model_post_init(context: 'Any') → None
```

***

<SourceLink />

### <kbd>method</kbd> `output`

```python theme={null}
output(content: 'str') → LLM
```

Append an assistant message to output\_messages.

***

<SourceLink />

### <kbd>method</kbd> `record`

```python theme={null}
record(
    input_messages: 'list[Message] | None' = None,
    output_messages: 'list[Message] | None' = None,
    media_attachments: 'list[MediaAttachment] | None' = None,
    usage: 'Usage | None' = None,
    reasoning: 'Reasoning | str | None' = None,
    response_id: 'str | None' = None,
    response_model: 'str | None' = None,
    finish_reasons: 'list[str] | None' = None,
    output_type: 'str | None' = None
) → LLM
```

Set multiple LLM-call fields in one call.

Manually-instrumented agents typically build up a chat span by assigning eight or more individual fields at the end of an LLM call (`input_messages`, `output_messages`, `usage`, `response_id`, etc.). `record(...)` collapses those into a single keyword call so the recording site stays compact.

Only fields explicitly passed (non-`None`) are applied — existing values are preserved. `reasoning` accepts either a `Reasoning` instance or a plain string (wrapped automatically). Returns `self` for chaining.

***

<SourceLink />

### <kbd>method</kbd> `think`

```python theme={null}
think(content: 'str') → LLM
```

Set reasoning/chain-of-thought content.

***

<SourceLink />

## <kbd>class</kbd> `LogResult`

Result of a batch log\_\* call.

**Pydantic Fields:**

* `session_id`: `<class 'str'>`
* `trace_ids`: `list[str]`
* `root_span_ids`: `list[str]`
* `span_count`: `<class 'int'>`

***

<SourceLink />

## <kbd>class</kbd> `Markdown`

A Markdown renderable.

**Args:**

* <b>`markup`</b> (str):  A string containing markdown.
* <b>`code_theme`</b> (str, optional):  Pygments theme for code blocks. Defaults to "monokai". See [https://pygments.org/styles/](https://pygments.org/styles/) for code themes.
* <b>`justify`</b> (JustifyMethod, optional):  Justify value for paragraphs. Defaults to None.
* <b>`style`</b> (Union\[str, Style], optional):  Optional style to apply to markdown.
* <b>`hyperlinks`</b> (bool, optional):  Enable hyperlinks. Defaults to `True`.

<SourceLink />

### <kbd>method</kbd> `__init__`

```python theme={null}
__init__(
    markup: 'str',
    code_theme: 'str' = 'monokai',
    justify: 'JustifyMethod | None' = None,
    style: 'str | Style' = 'none',
    hyperlinks: 'bool' = True,
    inline_code_lexer: 'str | None' = None,
    inline_code_theme: 'str | None' = None
) → None
```

***

<SourceLink />

## <kbd>class</kbd> `MediaAttachment`

A media attachment on an LLM call.

* <b>`inline_code_lexer`</b>: (str, optional): Lexer to use if inline code highlighting is  enabled. Defaults to None.

* <b>`inline_code_theme`</b>: (Optional\[str], optional): Pygments theme for inline code  highlighting, or None for no highlighting. Defaults to None.
  **Pydantic Fields:**

* `kind`: `typing.Literal['blob', 'uri', 'file']`

* `modality`: `<class 'str'>`

* `mime_type`: `<class 'str'>`

* `content`: `bytes | str`

* `uri`: `<class 'str'>`

* `file_id`: `<class 'str'>`

***

<SourceLink />

## <kbd>class</kbd> `Message`

A single message in a conversation.

Two construction styles are supported:

1. Flat (back-compat, ergonomic for plain text):  `Message(role="assistant", content="Hi there")`

2. Explicit parts (richer — supports tool calls, mixed reasoning+text,  inline media):  `Message(role="assistant", parts=[TextPart(content="Let me check"),  ToolCallPart(id="c1", name="get_weather", arguments='{...}')])`

When `parts` is non-empty it is the canonical representation. When empty, the serializer synthesizes a single TextPart (or ToolCallResponsePart for `role="tool"`) from the flat fields.

**Pydantic Fields:**

* `role`: `typing.Literal['user', 'assistant', 'system', 'tool']`
* `content`: `<class 'str'>`
* `tool_call_id`: `<class 'str'>`
* `tool_name`: `<class 'str'>`
* `parts`: `list[typing.Annotated[session.types.TextPart | session.types.ReasoningPart | session.types.ToolCallPart | session.types.ToolCallResponsePart | session.types.BlobPart | session.types.UriPart | session.types.FilePart, FieldInfo(annotation=NoneType, required=True, discriminator='type')]]`

<SourceLink />

### <kbd>classmethod</kbd> `assistant`

```python theme={null}
assistant(
    text: 'str' = '',
    tool_calls: 'list[ToolCallPart] | None' = None
) → Message
```

Build an assistant message with optional text and tool calls.

Use plain text for simple replies; pass `tool_calls` when the assistant requests one or more tools. When both are present the text is emitted as a leading `TextPart` followed by each `ToolCallPart` so the chat view renders them inline.

***

<SourceLink />

### <kbd>classmethod</kbd> `system`

```python theme={null}
system(text: 'str') → Message
```

Build a system message from plain text.

***

<SourceLink />

### <kbd>classmethod</kbd> `tool_result`

```python theme={null}
tool_result(call_id: 'str', output: 'Any') → Message
```

Build a tool-result message for a previously-requested tool call.

`output` may be a string, dict, list, scalar, or `None` — the underlying `ToolCallResponsePart` JSON-encodes non-strings.

***

<SourceLink />

### <kbd>classmethod</kbd> `user`

```python theme={null}
user(text: 'str') → Message
```

Build a user message from plain text.

***

<SourceLink />

## <kbd>class</kbd> `MessagesPrompt`

<SourceLink />

### <kbd>method</kbd> `__init__`

```python theme={null}
__init__(messages: list[dict])
```

**Pydantic Fields:**

* `name`: `str | None`
* `description`: `str | None`
* `ref`: `trace.refs.ObjectRef | None`
* `messages`: `list[dict]`

<SourceLink />

### <kbd>method</kbd> `format`

```python theme={null}
format(**kwargs: Any) → list
```

***

<SourceLink />

### <kbd>method</kbd> `format_message`

```python theme={null}
format_message(message: dict, **kwargs: Any) → dict
```

Format a single message by replacing template variables.

This method delegates to the standalone format\_message\_with\_template\_vars function for the actual formatting logic.

***

<SourceLink />

### <kbd>classmethod</kbd> `from_obj`

```python theme={null}
from_obj(obj: WeaveObject) → Self
```

***

<SourceLink />

## <kbd>class</kbd> `Model`

Intended to capture a combination of code and data the operates on an input. For example it might call an LLM with a prompt to make a prediction or generate text.

When you change the attributes or the code that defines your model, these changes will be logged and the version will be updated. This ensures that you can compare the predictions across different versions of your model. Use this to iterate on prompts or to try the latest LLM and compare predictions across different settings

**Examples:**

```python theme={null}
class YourModel(Model):
     attribute1: str
     attribute2: int

     @weave.op
     def predict(self, input_data: str) -> dict:
         # Model logic goes here
         prediction = self.attribute1 + ' ' + input_data
         return {'pred': prediction}
```

**Pydantic Fields:**

* `name`: `str | None`
* `description`: `str | None`
* `ref`: `trace.refs.ObjectRef | None`

<SourceLink />

### <kbd>method</kbd> `get_infer_method`

```python theme={null}
get_infer_method() → Callable
```

***

<SourceLink />

## <kbd>class</kbd> `Monitor`

Sets up a monitor to score incoming calls automatically.

**Examples:**

```python theme={null}
import weave
from weave.scorers import ValidJSONScorer

json_scorer = ValidJSONScorer()

my_monitor = weave.Monitor(
     name="my-monitor",
     description="This is a test monitor",
     sampling_rate=0.5,
     op_names=["my_op"],
     query={
         "$expr": {
             "$gt": [
                 {
                         "$getField": "started_at"
                     },
                     {
                         "$literal": 1742540400
                     }
                 ]
             }
         }
     },
     scorers=[json_scorer],
)

my_monitor.activate()
```

**Pydantic Fields:**

* `name`: `str | None`
* `description`: `str | None`
* `ref`: `trace.refs.ObjectRef | None`
* `sampling_rate`: `<class 'float'>`
* `scorers`: `list[flow.scorer.Scorer]`
* `op_names`: `list[typing.Union[typing.Literal['genai.turn_ended'], str]]`
* `query`: `trace_server.interface.query.Query | None`
* `is_traced`: `<class 'bool'>`
* `active`: `<class 'bool'>`
* `scorer_debounce_config`: `flow.monitor.ScorerDebounceConfig | None`

<SourceLink />

### <kbd>method</kbd> `activate`

```python theme={null}
activate() → ObjectRef
```

Activates the monitor.

**Returns:**
The ref to the monitor.

***

<SourceLink />

### <kbd>method</kbd> `deactivate`

```python theme={null}
deactivate() → ObjectRef
```

Deactivates the monitor.

**Returns:**
The ref to the monitor.

***

<SourceLink />

### <kbd>classmethod</kbd> `from_obj`

```python theme={null}
from_obj(obj: WeaveObject) → Self
```

***

<SourceLink />

## <kbd>class</kbd> `Object`

Base class for Weave objects that can be tracked and versioned.

This class extends Pydantic's BaseModel to provide Weave-specific functionality for object tracking, referencing, and serialization. Objects can have names, descriptions, and references that allow them to be stored and retrieved from the Weave system.

**Attributes:**

* <b>`name`</b> (str | None):  A human-readable name for the object.
* <b>`description`</b> (str | None):  A description of what the object represents.
* <b>`ref`</b> (ObjectRef | None):  A reference to the object in the Weave system.

**Examples:**

```python theme={null}
# Create a simple object
obj = Object(name="my_object", description="A test object")

# Create an object from a URI
obj = Object.from_uri("weave:///entity/project/object:digest")
```

**Pydantic Fields:**

* `name`: `str | None`
* `description`: `str | None`
* `ref`: `trace.refs.ObjectRef | None`

<SourceLink />

### <kbd>classmethod</kbd> `from_uri`

```python theme={null}
from_uri(uri: str, objectify: bool = True) → Self
```

Create an object instance from a Weave URI.

**Args:**

* <b>`uri`</b> (str):  The Weave URI pointing to the object.
* <b>`objectify`</b> (bool):  Whether to objectify the result. Defaults to True.

**Returns:**

* <b>`Self`</b>:  An instance of the class created from the URI.

**Raises:**

* <b>`NotImplementedError`</b>:  If the class doesn't implement the required  methods for deserialization.

**Examples:**

```python theme={null}
obj = MyObject.from_uri("weave:///entity/project/object:digest")
```

***

<SourceLink />

### <kbd>classmethod</kbd> `handle_relocatable_object`

```python theme={null}
handle_relocatable_object(
    v: Any,
    handler: ValidatorFunctionWrapHandler,
    info: ValidationInfo
) → Any
```

Handle validation of relocatable objects including ObjectRef and WeaveObject.

This validator handles special cases where the input is an ObjectRef or WeaveObject that needs to be properly converted to a standard Object instance. It ensures that references are preserved and that ignored types are handled correctly during the validation process.

**Args:**

* <b>`v`</b> (Any):  The value to validate.
* <b>`handler`</b> (ValidatorFunctionWrapHandler):  The standard pydantic validation handler.
* <b>`info`</b> (ValidationInfo):  Validation context information.

**Returns:**

* <b>`Any`</b>:  The validated object instance.

**Examples:**
This method is called automatically during object creation and validation. It handles cases like: \`\`\`python

# When an ObjectRef is passed

obj = MyObject(some\_object\_ref)

# When a WeaveObject is passed

obj = MyObject(some\_weave\_object)

````

---

<SourceLink url="https://github.com/wandb/weave/blob/v0.52.42/weave/object/obj.py#L91" />

### <kbd>classmethod</kbd> `strip_weave_serialization_metadata`

```python
strip_weave_serialization_metadata(data: Any) → Any
````

Strip weave serialization metadata from dict inputs.

Weave's serialization adds \_type, \_class\_name, and \_bases to dicts for type reconstruction.  These are not real model fields and must be removed before Pydantic validation, which uses extra="forbid".

***

<SourceLink />

## <kbd>class</kbd> `ObjectRef`

ObjectRef(entity: 'str', project: 'str', name: 'str', \_digest: 'str | Future\[str]', \_extra: 'tuple\[str | Future\[str], ...]' = ())

<SourceLink />

### <kbd>method</kbd> `__init__`

```python theme={null}
__init__(
    entity: 'str',
    project: 'str',
    name: 'str',
    _digest: 'str | Future[str]',
    _extra: 'tuple[str | Future[str], ]' = ()
) → None
```

***

#### <kbd>property</kbd> digest

***

#### <kbd>property</kbd> extra

***

#### <kbd>property</kbd> is\_digest\_resolved

***

<SourceLink />

### <kbd>method</kbd> `as_param_dict`

```python theme={null}
as_param_dict() → dict
```

***

<SourceLink />

### <kbd>method</kbd> `delete`

```python theme={null}
delete() → None
```

***

<SourceLink />

### <kbd>method</kbd> `get`

```python theme={null}
get(objectify: 'bool' = True) → Any
```

***

<SourceLink />

### <kbd>method</kbd> `is_descended_from`

```python theme={null}
is_descended_from(potential_ancestor: 'ObjectRef') → bool
```

***

<SourceLink />

### <kbd>method</kbd> `maybe_parse_uri`

```python theme={null}
maybe_parse_uri(s: 'str') → AnyRef | None
```

***

<SourceLink />

### <kbd>method</kbd> `parse_uri`

```python theme={null}
parse_uri(uri: 'str') → ObjectRef
```

***

<SourceLink />

### <kbd>method</kbd> `with_attr`

```python theme={null}
with_attr(attr: 'str') → Self
```

***

<SourceLink />

### <kbd>method</kbd> `with_extra`

```python theme={null}
with_extra(extra: 'tuple[str | Future[str], ]') → Self
```

***

<SourceLink />

### <kbd>method</kbd> `with_index`

```python theme={null}
with_index(index: 'int') → Self
```

***

<SourceLink />

### <kbd>method</kbd> `with_item`

```python theme={null}
with_item(item_digest: 'str | Future[str]') → Self
```

***

<SourceLink />

### <kbd>method</kbd> `with_key`

```python theme={null}
with_key(key: 'str') → Self
```

***

<SourceLink />

## <kbd>class</kbd> `Prompt`

**Pydantic Fields:**

* `name`: `str | None`
* `description`: `str | None`
* `ref`: `trace.refs.ObjectRef | None`

<SourceLink />

### <kbd>method</kbd> `format`

```python theme={null}
format(**kwargs: Any) → Any
```

***

<SourceLink />

## <kbd>class</kbd> `SavedView`

A fluent-style class for working with SavedView objects.

<SourceLink />

### <kbd>method</kbd> `__init__`

```python theme={null}
__init__(view_type: 'str' = 'traces', label: 'str' = 'SavedView') → None
```

***

#### <kbd>property</kbd> entity

***

#### <kbd>property</kbd> label

***

#### <kbd>property</kbd> project

***

#### <kbd>property</kbd> view\_type

***

<SourceLink />

### <kbd>method</kbd> `add_column`

```python theme={null}
add_column(path: 'str | ObjectPath', label: 'str | None' = None) → SavedView
```

***

<SourceLink />

### <kbd>method</kbd> `add_columns`

```python theme={null}
add_columns(*columns: 'str') → SavedView
```

Convenience method for adding multiple columns to the grid.

***

<SourceLink />

### <kbd>method</kbd> `add_filter`

```python theme={null}
add_filter(
    field: 'str',
    operator: 'str',
    value: 'Any | None' = None
) → SavedView
```

***

<SourceLink />

### <kbd>method</kbd> `add_sort`

```python theme={null}
add_sort(field: 'str', direction: 'SortDirection') → SavedView
```

***

<SourceLink />

### <kbd>method</kbd> `column_index`

```python theme={null}
column_index(path: 'int | str | ObjectPath') → int
```

***

<SourceLink />

### <kbd>method</kbd> `filter_op`

```python theme={null}
filter_op(op_name: 'str | None') → SavedView
```

***

<SourceLink />

### <kbd>method</kbd> `get_calls`

```python theme={null}
get_calls(
    limit: 'int | None' = None,
    offset: 'int | None' = None,
    include_costs: 'bool' = False,
    include_feedback: 'bool' = False,
    all_columns: 'bool' = False
) → CallsIter
```

Get calls matching this saved view's filters and settings.

***

<SourceLink />

### <kbd>method</kbd> `get_known_columns`

```python theme={null}
get_known_columns(num_calls_to_query: 'int | None' = None) → list[str]
```

Get the set of columns that are known to exist.

***

<SourceLink />

### <kbd>method</kbd> `get_table_columns`

```python theme={null}
get_table_columns() → list[TableColumn]
```

***

<SourceLink />

### <kbd>method</kbd> `hide_column`

```python theme={null}
hide_column(col_name: 'str') → SavedView
```

***

<SourceLink />

### <kbd>method</kbd> `insert_column`

```python theme={null}
insert_column(
    idx: 'int',
    path: 'str | ObjectPath',
    label: 'str | None' = None
) → SavedView
```

***

<SourceLink />

### <kbd>classmethod</kbd> `load`

```python theme={null}
load(ref: 'str') → Self
```

***

<SourceLink />

### <kbd>method</kbd> `page_size`

```python theme={null}
page_size(page_size: 'int') → SavedView
```

***

<SourceLink />

### <kbd>method</kbd> `pin_column_left`

```python theme={null}
pin_column_left(col_name: 'str') → SavedView
```

***

<SourceLink />

### <kbd>method</kbd> `pin_column_right`

```python theme={null}
pin_column_right(col_name: 'str') → SavedView
```

***

<SourceLink />

### <kbd>method</kbd> `remove_column`

```python theme={null}
remove_column(path: 'int | str | ObjectPath') → SavedView
```

***

<SourceLink />

### <kbd>method</kbd> `remove_columns`

```python theme={null}
remove_columns(*columns: 'str') → SavedView
```

Remove columns from the saved view.

***

<SourceLink />

### <kbd>method</kbd> `remove_filter`

```python theme={null}
remove_filter(index_or_field: 'int | str') → SavedView
```

***

<SourceLink />

### <kbd>method</kbd> `remove_filters`

```python theme={null}
remove_filters() → SavedView
```

Remove all filters from the saved view.

***

<SourceLink />

### <kbd>method</kbd> `rename`

```python theme={null}
rename(label: 'str') → SavedView
```

***

<SourceLink />

### <kbd>method</kbd> `rename_column`

```python theme={null}
rename_column(path: 'int | str | ObjectPath', label: 'str') → SavedView
```

***

<SourceLink />

### <kbd>method</kbd> `save`

```python theme={null}
save() → SavedView
```

Publish the saved view to the server.

***

<SourceLink />

### <kbd>method</kbd> `set_columns`

```python theme={null}
set_columns(*columns: 'str') → SavedView
```

Set the columns to be displayed in the grid.

***

<SourceLink />

### <kbd>method</kbd> `show_column`

```python theme={null}
show_column(col_name: 'str') → SavedView
```

***

<SourceLink />

### <kbd>method</kbd> `sort_by`

```python theme={null}
sort_by(field: 'str', direction: 'SortDirection') → SavedView
```

***

<SourceLink />

### <kbd>method</kbd> `to_grid`

```python theme={null}
to_grid(limit: 'int | None' = None) → Grid
```

***

<SourceLink />

### <kbd>method</kbd> `to_rich_table_str`

```python theme={null}
to_rich_table_str() → str
```

***

<SourceLink />

### <kbd>method</kbd> `ui_url`

```python theme={null}
ui_url() → str | None
```

URL to show this saved view in the UI.

Note this is the "result" page with traces etc, not the URL for the view object.

***

<SourceLink />

### <kbd>method</kbd> `unpin_column`

```python theme={null}
unpin_column(col_name: 'str') → SavedView
```

***

<SourceLink />

## <kbd>class</kbd> `Scorer`

**Pydantic Fields:**

* `name`: `str | None`
* `description`: `str | None`
* `ref`: `trace.refs.ObjectRef | None`
* `column_map`: `dict[str, str] | None`

***

#### <kbd>property</kbd> display\_name

<SourceLink />

### <kbd>classmethod</kbd> `from_obj`

```python theme={null}
from_obj(obj: WeaveObject) → Self
```

***

<SourceLink />

### <kbd>method</kbd> `model_post_init`

```python theme={null}
model_post_init(context: Any) → None
```

***

<SourceLink />

### <kbd>method</kbd> `score`

```python theme={null}
score(output: Any, **kwargs: Any) → Any
```

***

<SourceLink />

### <kbd>method</kbd> `summarize`

```python theme={null}
summarize(score_rows: list) → dict | None
```

***

<SourceLink />

## <kbd>class</kbd> `Session`

A conversation session. Groups turns by conversation\_id (no span).

`continue_parent_trace` controls trace isolation for the turns this session creates. Default `False` means each turn starts its own OTel trace (the right choice for the standalone Agents tab view). Set `True` when the application has an outer trace (e.g. a fastapi-instrumented request) that should contain the agent invocation.

**Pydantic Fields:**

* `session_id`: `<class 'str'>`
* `session_name`: `<class 'str'>`
* `agent_name`: `<class 'str'>`
* `model`: `<class 'str'>`
* `include_content`: `<class 'bool'>`
* `continue_parent_trace`: `<class 'bool'>`

<SourceLink />

### <kbd>method</kbd> `end`

```python theme={null}
end() → None
```

***

<SourceLink />

### <kbd>method</kbd> `model_post_init`

```python theme={null}
model_post_init(context: 'Any') → None
```

***

<SourceLink />

### <kbd>method</kbd> `start_turn`

```python theme={null}
start_turn(
    user_message: 'str' = '',
    model: 'str' = '',
    agent_name: 'str' = ''
) → Turn
```

Create a new turn. Auto-ends the previous turn if still open.

Sets the `_current_turn` contextvar so the turn is visible via `get_current_turn()` regardless of whether a context manager is used. Propagates `continue_parent_trace` from this session.

***

<SourceLink />

## <kbd>class</kbd> `StringPrompt`

<SourceLink />

### <kbd>method</kbd> `__init__`

```python theme={null}
__init__(content: str)
```

**Pydantic Fields:**

* `name`: `str | None`
* `description`: `str | None`
* `ref`: `trace.refs.ObjectRef | None`
* `content`: `<class 'str'>`

<SourceLink />

### <kbd>method</kbd> `format`

```python theme={null}
format(**kwargs: Any) → str
```

***

<SourceLink />

### <kbd>classmethod</kbd> `from_obj`

```python theme={null}
from_obj(obj: WeaveObject) → Self
```

***

<SourceLink />

## <kbd>class</kbd> `SubAgent`

A delegated agent invocation within a turn.

Maps to a nested invoke\_agent OTel span in the same trace.

**Pydantic Fields:**

* `name`: `<class 'str'>`
* `model`: `<class 'str'>`
* `agent_id`: `<class 'str'>`
* `agent_description`: `<class 'str'>`
* `agent_version`: `<class 'str'>`
* `started_at`: `datetime.datetime | None`
* `ended_at`: `datetime.datetime | None`

<SourceLink />

### <kbd>method</kbd> `end`

```python theme={null}
end() → None
```

***

<SourceLink />

### <kbd>method</kbd> `llm`

```python theme={null}
llm(
    model: 'str' = '',
    provider_name: 'str' = '',
    system_instructions: 'list[str] | None' = None
) → LLM
```

Start an LLM call within this sub-agent.

Sets the `_current_llm` contextvar so the LLM is visible via `get_current_llm()` regardless of whether a context manager is used.

***

<SourceLink />

### <kbd>method</kbd> `tool`

```python theme={null}
tool(name: 'str', arguments: 'str' = '', tool_call_id: 'str' = '') → Tool
```

Start a tool execution within this sub-agent.

***

<SourceLink />

## <kbd>class</kbd> `Table`

<SourceLink />

### <kbd>method</kbd> `__init__`

```python theme={null}
__init__(rows: 'list[dict]') → None
```

***

#### <kbd>property</kbd> rows

***

<SourceLink />

### <kbd>method</kbd> `append`

```python theme={null}
append(row: 'dict') → None
```

Add a row to the table.

***

<SourceLink />

### <kbd>method</kbd> `pop`

```python theme={null}
pop(index: 'int') → None
```

Remove a row at the given index from the table.

***

<SourceLink />

## <kbd>class</kbd> `ContextAwareThread`

A Thread that runs functions with the context of the caller.

This is a drop-in replacement for threading.Thread that ensures calls behave as expected inside the thread.  Weave requires certain contextvars to be set (see call\_context.py), but new threads do not automatically copy context from the parent, which can cause the call context to be lost -- not good!  This class automates contextvar copying so using this thread "just works" as the user probably expects.

You can achieve the same effect without this class by instead writing:

```python theme={null}
def run_with_context(func, *args, **kwargs):
     context = copy_context()
     def wrapper():
         context.run(func, *args, **kwargs)
     return wrapper

thread = threading.Thread(target=run_with_context(your_func, *args, **kwargs))
thread.start()
```

<SourceLink />

### <kbd>method</kbd> `__init__`

```python theme={null}
__init__(*args: 'Any', **kwargs: 'Any') → None
```

***

#### <kbd>property</kbd> daemon

A boolean value indicating whether this thread is a daemon thread.

This must be set before start() is called, otherwise RuntimeError is raised. Its initial value is inherited from the creating thread; the main thread is not a daemon thread and therefore all threads created in the main thread default to daemon = False.

The entire Python program exits when only daemon threads are left.

***

#### <kbd>property</kbd> ident

Thread identifier of this thread or None if it has not been started.

This is a nonzero integer. See the get\_ident() function. Thread identifiers may be recycled when a thread exits and another thread is created. The identifier is available even after the thread has exited.

***

#### <kbd>property</kbd> name

A string used for identification purposes only.

It has no semantics. Multiple threads may be given the same name. The initial name is set by the constructor.

***

#### <kbd>property</kbd> native\_id

Native integral thread ID of this thread, or None if it has not been started.

This is a non-negative integer. See the get\_native\_id() function. This represents the Thread ID as reported by the kernel.

***

<SourceLink />

### <kbd>method</kbd> `run`

```python theme={null}
run() → None
```

***

<SourceLink />

## <kbd>class</kbd> `ThreadContext`

Context object providing access to current thread and turn information.

<SourceLink />

### <kbd>method</kbd> `__init__`

```python theme={null}
__init__(thread_id: 'str | None')
```

Initialize ThreadContext with the specified thread\_id.

**Args:**

***

#### <kbd>property</kbd> thread\_id

Get the thread\_id for this context.

* <b>`thread_id`</b>: The thread identifier for this context, or None if disabled.
  **Returns:**
  The thread identifier, or None if thread tracking is disabled.

***

#### <kbd>property</kbd> turn\_id

Get the current turn\_id from the active context.

**Returns:**
The current turn\_id if set, None otherwise.

***

<SourceLink />

## <kbd>class</kbd> `ContextAwareThreadPoolExecutor`

A ThreadPoolExecutor that runs functions with the context of the caller.

This is a drop-in replacement for concurrent.futures.ThreadPoolExecutor that ensures weave calls behave as expected inside the executor.  Weave requires certain contextvars to be set (see call\_context.py), but new threads do not automatically copy context from the parent, which can cause the call context to be lost -- not good!  This class automates contextvar copying so using this executor "just works" as the user probably expects.

You can achieve the same effect without this class by instead writing:

```python theme={null}
with concurrent.futures.ThreadPoolExecutor() as executor:
     contexts = [copy_context() for _ in range(len(vals))]

     def _wrapped_fn(*args):
         return contexts.pop().run(fn, *args)

     executor.map(_wrapped_fn, vals)
```

<SourceLink />

### <kbd>method</kbd> `__init__`

```python theme={null}
__init__(*args: 'Any', **kwargs: 'Any') → None
```

***

<SourceLink />

### <kbd>method</kbd> `map`

```python theme={null}
map(
    fn: 'Callable',
    *iterables: 'Iterable[Any]',
    timeout: 'float | None' = None,
    chunksize: 'int' = 1
) → Iterator
```

***

<SourceLink />

### <kbd>method</kbd> `submit`

```python theme={null}
submit(fn: 'Callable', *args: 'Any', **kwargs: 'Any') → Any
```

***

<SourceLink />

## <kbd>class</kbd> `Tool`

One tool execution. Maps to an execute\_tool OTel span.

`arguments` and `result` use the `JSONString` annotation: callers can assign a dict / list / scalar and the SDK JSON-encodes it at construction or assignment. The stored value is always a string, matching the wire format per GenAI semconv.

**Pydantic Fields:**

* `name`: `<class 'str'>`
* `arguments`: `<class 'str'>`
* `result`: `<class 'str'>`
* `tool_call_id`: `<class 'str'>`
* `tool_type`: `<class 'str'>`
* `tool_description`: `<class 'str'>`
* `tool_definitions`: `<class 'str'>`
* `duration_ms`: `<class 'int'>`
* `started_at`: `datetime.datetime | None`
* `ended_at`: `datetime.datetime | None`

<SourceLink />

### <kbd>method</kbd> `end`

```python theme={null}
end() → None
```

***

<SourceLink />

## <kbd>class</kbd> `Turn`

One user-agent exchange. Maps to an invoke\_agent OTel span.

By default each turn starts its own OTel trace (`continue_parent_trace=False`) so the Agents tab shows one trace per turn. Set `continue_parent_trace=True` on the Session (or directly on the Turn) when an outer trace is already active and you want the agent invocation to nest inside it — e.g. inside a fastapi-instrumented request.

**Pydantic Fields:**

* `agent_name`: `<class 'str'>`
* `model`: `<class 'str'>`
* `agent_id`: `<class 'str'>`
* `agent_description`: `<class 'str'>`
* `agent_version`: `<class 'str'>`
* `messages`: `list[session.types.Message]`
* `spans`: `list[session.session.LLM | session.session.Tool | session.session.SubAgent]`
* `continue_parent_trace`: `<class 'bool'>`
* `started_at`: `datetime.datetime | None`
* `ended_at`: `datetime.datetime | None`

<SourceLink />

### <kbd>method</kbd> `end`

```python theme={null}
end() → None
```

***

<SourceLink />

### <kbd>method</kbd> `llm`

```python theme={null}
llm(
    model: 'str' = '',
    provider_name: 'str' = '',
    system_instructions: 'list[str] | None' = None
) → LLM
```

Start an LLM call (chat span, child of this turn).

Sets the `_current_llm` contextvar so the LLM is visible via `get_current_llm()` regardless of whether a context manager is used.

***

<SourceLink />

### <kbd>method</kbd> `model_post_init`

```python theme={null}
model_post_init(context: 'Any') → None
```

***

<SourceLink />

### <kbd>method</kbd> `subagent`

```python theme={null}
subagent(name: 'str', model: 'str' = '') → SubAgent
```

Start a sub-agent invocation (nested invoke\_agent span, same trace).

***

<SourceLink />

### <kbd>method</kbd> `tool`

```python theme={null}
tool(name: 'str', arguments: 'str' = '', tool_call_id: 'str' = '') → Tool
```

Start a tool execution (execute\_tool span, child of this turn).

***

<SourceLink />

### <kbd>method</kbd> `user`

```python theme={null}
user(content: 'str') → Turn
```

Append a user message mid-turn.

***

<SourceLink />

## <kbd>class</kbd> `Usage`

Token usage for an LLM call.

**Pydantic Fields:**

* `input_tokens`: `<class 'int'>`
* `output_tokens`: `<class 'int'>`
* `reasoning_tokens`: `<class 'int'>`
* `cache_creation_input_tokens`: `<class 'int'>`
* `cache_read_input_tokens`: `<class 'int'>`

***

<SourceLink />

### <kbd>function</kbd> `add_tags`

```python theme={null}
add_tags(obj_ref: 'ObjectRef | str', tags: 'list[str]') → None
```

Add tags to an object version.

**Args:**

***

<SourceLink />

### <kbd>function</kbd> `as_op`

```python theme={null}
as_op(fn: 'Callable[P, R]') → Op[P, R]
```

Given a @weave.op decorated function, return its Op.

@weave.op decorated functions are instances of Op already, so this function should be a no-op at runtime. But you can use it to satisfy type checkers if you need to access OpDef attributes in a typesafe way.

* <b>`obj_ref`</b>: Reference to the object version, either an ObjectRef (returned by weave.publish()) or a weave /// URI string.

* <b>`tags`</b>: List of tag strings to add.
  **Args:**

* <b>`fn`</b>: A weave.op decorated function.
  **Returns:**
  The Op of the function.

***

<SourceLink />

### <kbd>function</kbd> `attributes`

```python theme={null}
attributes(attributes: 'dict[str, Any]') → Iterator
```

Context manager for setting attributes on a call.

**Example:**

```python theme={null}
with weave.attributes({'env': 'production'}):
     print(my_function.call("World"))
```

***

<SourceLink />

### <kbd>function</kbd> `end_llm`

```python theme={null}
end_llm() → None
```

End the current LLM call (from contextvar).

***

<SourceLink />

### <kbd>function</kbd> `end_session`

```python theme={null}
end_session() → None
```

End the current session (from contextvar).

***

<SourceLink />

### <kbd>function</kbd> `end_turn`

```python theme={null}
end_turn() → None
```

End the current turn (from contextvar).

***

<SourceLink />

### <kbd>function</kbd> `finish`

```python theme={null}
finish() → None
```

Stops logging to weave.

Following finish, calls of weave.op decorated functions will no longer be logged. You will need to run weave.init() again to resume logging.

***

<SourceLink />

### <kbd>function</kbd> `get`

```python theme={null}
get(uri: 'str | ObjectRef') → Any
```

A convenience function for getting an object from a URI.

Many objects logged by Weave are automatically registered with the Weave server. This function allows you to retrieve those objects by their URI.

**Args:**

* <b>`uri`</b>: A fully-qualified weave ref URI.
  **Returns:**
  The object.

**Example:**

```python theme={null}
weave.init("weave_get_example")
dataset = weave.Dataset(rows=[{"a": 1, "b": 2}])
ref = weave.publish(dataset)

dataset2 = weave.get(ref)  # same as dataset!
```

***

<SourceLink />

### <kbd>function</kbd> `get_aliases`

```python theme={null}
get_aliases(obj_ref: 'ObjectRef | str') → list[str]
```

Get aliases for an object version.

**Args:**

* <b>`obj_ref`</b>: Reference to the object version, either an ObjectRef or a weave /// URI string.
  **Returns:**
  List of alias strings.

***

<SourceLink />

### <kbd>function</kbd> `get_client`

```python theme={null}
get_client() → WeaveClient | None
```

***

<SourceLink />

### <kbd>function</kbd> `get_current_call`

```python theme={null}
get_current_call() → Call | None
```

Get the Call object for the currently executing Op, within that Op.

**Returns:**
The Call object for the currently executing Op, or  None if tracking has not been initialized or this method is  invoked outside an Op.

**Note:**

> The returned Call's `attributes` dictionary becomes immutable once the call starts. Use :func:`weave.attributes` to set call metadata before invoking an Op. The `summary` field may be updated while the Op executes and will be merged with computed summary information when the call finishes.

***

<SourceLink />

### <kbd>function</kbd> `get_current_llm`

```python theme={null}
get_current_llm() → LLM | None
```

Return the active LLM call from contextvar, or None.

***

<SourceLink />

### <kbd>function</kbd> `get_current_session`

```python theme={null}
get_current_session() → Session | None
```

Return the active session from contextvar, or None.

***

<SourceLink />

### <kbd>function</kbd> `get_current_turn`

```python theme={null}
get_current_turn() → Turn | None
```

Return the active turn from contextvar, or None.

***

<SourceLink />

### <kbd>function</kbd> `get_tags`

```python theme={null}
get_tags(obj_ref: 'ObjectRef | str') → list[str]
```

Get tags for an object version.

**Args:**

* <b>`obj_ref`</b>: Reference to the object version, either an ObjectRef or a weave /// URI string.
  **Returns:**
  List of tag strings.

***

<SourceLink />

### <kbd>function</kbd> `get_tags_and_aliases`

```python theme={null}
get_tags_and_aliases(obj_ref: 'ObjectRef | str') → tuple[list[str], list[str]]
```

Get both tags and aliases for an object version in a single call.

**Args:**

* <b>`obj_ref`</b>: Reference to the object version, either an ObjectRef or a weave /// URI string.
  **Returns:**
  A tuple of (tags, aliases). Each is a list of strings.

***

<SourceLink />

### <kbd>function</kbd> `init`

```python theme={null}
init(
    project_name: 'str',
    settings: 'UserSettings | dict[str, Any] | None' = None,
    autopatch_settings: 'AutopatchSettings | None' = None,
    postprocess_inputs: 'PostprocessInputsFunc | None' = None,
    postprocess_output: 'PostprocessOutputFunc | None' = None,
    attributes: 'dict[str, Any] | None' = None,
    global_postprocess_inputs: 'PostprocessInputsFunc | None' = None,
    global_postprocess_output: 'PostprocessOutputFunc | None' = None,
    global_attributes: 'dict[str, Any] | None' = None
) → WeaveClient
```

Initialize weave tracking, logging to a wandb project.

Logging is initialized globally, so you do not need to keep a reference to the return value of init.

Following init, calls of weave.op decorated functions will be logged to the specified project.

**Args:**

NOTE: Client-level postprocessing runs after each op's own postprocessing. The order is always: 1. Op-specific postprocessing 2. Client-level postprocessing

* <b>`project_name`</b>: The name of the Weights & Biases team and project to log to. If you don't  specify a team, your default entity is used. To find or update your default entity, refer to [User Settings](https://docs.wandb.ai/platform/app/settings-page/user-settings#default-team) in the W\&B Models documentation.
* <b>`settings`</b>: Configuration for the Weave client generally. Can be a UserSettings instance or a dict  with any of the following keys (all optional). All settings can also be configured  via environment variables using the prefix WEAVE\_ (e.g., WEAVE\_DISABLED=true).  Available settings: - `disabled` (bool): Disables traces on all functions. Default: `False` - `print_call_link` (bool): Prints links in terminal to Weave UI for ops. Default: `True` - `log_level` (str): Sets what type of information to log (`DEBUG`, `INFO`, `WARNING`, `ERROR`, `CRITICAL`). Default: `INFO` - `display_viewer` (str): Controls how Weave displays objects in the console (`auto`, `rich`, `print`). Default: `auto` - `capture_code` (bool): Captures code of traced ops to your Weave project. Default: `True` - `implicitly_patch_integrations` (bool): Auto-patches supported libraries. Default: `True` - `redact_pii` (bool): Scans all trace data for sensitive information,  like emails, phone numbers, and credit cards, and replaces them with placeholder values  before sending to the server. Requires presidio-analyzer and presidio-anonymizer packages.
* <b>`Default`</b>: `False` - `redact_pii_fields` (list\[str]): Specifies which PII entity types to redact when `redact_pii` is True. If empty, uses Presidio's default set. Examples  \['EMAIL','PHONE\_NUMBER','CREDIT\_CARD','US\_SSN']. See full list at  [https://microsoft.github.io/presidio/supported\_entities/](https://microsoft.github.io/presidio/supported_entities/)
* <b>`Default`</b>: `[]` - `redact_pii_exclude_fields` (list\[str]): PII entity types to exclude. Default: `[]` - `capture_client_info` (bool): Captures Python/SDK version info. Default: `True` - `capture_system_info` (bool): Captures OS information. Default: `True` - `client_parallelism` (int): Number of workers for background ops. Default: `auto` - `use_server_cache` (bool): Enables local disk caching of server responses. - `server_cache_size_limit` (int): Cache size limit in bytes. Default: `1_000_000_000` - `server_cache_dir` (str): Directory for server cache. Default: `temporary` - `scorers_dir` (str): Directory for scorer model checkpoints. Default: `~/.cache/wandb/weave-scorers` - `max_calls_queue_size` (int): Maximum queue size (0 = unbounded). Default: `100_000` - `retry_max_interval` (float): Maximum retry interval in seconds. Default: `300` - `retry_max_attempts` (int): Maximum number of retries. Default: `3` - `enable_disk_fallback` (bool): Writes dropped items to disk. Default: `True` - `use_parallel_table_upload` (bool): Enables parallel chunked upload for large tables. If False, tables are uploaded sequentially in smaller chunks.
* <b>`Default`</b>: `True` - `http_timeout` (float): Maximum time in seconds to wait for HTTP requests to complete. This includes connection time, data transfer, and server processing. Increase for slow networks or when working with large payloads.
* <b>`Default`</b>: `30.0` - `use_stainless_server` (bool): Uses the Stainless-generated HTTP client which provides better type safety, automatic retries, and improved error handling. This is experimental and may become the default in future versions.
* <b>`Default`</b>: `False` - `use_calls_complete` (bool): Uses an optimized write path that batches complete call data (start and end) into a single request instead of separate start/end requests. This reduces server load and improves performance, especially for short-lived ops.
* <b>`Default`</b>: `True`
* <b>`autopatch_settings`</b>: (Deprecated) Configuration for autopatch integrations. Use explicit patching instead.
* <b>`postprocess_inputs`</b>: A function applied to the inputs of every op traced by this client.
* <b>`postprocess_output`</b>: A function applied to the output of every op traced by this client.
* <b>`attributes`</b>: A dictionary of attributes applied to every trace produced by this client.
  **Returns:**
  A Weave client.

***

<SourceLink />

### <kbd>function</kbd> `link_prompt_to_registry`

```python theme={null}
link_prompt_to_registry(
    prompt: 'LinkablePrompt',
    target_path: 'str',
    aliases: 'Sequence[str] | None' = None
) → LinkAssetToRegistryRes
```

Link a published prompt version into the registry.

**Args:**

* <b>`prompt`</b>: A published prompt, an `ObjectRef`, or a fully qualified weave ///... URI string.

* <b>`target_path`</b>: Registry destination path in the format  `<registry_project>/<portfolio_name>`, for example  `wandb-registry-prompts/my-prompt-collection`.

* <b>`aliases`</b>: Optional aliases to attach to the created registry version.
  **Returns:**

* <b>`LinkAssetToRegistryRes`</b>:  Parsed response from the registry-link endpoint.

***

<SourceLink />

### <kbd>function</kbd> `list_aliases`

```python theme={null}
list_aliases() → list[str]
```

List all distinct aliases in the project.

**Returns:**
Sorted list of all alias strings in the project.

***

<SourceLink />

### <kbd>function</kbd> `list_tags`

```python theme={null}
list_tags() → list[str]
```

List all distinct tags in the project.

**Returns:**
Sorted list of all tag strings in the project.

***

<SourceLink />

### <kbd>function</kbd> `log_call`

```python theme={null}
log_call(
    op: 'str',
    inputs: 'dict[str, Any]',
    output: 'Any',
    parent: 'Call | None' = None,
    attributes: 'dict[str, Any] | None' = None,
    display_name: 'str | Callable[[Call], str] | None' = None,
    use_stack: 'bool' = True,
    exception: 'BaseException | None' = None
) → Call
```

Log a call directly to Weave without using the decorator pattern.

This function provides an imperative API for logging operations to Weave, useful when you want to log calls after they've already been executed or when the decorator pattern isn't suitable for your use case.

**Args:**

* <b>`op`</b> (str):  The operation name to log. This will be used as the op\_name  for the call. Anonymous operations (strings not referring to published  ops) are supported.
* <b>`inputs`</b> (dict\[str, Any]):  A dictionary of input parameters for the operation.
* <b>`output`</b> (Any):  The output/result of the operation.
* <b>`parent`</b> (Call | None):  Optional parent call to nest this call under.  If not provided, the call will be a root-level call (or nested under  the current call context if one exists). Defaults to None.
* <b>`attributes`</b> (dict\[str, Any] | None):  Optional metadata to attach to the call.  These are frozen once the call is created. Defaults to None.
* <b>`display_name`</b> (str | Callable\[\[Call], str] | None):  Optional display name  for the call in the UI. Can be a string or a callable that takes the  call and returns a string. Defaults to None.
* <b>`use_stack`</b> (bool):  Whether to push the call onto the runtime stack. When True,  the call will be available in the call context and can be accessed via  weave.require\_current\_call(). When False, the call is logged but not  added to the call stack. Defaults to True.
* <b>`exception`</b> (BaseException | None):  Optional exception to log if the operation  failed. Defaults to None.

**Returns:**

* <b>`Call`</b>:  The created and finished Call object with full trace information.

**Examples:**
Basic usage:

````python theme={null}
import weave
    >>> weave.init('my-project')
    >>> call = weave.log_call(
    ...     op="my_function",
    ...     inputs={"x": 5, "y": 10},
    ...     output=15
    ... )

    Logging with attributes and display name:
    >>> call = weave.log_call(
    ...     op="process_data",
    ...     inputs={"data": [1, 2, 3]},
    ...     output={"mean": 2.0},
    ...     attributes={"version": "1.0", "env": "prod"},
    ...     display_name="Data Processing"
    ... )

    Logging a failed operation:
    >>> try:
    ...     result = risky_operation()
    ... except Exception as e:
    ...     call = weave.log_call(
    ...         op="risky_operation",
    ...         inputs={},
    ...         output=None,
    ...         exception=e
    ...     )

    Nesting calls:
    >>> parent_call = weave.log_call("parent", {"input": 1}, 2)
    >>> child_call = weave.log_call(
    ...     "child",
    ...     {"input": 2},
    ...     4,
    ...     parent=parent_call
    ... )

    Logging without adding to call stack:
    >>> call = weave.log_call(
    ...     op="background_task",
    ...     inputs={"task_id": 123},
    ...     output="completed",
    ...     use_stack=False  # Don't push to call stack
    ... )

---

<SourceLink url="https://github.com/wandb/weave/blob/v0.52.42/weave/session/session.py#L1166" />

### <kbd>function</kbd> `log_session`

```python
log_session(
    turns: 'list[Turn]',
    session_id: 'str' = '',
    session_name: 'str' = '',
    agent_name: 'str' = '',
    model: 'str' = '',
    include_content: 'bool' = True,
    continue_parent_trace: 'bool' = False
) → LogResult
````

Imperatively emit a complete session.

Each Turn's `.spans` attribute provides its children. Auto-generates `session_id` if empty. By default each turn gets its own OTel trace.

***

<SourceLink />

### <kbd>function</kbd> `log_turn`

```python theme={null}
log_turn(
    session_id: 'str',
    agent_name: 'str' = '',
    session_name: 'str' = '',
    model: 'str' = '',
    messages: 'list[Message] | None' = None,
    spans: 'list[LLM | Tool | SubAgent] | None' = None,
    started_at: 'datetime | None' = None,
    ended_at: 'datetime | None' = None,
    include_content: 'bool' = True,
    continue_parent_trace: 'bool' = False
) → LogResult
```

Imperatively emit one turn and its child spans to OTel.

Use when context managers aren't viable (stateless containers, callbacks, queue workers). Each child span passed in should have `started_at` / `ended_at` set; the emitted OTel span timestamps come from those fields. Falls back to the earliest/latest child timestamp, then `now()`, when the turn doesn't supply its own.

***

<SourceLink />

### <kbd>function</kbd> `op`

```python theme={null}
op(
    func: 'Callable[P, R] | None' = None,
    name: 'str | None' = None,
    call_display_name: 'str | CallDisplayNameFunc | None' = None,
    postprocess_inputs: 'PostprocessInputsFunc | None' = None,
    postprocess_output: 'PostprocessOutputFunc | None' = None,
    tracing_sample_rate: 'float' = 1.0,
    enable_code_capture: 'bool' = True,
    accumulator: 'Callable[[Any | None, Any], Any] | None' = None,
    kind: 'OpKind | None' = None,
    color: 'OpColor | None' = None,
    eager_call_start: 'bool' = False
) → Callable[[Callable[P, R]], Op[P, R]] | Op[P, R]
```

A decorator to weave op-ify a function or method. Works for both sync and async. Automatically detects iterator functions and applies appropriate behavior.

**Args:**

***

<SourceLink />

### <kbd>function</kbd> `otel_traces_endpoint`

```python theme={null}
otel_traces_endpoint(base_url: str | None = None) → str
```

Return the full OTLP HTTP endpoint URL for Weave GenAI trace ingestion.

External callers (e.g. boot-time probes that want to verify the ingest endpoint is reachable before relying on the BatchSpanProcessor to silently drop exports) should call this rather than constructing the URL by hand. The path is owned by the SDK and may move.

* <b>`func`</b>: The function to decorate.
* <b>`name`</b>: Custom name for the op. Defaults to the function name.
* <b>`call_display_name`</b>: Display name for calls, can be a string or callable.
* <b>`postprocess_inputs`</b>: Function to transform inputs before logging.
* <b>`postprocess_output`</b>: Function to transform output before logging.
* <b>`tracing_sample_rate`</b>: Fraction of calls to trace (0.0 to 1.0).
* <b>`enable_code_capture`</b>: Whether to capture source code for this op.
* <b>`accumulator`</b>: Function to accumulate results for streaming ops.
* <b>`eager_call_start`</b>: If True, call starts are sent immediately rather than batched.  Useful for long-running operations like evaluations that should  be visible in the UI immediately.
  **Args:**

***

<SourceLink />

### <kbd>function</kbd> `publish`

```python theme={null}
publish(
    obj: 'Any',
    name: 'str | None' = None,
    tags: 'list[str] | None' = None,
    aliases: 'list[str] | None' = None
) → ObjectRef
```

Save and version a Python object.

Weave creates a new version of the object if the object's name already exists and its content hash does not match the latest version of that object.

* <b>`base_url`</b>: Trace server base URL. Defaults to  `weave_trace_server_url()`.
  **Args:**

* <b>`obj`</b>: The object to save and version.

* <b>`name`</b>: The name to save the object under.

* <b>`tags`</b>: Optional list of tags to add to the published object version.

* <b>`aliases`</b>: Optional list of aliases to set on the published object version.
  **Returns:**
  A Weave Ref to the saved object.

***

<SourceLink />

### <kbd>function</kbd> `ref`

```python theme={null}
ref(location: 'str') → ObjectRef
```

Creates a Ref to an existing Weave object. This does not directly retrieve the object but allows you to pass it to other Weave API functions.

**Args:**

* <b>`location`</b>: A Weave Ref URI, or if `weave.init()` has been called, `name:version` or `name`. If no version is provided, `latest` is used.
  **Returns:**
  A Weave Ref to the object.

***

<SourceLink />

### <kbd>function</kbd> `remove_aliases`

```python theme={null}
remove_aliases(obj_ref: 'ObjectRef | str', alias: 'str | list[str]') → None
```

Remove one or more aliases from an object.

**Args:**

***

<SourceLink />

### <kbd>function</kbd> `remove_tags`

```python theme={null}
remove_tags(obj_ref: 'ObjectRef | str', tags: 'list[str]') → None
```

Remove tags from an object version.

* <b>`obj_ref`</b>: Reference to the object, either an ObjectRef or a weave /// URI string.
* <b>`alias`</b>: An alias name or list of alias names to remove.
  **Args:**

***

<SourceLink />

### <kbd>function</kbd> `require_current_call`

```python theme={null}
require_current_call() → Call
```

Get the Call object for the currently executing Op, within that Op.

This allows you to access attributes of the Call such as its id or feedback while it is running.

```python theme={null}
@weave.op
def hello(name: str) -> None:
     print(f"Hello {name}!")
     current_call = weave.require_current_call()
     print(current_call.id)
```

It is also possible to access a Call after the Op has returned.

If you have the Call's id, perhaps from the UI, you can use the `get_call` method on the `WeaveClient` returned from `weave.init` to retrieve the Call object.

```python theme={null}
client = weave.init("<project>")
mycall = client.get_call("<call_id>")
```

Alternately, after defining your Op you can use its `call` method. For example:

```python theme={null}
@weave.op
def add(a: int, b: int) -> int:
     return a + b

result, call = add.call(1, 2)
print(call.id)
```

* <b>`obj_ref`</b>: Reference to the object version, either an ObjectRef or a weave /// URI string.
* <b>`tags`</b>: List of tag strings to remove.
  **Returns:**
  The Call object for the currently executing Op

**Raises:**

* <b>`NoCurrentCallError`</b>:  If tracking has not been initialized or this method is  invoked outside an Op.

***

<SourceLink />

### <kbd>function</kbd> `set_aliases`

```python theme={null}
set_aliases(obj_ref: 'ObjectRef | str', alias: 'str | list[str]') → None
```

Set one or more aliases for an object version.

**Args:**

***

<SourceLink />

### <kbd>function</kbd> `set_view`

```python theme={null}
set_view(
    name: 'str',
    content: 'Content | str',
    extension: 'str | None' = None,
    mimetype: 'str | None' = None,
    metadata: 'dict[str, Any] | None' = None,
    encoding: 'str' = 'utf-8'
) → None
```

Attach a custom view to the current call summary at `_weave.views.<name>`.

* <b>`obj_ref`</b>: Reference to the object version, either an ObjectRef or a weave /// URI string.

* <b>`alias`</b>: An alias name or list of alias names to set (e.g., "production").
  **Args:**

* <b>`name`</b>: The view name (key under `summary._weave.views`).

* <b>`content`</b>: A `weave.Content` instance or raw string. Strings are wrapped via  `Content.from_text` using the supplied extension or mimetype.

* <b>`extension`</b>: Optional file extension to use when `content` is a string.

* <b>`mimetype`</b>: Optional MIME type to use when `content` is a string.

* <b>`metadata`</b>: Optional metadata to attach when creating `Content` from text.

* <b>`encoding`</b>: Text encoding to apply when creating `Content` from text.
  **Returns:**
  None

**Examples:**
` import weave`

> > > weave.init("proj")
> > > @weave.op
> > > ... def foo():
> > > ...     weave.set\_view("readme", "# Hello", extension="md")
> > > ...     return 1
> > > foo()

***

<SourceLink />

### <kbd>function</kbd> `start_llm`

```python theme={null}
start_llm(
    model: 'str' = '',
    provider_name: 'str' = '',
    system_instructions: 'list[str] | None' = None
) → LLM
```

Create and activate an LLM call. Uses the current turn if available.

If no turn is active, returns a disconnected LLM (no contextvar set).

Pass `provider_name` explicitly. The SDK does not infer it from the model identifier: prefix-based guessing misattributes user fine-tunes (e.g. a model named `text-...`) and bakes assumptions about future model names into telemetry that's expensive to correct after the fact.

***

<SourceLink />

### <kbd>function</kbd> `start_session`

```python theme={null}
start_session(
    agent_name: 'str' = '',
    model: 'str' = '',
    session_id: 'str' = '',
    session_name: 'str' = '',
    include_content: 'bool' = True,
    continue_parent_trace: 'bool' = False
) → Session
```

Create and activate a session. Sets the contextvar for cross-module access.

***

<SourceLink />

### <kbd>function</kbd> `start_subagent`

```python theme={null}
start_subagent(name: 'str', model: 'str' = '') → SubAgent
```

Create a sub-agent invocation span.

The SubAgent's OTel span automatically becomes a child of whatever span is current in OTel context — typically a Turn span if one is active. Mirrors `start_tool` in shape; OTel context handles parent-child propagation, no explicit delegation is needed.

***

<SourceLink />

### <kbd>function</kbd> `start_tool`

```python theme={null}
start_tool(name: 'str', arguments: 'str' = '', tool_call_id: 'str' = '') → Tool
```

Create a tool execution span.

The Tool's OTel span automatically becomes a child of whatever span is current in OTel context — typically a Turn span if one is active. No explicit turn delegation is needed: parent-child propagation happens via OTel context, not via the Session SDK contextvars.

***

<SourceLink />

### <kbd>function</kbd> `start_turn`

```python theme={null}
start_turn(
    user_message: 'str' = '',
    model: 'str' = '',
    agent_name: 'str' = ''
) → Turn
```

Create and activate a turn. Uses the current session if available.

If no session is active, returns a disconnected Turn that is NOT set in the contextvar. This means `get_current_turn()` will return None. Use `session.start_turn()` instead if you need contextvar-based cross-module access.

***

<SourceLink />

### <kbd>function</kbd> `thread`

```python theme={null}
thread(
    thread_id: 'str | object | None' = <object object at 0x7f1615fc8990>
) → Iterator[ThreadContext]
```

Context manager for setting thread\_id on calls within the context.

**Examples:**

```python theme={null}
# Auto-generate thread_id
with weave.thread() as t:
     print(f"Thread ID: {t.thread_id}")
     result = my_function("input")  # This call will have the auto-generated thread_id
     print(f"Current turn: {t.turn_id}")

# Explicit thread_id
with weave.thread("custom_thread") as t:
     result = my_function("input")  # This call will have thread_id="custom_thread"

# Disable threading
with weave.thread(None) as t:
     result = my_function("input")  # This call will have thread_id=None
```

**Args:**

* <b>`thread_id`</b>: The thread identifier to associate with calls in this context.  If not provided, a UUID v7 will be auto-generated.  If None, thread tracking will be disabled.
  **Yields:**

* <b>`ThreadContext`</b>:  An object providing access to thread\_id and current turn\_id.

***

<SourceLink />

### <kbd>function</kbd> `wandb_init_hook`

```python theme={null}
wandb_init_hook() → None
```


# feedback
Source: https://docs.wandb.ai/weave/reference/python-sdk/trace/feedback

Python SDK reference for weave.trace.feedback

# API Overview

***

<SourceLink />

## <kbd>class</kbd> `FeedbackQuery`

Lazy-loading object for fetching feedback from the server.

<SourceLink />

### <kbd>method</kbd> `__init__`

```python theme={null}
__init__(
    entity: 'str',
    project: 'str',
    query: 'Query',
    offset: 'int | None' = None,
    limit: 'int | None' = None,
    show_refs: 'bool' = False
)
```

***

<SourceLink />

### <kbd>method</kbd> `execute`

```python theme={null}
execute() → Feedbacks
```

***

<SourceLink />

### <kbd>method</kbd> `refresh`

```python theme={null}
refresh() → Feedbacks
```

***

<SourceLink />

### <kbd>method</kbd> `refs`

```python theme={null}
refs() → Refs
```

***

<SourceLink />

## <kbd>class</kbd> `Feedbacks`

A collection of Feedback objects with utilities.

<SourceLink />

### <kbd>method</kbd> `__init__`

```python theme={null}
__init__(
    show_refs: 'bool',
    feedbacks: 'Iterable[Feedback] | None' = None
) → None
```

***

<SourceLink />

### <kbd>method</kbd> `refs`

```python theme={null}
refs() → Refs
```

Return the unique refs associated with these feedbacks.

***

<SourceLink />

## <kbd>class</kbd> `RefFeedbackQuery`

Object for interacting with feedback associated with a particular ref.

<SourceLink />

### <kbd>method</kbd> `__init__`

```python theme={null}
__init__(ref: 'str') → None
```

***

<SourceLink />

### <kbd>method</kbd> `add`

```python theme={null}
add(
    feedback_type: 'str',
    payload: 'dict[str, Any] | None' = None,
    creator: 'str | None' = None,
    annotation_ref: 'str | None' = None,
    **kwargs: 'dict[str, Any]'
) → str
```

Add feedback to the ref.

feedback\_type: A string identifying the type of feedback. The "wandb." prefix is reserved. creator: The name to display for the originator of the feedback.

***

<SourceLink />

### <kbd>method</kbd> `add_note`

```python theme={null}
add_note(note: 'str', creator: 'str | None' = None) → str
```

***

<SourceLink />

### <kbd>method</kbd> `add_reaction`

```python theme={null}
add_reaction(emoji: 'str', creator: 'str | None' = None) → str
```

***

<SourceLink />

### <kbd>method</kbd> `execute`

```python theme={null}
execute() → Feedbacks
```

***

<SourceLink />

### <kbd>method</kbd> `purge`

```python theme={null}
purge(feedback_id: 'str') → None
```

***

<SourceLink />

### <kbd>method</kbd> `refresh`

```python theme={null}
refresh() → Feedbacks
```

***

<SourceLink />

### <kbd>method</kbd> `refs`

```python theme={null}
refs() → Refs
```


# op
Source: https://docs.wandb.ai/weave/reference/python-sdk/trace/op

Python SDK reference for weave.trace.op

# API Overview

***

<SourceLink />

## <kbd>class</kbd> `DisplayNameFuncError`

***

<SourceLink />

## <kbd>class</kbd> `OpCallError`

***

<SourceLink />

## <kbd>class</kbd> `OpKwargs`

TypedDict for op() keyword arguments.

***

<SourceLink />

## <kbd>class</kbd> `Sentinel`

Sentinel(package: 'str', path: 'str', name: 'str')

<SourceLink />

### <kbd>method</kbd> `__init__`

```python theme={null}
__init__(package: 'str', path: 'str', name: 'str') → None
```

***

<SourceLink />

## <kbd>class</kbd> `WeaveKwargs`

***

<SourceLink />

### <kbd>function</kbd> `as_op`

```python theme={null}
as_op(fn: 'Callable[P, R]') → Op[P, R]
```

Given a @weave.op decorated function, return its Op.

@weave.op decorated functions are instances of Op already, so this function should be a no-op at runtime. But you can use it to satisfy type checkers if you need to access OpDef attributes in a typesafe way.

**Args:**

* <b>`fn`</b>: A weave.op decorated function.
  **Returns:**
  The Op of the function.

***

<SourceLink />

### <kbd>function</kbd> `call`

```python theme={null}
call(
    op: 'Op',
    *args: 'Any',
    __weave: 'WeaveKwargs | None' = None,
    __should_raise: 'bool' = False,
    __require_explicit_finish: 'bool' = False,
    **kwargs: 'Any'
) → tuple[Any, Call] | Coroutine[Any, Any, tuple[Any, Call]]
```

Executes the op and returns both the result and a Call representing the execution.

This function will never raise.  Any errors are captured in the Call object.

This method is automatically bound to any function decorated with `@weave.op`, allowing for usage like:

```python theme={null}
@weave.op
def add(a: int, b: int) -> int:
     return a + b

result, call = add.call(1, 2)
```

***

<SourceLink />

### <kbd>function</kbd> `calls`

```python theme={null}
calls(op: 'Op') → CallsIter
```

Get an iterator over all calls to this op.

This method is automatically bound to any function decorated with `@weave.op`, allowing for usage like:

```python theme={null}
@weave.op
def add(a: int, b: int) -> int:
     return a + b

calls = add.calls()
for call in calls:
     print(call)
```

***

<SourceLink />

### <kbd>function</kbd> `get_captured_code`

```python theme={null}
get_captured_code(op: 'Op') → str
```

Get the captured code of the op.

This only works when you get an op back from a ref.  The pattern is:

ref = weave.publish(func) op = ref.get() captured\_code = op.get\_captured\_code()

***

<SourceLink />

### <kbd>function</kbd> `is_op`

```python theme={null}
is_op(obj: 'Any') → TypeIs[Op]
```

Check if an object is an Op.

***

<SourceLink />

### <kbd>function</kbd> `is_placeholder_call`

```python theme={null}
is_placeholder_call(call: 'Call') → TypeIs[NoOpCall]
```

***

<SourceLink />

### <kbd>function</kbd> `is_tracing_setting_disabled`

```python theme={null}
is_tracing_setting_disabled() → bool
```

***

<SourceLink />

### <kbd>function</kbd> `maybe_bind_method`

```python theme={null}
maybe_bind_method(func: 'Callable', self: 'Any' = None) → Callable | MethodType
```

Bind a function to any object (even if it's not a class).

If self is None, return the function as is.

***

<SourceLink />

### <kbd>function</kbd> `maybe_unbind_method`

```python theme={null}
maybe_unbind_method(oplike: 'Op | MethodType | partial') → Op
```

Unbind an Op-like method or partial to a plain Op function.

For:

* methods, remove set `self` param
* partials, remove any preset params

***

<SourceLink />

### <kbd>function</kbd> `op`

```python theme={null}
op(
    func: 'Callable[P, R] | None' = None,
    name: 'str | None' = None,
    call_display_name: 'str | CallDisplayNameFunc | None' = None,
    postprocess_inputs: 'PostprocessInputsFunc | None' = None,
    postprocess_output: 'PostprocessOutputFunc | None' = None,
    tracing_sample_rate: 'float' = 1.0,
    enable_code_capture: 'bool' = True,
    accumulator: 'Callable[[Any | None, Any], Any] | None' = None,
    kind: 'OpKind | None' = None,
    color: 'OpColor | None' = None,
    eager_call_start: 'bool' = False
) → Callable[[Callable[P, R]], Op[P, R]] | Op[P, R]
```

A decorator to weave op-ify a function or method. Works for both sync and async. Automatically detects iterator functions and applies appropriate behavior.

**Args:**

***

<SourceLink />

### <kbd>function</kbd> `placeholder_call`

```python theme={null}
placeholder_call() → Call
```

***

<SourceLink />

### <kbd>function</kbd> `setup_dunder_weave_dict`

```python theme={null}
setup_dunder_weave_dict(op: 'Op', d: 'WeaveKwargs | None' = None) → WeaveKwargs
```

Sets up a \_\_weave dict used to pass WeaveKwargs to ops.

* <b>`func`</b>: The function to decorate.

* <b>`name`</b>: Custom name for the op. Defaults to the function name.

* <b>`call_display_name`</b>: Display name for calls, can be a string or callable.

* <b>`postprocess_inputs`</b>: Function to transform inputs before logging.

* <b>`postprocess_output`</b>: Function to transform output before logging.

* <b>`tracing_sample_rate`</b>: Fraction of calls to trace (0.0 to 1.0).

* <b>`enable_code_capture`</b>: Whether to capture source code for this op.

* <b>`accumulator`</b>: Function to accumulate results for streaming ops.

* <b>`eager_call_start`</b>: If True, call starts are sent immediately rather than batched.  Useful for long-running operations like evaluations that should  be visible in the UI immediately.
  **Args:**

* <b>`d`</b>: Optional existing WeaveKwargs dict to update.

* <b>`op`</b>: Op to extract kind and color from.
  **Returns:**
  WeaveKwargs dict with attributes, display\_name, and optionally kind/color set.

***

<SourceLink />

### <kbd>function</kbd> `should_skip_tracing_for_op`

```python theme={null}
should_skip_tracing_for_op(op: 'Op') → bool
```


# util
Source: https://docs.wandb.ai/weave/reference/python-sdk/trace/util

Python SDK reference for weave.trace.util

# API Overview

***

<SourceLink />

## <kbd>class</kbd> `ContextAwareThread`

A Thread that runs functions with the context of the caller.

This is a drop-in replacement for threading.Thread that ensures calls behave as expected inside the thread.  Weave requires certain contextvars to be set (see call\_context.py), but new threads do not automatically copy context from the parent, which can cause the call context to be lost -- not good!  This class automates contextvar copying so using this thread "just works" as the user probably expects.

You can achieve the same effect without this class by instead writing:

```python theme={null}
def run_with_context(func, *args, **kwargs):
     context = copy_context()
     def wrapper():
         context.run(func, *args, **kwargs)
     return wrapper

thread = threading.Thread(target=run_with_context(your_func, *args, **kwargs))
thread.start()
```

<SourceLink />

### <kbd>method</kbd> `__init__`

```python theme={null}
__init__(*args: 'Any', **kwargs: 'Any') → None
```

***

#### <kbd>property</kbd> daemon

A boolean value indicating whether this thread is a daemon thread.

This must be set before start() is called, otherwise RuntimeError is raised. Its initial value is inherited from the creating thread; the main thread is not a daemon thread and therefore all threads created in the main thread default to daemon = False.

The entire Python program exits when only daemon threads are left.

***

#### <kbd>property</kbd> ident

Thread identifier of this thread or None if it has not been started.

This is a nonzero integer. See the get\_ident() function. Thread identifiers may be recycled when a thread exits and another thread is created. The identifier is available even after the thread has exited.

***

#### <kbd>property</kbd> name

A string used for identification purposes only.

It has no semantics. Multiple threads may be given the same name. The initial name is set by the constructor.

***

#### <kbd>property</kbd> native\_id

Native integral thread ID of this thread, or None if it has not been started.

This is a non-negative integer. See the get\_native\_id() function. This represents the Thread ID as reported by the kernel.

***

<SourceLink />

### <kbd>method</kbd> `run`

```python theme={null}
run() → None
```

***

<SourceLink />

## <kbd>class</kbd> `ContextAwareThreadPoolExecutor`

A ThreadPoolExecutor that runs functions with the context of the caller.

This is a drop-in replacement for concurrent.futures.ThreadPoolExecutor that ensures weave calls behave as expected inside the executor.  Weave requires certain contextvars to be set (see call\_context.py), but new threads do not automatically copy context from the parent, which can cause the call context to be lost -- not good!  This class automates contextvar copying so using this executor "just works" as the user probably expects.

You can achieve the same effect without this class by instead writing:

```python theme={null}
with concurrent.futures.ThreadPoolExecutor() as executor:
     contexts = [copy_context() for _ in range(len(vals))]

     def _wrapped_fn(*args):
         return contexts.pop().run(fn, *args)

     executor.map(_wrapped_fn, vals)
```

<SourceLink />

### <kbd>method</kbd> `__init__`

```python theme={null}
__init__(*args: 'Any', **kwargs: 'Any') → None
```

***

<SourceLink />

### <kbd>method</kbd> `map`

```python theme={null}
map(
    fn: 'Callable',
    *iterables: 'Iterable[Any]',
    timeout: 'float | None' = None,
    chunksize: 'int' = 1
) → Iterator
```

***

<SourceLink />

### <kbd>method</kbd> `submit`

```python theme={null}
submit(fn: 'Callable', *args: 'Any', **kwargs: 'Any') → Any
```

***

<SourceLink />

### <kbd>function</kbd> `deprecated`

```python theme={null}
deprecated(new_name: 'str') → Callable[[Callable[P, R]], Callable[P, R]]
```

Decorator to mark a function as deprecated and redirect users to `new_name`.

***

<SourceLink />

### <kbd>function</kbd> `is_colab`

```python theme={null}
is_colab()
```

***

<SourceLink />

### <kbd>function</kbd> `is_notebook`

```python theme={null}
is_notebook() → bool
```

***

<SourceLink />

### <kbd>function</kbd> `log_once`

```python theme={null}
log_once(log_method: 'Callable[[str], None]', message: 'str') → None
```

Logs a message once, suppressing subsequent messages of the same type. This is useful for notifying the user about errors without spamming the logs.

This is mostly useful for cases where the same error message might occur many times. For example, if an op fails to save, it is likely going to happen every time that op is called. Or, if we have an error in our patched iterator, then it likely happens every time we iterate over the result. This allows use to inform the user about the error without clogging up their logs.

**Args:**

* <b>`log_method`</b>: The method to use to log the message. This should accept a string argument.
* <b>`message`</b>: The message to log.
  **Example:**

```python theme={null}
log_once(logger.error, "Failed to save op")
```


# weave_client
Source: https://docs.wandb.ai/weave/reference/python-sdk/trace/weave_client

Python SDK reference for weave.trace.weave_client

# API Overview

***

<SourceLink />

## <kbd>class</kbd> `CrossProjectRefError`

Raised when client-side digest computation encounters a ref to a different project that cannot be resolved to an internal ID.

***

<SourceLink />

## <kbd>class</kbd> `FlushStatus`

Status information about the current flush operation.

***

<SourceLink />

## <kbd>class</kbd> `NoInternalProjectIDError`

Raised when client-side digest computation cannot proceed because no internal project ID has been resolved yet.

***

<SourceLink />

## <kbd>class</kbd> `PendingJobCounts`

Counts of pending jobs for each type.

***

<SourceLink />

## <kbd>class</kbd> `WeaveClient`

<SourceLink />

### <kbd>method</kbd> `__init__`

```python theme={null}
__init__(
    entity: 'str',
    project: 'str',
    server: 'TraceServerClientInterface',
    ensure_project_exists: 'bool' = True,
    postprocess_inputs: 'PostprocessInputsFunc | None' = None,
    postprocess_output: 'PostprocessOutputFunc | None' = None,
    attributes: 'dict[str, Any] | None' = None,
    api_key: 'str | None' = None
)
```

***

#### <kbd>property</kbd> num\_outstanding\_jobs

Returns the total number of pending jobs across all executors and the server.

This property can be used to check the progress of background tasks without blocking the main thread.

**Returns:**

* <b>`int`</b>:  The total number of pending jobs

***

#### <kbd>property</kbd> project\_id

***

<SourceLink />

### <kbd>method</kbd> `add_calls_to_annotation_queue`

```python theme={null}
add_calls_to_annotation_queue(
    queue_id: 'str',
    call_ids: 'list[str]',
    display_fields: 'list[str]'
) → AnnotationQueueAddCallsRes
```

Add calls to an annotation queue.

**Args:**

***

<SourceLink />

### <kbd>method</kbd> `add_cost`

```python theme={null}
add_cost(
    llm_id: 'str',
    prompt_token_cost: 'float',
    completion_token_cost: 'float',
    effective_date: 'datetime | None' = None,
    prompt_token_cost_unit: 'str | None' = 'USD',
    completion_token_cost_unit: 'str | None' = 'USD',
    provider_id: 'str | None' = 'default'
) → CostCreateRes
```

Add a cost to the current project.

* <b>`queue_id`</b>: Annotation queue ID.
* <b>`call_ids`</b>: Call IDs to add to the queue.
* <b>`display_fields`</b>: JSON paths to show reviewers, such as  `inputs.prompt` or `output.text`.
  **Examples:**

```python theme={null}
client.add_cost(llm_id="my_expensive_custom_model", prompt_token_cost=1, completion_token_cost=2)
client.add_cost(llm_id="my_expensive_custom_model", prompt_token_cost=500, completion_token_cost=1000, effective_date=datetime(1998, 10, 3))
```

**Args:**

* <b>`llm_id`</b>: The ID of the LLM. eg "gpt-4o-mini-2024-07-18"
* <b>`prompt_token_cost`</b>: The cost per prompt token. eg .0005
* <b>`completion_token_cost`</b>: The cost per completion token. eg .0015
* <b>`effective_date`</b>: Defaults to the current date. A datetime.datetime object.
* <b>`provider_id`</b>: The provider of the LLM. Defaults to "default". eg "openai"
* <b>`prompt_token_cost_unit`</b>: The unit of the cost for the prompt tokens. Defaults to "USD". (Currently unused, will be used in the future to specify the currency type for the cost eg "tokens" or "time")
* <b>`completion_token_cost_unit`</b>: The unit of the cost for the completion tokens. Defaults to "USD". (Currently unused, will be used in the future to specify the currency type for the cost eg "tokens" or "time")
  **Returns:**
  A CostCreateRes object. Which has one field called a list of tuples called ids. Each tuple contains the llm\_id and the id of the created cost object.

***

<SourceLink />

### <kbd>method</kbd> `add_tags`

```python theme={null}
add_tags(obj_ref: 'ObjectRef | str', tags: 'list[str]') → None
```

Add tags to an object version.

**Args:**

***

<SourceLink />

### <kbd>method</kbd> `clear_wandb_run_context`

```python theme={null}
clear_wandb_run_context() → None
```

Clear wandb run context override.

After calling this, calls will fall back to using the global wandb.run (if available) for run\_id and step information.

* <b>`obj_ref`</b>: Reference to the object version, either an ObjectRef or a weave /// URI string.
* <b>`tags`</b>: List of tag strings to add.
  **Examples:**

```python theme={null}
client = weave.init("my-project")
client.set_wandb_run_context(run_id="my-run-id", step=5)
# ... make some calls ...
client.clear_wandb_run_context()
# Now calls will use global wandb.run again
```

***

<SourceLink />

### <kbd>method</kbd> `create_annotation_queue`

```python theme={null}
create_annotation_queue(
    name: 'str',
    scorer_refs: 'list[str]',
    description: 'str' = ''
) → str
```

Create an annotation queue for this project.

**Args:**

* <b>`name`</b>: Display name for the queue.
* <b>`scorer_refs`</b>: Weave refs for the scorers/annotation fields reviewers complete.
* <b>`description`</b>: Optional reviewer guidelines or queue description.
  **Returns:**
  The ID of the created annotation queue.

***

<SourceLink />

### <kbd>method</kbd> `create_call`

```python theme={null}
create_call(
    op: 'str | Op',
    inputs: 'dict[str, Any]',
    parent: 'Call | None' = None,
    attributes: 'dict[str, Any] | None' = None,
    display_name: 'str | Callable[[Call], str] | None' = None,
    use_stack: 'bool' = True,
    _call_id_override: 'str | None' = None,
    started_at: 'datetime | None' = None
) → Call
```

Create, log, and push a call onto the runtime stack.

**Args:**

* <b>`op`</b>: The operation producing the call, or the name of an anonymous operation.
* <b>`inputs`</b>: The inputs to the operation.
* <b>`parent`</b>: The parent call. If parent is not provided, the current run is used as the parent.
* <b>`display_name`</b>: The display name for the call. Defaults to None.
* <b>`attributes`</b>: The attributes for the call. Defaults to None.
* <b>`use_stack`</b>: Whether to push the call onto the runtime stack. Defaults to True.
* <b>`started_at`</b>: Override the call start time. If None, uses current time.
  **Returns:**
  The created Call object.

***

<SourceLink />

### <kbd>method</kbd> `delete_all_object_versions`

```python theme={null}
delete_all_object_versions(object_name: 'str') → int
```

Delete all versions of an object.

**Args:**

* <b>`object_name`</b>: The name of the object whose versions should be deleted.
  **Returns:**
  The number of versions deleted.

***

<SourceLink />

### <kbd>method</kbd> `delete_all_op_versions`

```python theme={null}
delete_all_op_versions(op_name: 'str') → int
```

Delete all versions of an op.

**Args:**

* <b>`op_name`</b>: The name of the op whose versions should be deleted.
  **Returns:**
  The number of versions deleted.

***

<SourceLink />

### <kbd>method</kbd> `delete_annotation_queue`

```python theme={null}
delete_annotation_queue(queue_id: 'str') → AnnotationQueueSchema
```

Soft-delete an annotation queue.

***

<SourceLink />

### <kbd>method</kbd> `delete_call`

```python theme={null}
delete_call(call: 'Call') → None
```

***

<SourceLink />

### <kbd>method</kbd> `delete_calls`

```python theme={null}
delete_calls(call_ids: 'list[str]') → None
```

Delete calls by their IDs.

Deleting a call will also delete all of its children.

**Args:**

***

<SourceLink />

### <kbd>method</kbd> `delete_object_version`

```python theme={null}
delete_object_version(object: 'ObjectRef') → None
```

***

<SourceLink />

### <kbd>method</kbd> `delete_object_versions`

```python theme={null}
delete_object_versions(object_name: 'str', digests: 'list[str]') → int
```

Delete specific versions of an object.

* <b>`call_ids`</b>: A list of call IDs to delete. Ex: \["2F0193e107-8fcf-7630-b576-977cc3062e2e"]
  **Args:**

* <b>`object_name`</b>: The name of the object whose versions should be deleted.

* <b>`digests`</b>: List of digests to delete. Can include aliases like "latest" or "v0".
  **Returns:**
  The number of versions deleted.

***

<SourceLink />

### <kbd>method</kbd> `delete_op_version`

```python theme={null}
delete_op_version(op: 'OpRef') → None
```

***

<SourceLink />

### <kbd>method</kbd> `fail_call`

```python theme={null}
fail_call(call: 'Call', exception: 'BaseException') → None
```

Fail a call with an exception. This is a convenience method for finish\_call.

***

<SourceLink />

### <kbd>method</kbd> `finish`

```python theme={null}
finish(
    use_progress_bar: 'bool' = True,
    callback: 'Callable[[FlushStatus], None] | None' = None
) → None
```

Flushes all background tasks to ensure they are processed.

This method blocks until all currently enqueued jobs are processed, displaying a progress bar to show the status of the pending tasks. It ensures parallel processing during main thread execution and can improve performance when user code completes before data has been uploaded to the server.

**Args:**

***

<SourceLink />

### <kbd>method</kbd> `finish_call`

```python theme={null}
finish_call(
    call: 'Call',
    output: 'Any' = None,
    exception: 'BaseException | None' = None,
    op: 'Op | None' = None,
    ended_at: 'datetime | None' = None
) → None
```

Finalize a call and persist its results.

Any values present in `call.summary` are deep-merged with computed summary statistics (e.g. usage and status counts) before being written to the database.

***

<SourceLink />

### <kbd>method</kbd> `flush`

```python theme={null}
flush() → None
```

Flushes background asynchronous tasks, safe to call multiple times.

***

<SourceLink />

### <kbd>method</kbd> `get`

```python theme={null}
get(ref: 'ObjectRef', objectify: 'bool' = True) → Any
```

***

<SourceLink />

### <kbd>method</kbd> `get_aliases`

```python theme={null}
get_aliases(obj_ref: 'ObjectRef | str') → list[str]
```

Get aliases for an object version.

* <b>`use_progress_bar`</b>: Whether to display a progress bar during flush.  Set to False for environments where a progress bar  would not render well (e.g., CI environments).

* <b>`callback`</b>: Optional callback function that receives status updates.  Overrides use\_progress\_bar.
  **Args:**

* <b>`obj_ref`</b>: Reference to the object version, either an ObjectRef or a weave /// URI string.
  **Returns:**
  List of alias strings. Includes the virtual "latest" alias if the object version is the latest.

***

<SourceLink />

### <kbd>method</kbd> `get_annotation_queue`

```python theme={null}
get_annotation_queue(queue_id: 'str') → AnnotationQueueSchema
```

Read a single annotation queue by ID.

***

<SourceLink />

### <kbd>method</kbd> `get_annotation_queue_stats`

```python theme={null}
get_annotation_queue_stats(
    queue_ids: 'list[str]'
) → list[AnnotationQueueStatsSchema]
```

Get item completion stats for annotation queues.

***

<SourceLink />

### <kbd>method</kbd> `get_call`

```python theme={null}
get_call(
    call_id: 'str',
    include_costs: 'bool' = False,
    include_feedback: 'bool' = False,
    columns: 'list[str] | None' = None
) → WeaveObject
```

Get a single call by its ID.

**Args:**

* <b>`call_id`</b>: The ID of the call to get.
* <b>`include_costs`</b>: If true, cost info is included at summary.weave
* <b>`include_feedback`</b>: If true, feedback info is included at summary.weave.feedback
* <b>`columns`</b>: A list of columns to include in the response. If None,  all columns are included. Specifying fewer columns may be more performant. Some columns are always included  id, project\_id, trace\_id, op\_name, started\_at
  **Returns:**
  A call object.

***

<SourceLink />

### <kbd>method</kbd> `get_calls`

```python theme={null}
get_calls(
    filter: 'CallsFilterLike | None' = None,
    limit: 'int | None' = None,
    offset: 'int | None' = None,
    sort_by: 'list[SortByLike] | None' = None,
    query: 'QueryLike | None' = None,
    include_costs: 'bool' = False,
    include_feedback: 'bool' = False,
    include_storage_size: 'bool' = False,
    include_total_storage_size: 'bool' = False,
    include_usernames: 'bool' = False,
    columns: 'list[str] | None' = None,
    expand_columns: 'list[str] | None' = None,
    return_expanded_column_values: 'bool' = True,
    scored_by: 'str | list[str] | None' = None,
    page_size: 'int' = 1000
) → CallsIter
```

Retrieve a list of traced calls (operations) for this project.

This method provides a powerful and flexible interface for querying trace data. It supports pagination, filtering, sorting, field projection, and scoring metadata, and can be used to power custom trace UIs or analysis tools.

Performance Tip: Specify `columns` and use `filter` or `query` to reduce result size.

**Args:**

* <b>`filter`</b>:  High-level filter for narrowing results by fields like `op_name`, `parent_ids`, etc.
* <b>`limit`</b>:  Maximum number of calls to return.
* <b>`offset`</b>:  Number of calls to skip before returning results (used for pagination).
* <b>`sort_by`</b>:  List of fields to sort the results by (e.g., `started_at desc`).
* <b>`query`</b>:  A mongo-like expression for advanced filtering. Not all Mongo operators are supported.
* <b>`include_costs`</b>:  If True, includes token/cost info in `summary.weave`.
* <b>`include_feedback`</b>:  If True, includes feedback in `summary.weave.feedback`.
* <b>`include_storage_size`</b>:  If True, includes the storage size for a call.
* <b>`include_total_storage_size`</b>:  If True, includes the total storage size for a trace.
* <b>`include_usernames`</b>:  If True, attempts to resolve each call's `wb_user_id` to a `wb_username`.
* <b>`columns`</b>:  List of fields to return per call. Reducing this can significantly improve performance.  (Some fields like `id`, `trace_id`, `op_name`, and `started_at` are always included.)
* <b>`scored_by`</b>:  Filter by one or more scorers (name or ref URI). Multiple scorers are AND-ed.
* <b>`page_size`</b>:  Number of calls fetched per page. Tune this for performance in large queries.

**Returns:**

* <b>`CallsIter`</b>:  An iterator over `Call` objects. Supports slicing, iteration, and `.to_pandas()`.

**Example:**

```python theme={null}
calls = client.get_calls(
     filter=CallsFilter(op_names=["my_op"]),
     columns=["inputs", "output", "summary"],
     limit=100,
)
for call in calls:
     print(call.inputs, call.output)
```

***

<SourceLink />

### <kbd>method</kbd> `get_evaluation`

```python theme={null}
get_evaluation(uri: 'str') → Evaluation
```

Retrieve a specific Evaluation object by its URI.

Evaluation URIs typically follow the format: `weave:///entity/project/object/Evaluation:version`

You can also get the evaluation by its "friendly" name: get\_evaluation("Evaluation:v1")

**Args:**

* <b>`uri`</b> (str):  The unique resource identifier of the evaluation to retrieve.

**Returns:**

* <b>`Evaluation`</b>:  The Evaluation object corresponding to the provided URI.

**Raises:**

* <b>`TypeError`</b>:  If the object at the URI is not an Evaluation instance.
* <b>`ValueError`</b>:  If the URI is invalid or the object cannot be found.

**Examples:**

```python theme={null}
client = weave.init("my-project")
evaluation = client.get_evaluation("weave:///entity/project/object/my-eval:v1")
print(evaluation.name)
```

***

<SourceLink />

### <kbd>method</kbd> `get_evaluations`

```python theme={null}
get_evaluations() → list[Evaluation]
```

Retrieve all Evaluation objects from the current project.

**Returns:**

* <b>`list[Evaluation]`</b>:  A list of all Evaluation objects in the current project.  Empty list if no evaluations are found or if all conversions fail.

**Examples:**

```python theme={null}
client = weave.init("my-project")
evaluations = client.get_evaluations()
print(f"Found {len(evaluations)} evaluations")
for eval in evaluations:
     print(f"Evaluation: {eval.name}")
```

***

<SourceLink />

### <kbd>method</kbd> `get_feedback`

```python theme={null}
get_feedback(
    query: 'Query | str | None' = None,
    reaction: 'str | None' = None,
    offset: 'int' = 0,
    limit: 'int' = 100
) → FeedbackQuery
```

Query project for feedback.

**Examples:**

```python theme={null}
# Fetch a specific feedback object.
# Note that this still returns a collection, which is expected
# to contain zero or one item(s).
client.get_feedback("1B4082A3-4EDA-4BEB-BFEB-2D16ED59AA07")

# Find all feedback objects with a specific reaction.
client.get_feedback(reaction="👍", limit=10)

# Find all feedback objects with a specific feedback type with
# mongo-style query.
from weave.trace_server.interface.query import Query

query = Query(
    **{
        "$expr": {
            "$eq": [
                {"$getField": "feedback_type"},
                {"$literal": "wandb.reaction.1"},
            ],
        }
    }
)
client.get_feedback(query=query)
```

**Args:**

* <b>`query`</b>: A mongo-style query expression. For convenience, also accepts a feedback UUID string.
* <b>`reaction`</b>: For convenience, filter by a particular reaction emoji.
* <b>`offset`</b>: The offset to start fetching feedback objects from.
* <b>`limit`</b>: The maximum number of feedback objects to fetch.
  **Returns:**
  A FeedbackQuery object.

***

<SourceLink />

### <kbd>method</kbd> `get_tags`

```python theme={null}
get_tags(obj_ref: 'ObjectRef | str') → list[str]
```

Get tags for an object version.

**Args:**

* <b>`obj_ref`</b>: Reference to the object version, either an ObjectRef or a weave /// URI string.
  **Returns:**
  List of tag strings. Returns empty list if the object version has no tags.

***

<SourceLink />

### <kbd>method</kbd> `get_tags_and_aliases`

```python theme={null}
get_tags_and_aliases(obj_ref: 'ObjectRef | str') → tuple[list[str], list[str]]
```

Get both tags and aliases for an object version in a single call.

**Args:**

* <b>`obj_ref`</b>: Reference to the object version, either an ObjectRef or a weave /// URI string.
  **Returns:**
  A tuple of (tags, aliases). Each is a list of strings. Returns empty lists if the object version has no tags or aliases.

***

<SourceLink />

### <kbd>method</kbd> `link_prompt_to_registry`

```python theme={null}
link_prompt_to_registry(
    prompt: 'LinkablePrompt',
    target_path: 'str',
    aliases: 'Sequence[str] | None' = None
) → LinkAssetToRegistryRes
```

Link a published prompt version into the registry.

**Args:**

* <b>`prompt`</b>: A published prompt, an `ObjectRef`, or a fully qualified weave ///... URI string.

* <b>`target_path`</b>: Registry destination path in the format  `<registry_project>/<portfolio_name>`, for example  `wandb-registry-prompts/my-prompt-collection`.

* <b>`aliases`</b>: Optional aliases to attach to the created registry version.
  **Returns:**

* <b>`LinkAssetToRegistryRes`</b>:  Parsed response from the registry-link endpoint.

***

<SourceLink />

### <kbd>method</kbd> `list_aliases`

```python theme={null}
list_aliases() → list[str]
```

List all distinct aliases in the project.

**Returns:**
List of all alias strings in the project.

***

<SourceLink />

### <kbd>method</kbd> `list_annotation_queue_items`

```python theme={null}
list_annotation_queue_items(
    queue_id: 'str',
    filter: 'AnnotationQueueItemsFilter | None' = None,
    sort_by: 'list[SortBy] | None' = None,
    limit: 'int | None' = None,
    offset: 'int | None' = None,
    include_position: 'bool' = False
) → list[AnnotationQueueItemSchema]
```

List calls assigned to an annotation queue.

***

<SourceLink />

### <kbd>method</kbd> `list_annotation_queues`

```python theme={null}
list_annotation_queues(
    name: 'str | None' = None,
    sort_by: 'list[SortBy] | None' = None,
    limit: 'int | None' = None,
    offset: 'int | None' = None
) → list[AnnotationQueueSchema]
```

List annotation queues for this project.

***

<SourceLink />

### <kbd>method</kbd> `list_tags`

```python theme={null}
list_tags() → list[str]
```

List all distinct tags in the project.

**Returns:**
List of all tag strings in the project.

***

<SourceLink />

### <kbd>method</kbd> `purge_costs`

```python theme={null}
purge_costs(ids: 'list[str] | str') → None
```

Purge costs from the current project.

**Examples:**

```python theme={null}
client.purge_costs([ids])
client.purge_costs(ids)
```

**Args:**

***

<SourceLink />

### <kbd>method</kbd> `query_costs`

```python theme={null}
query_costs(
    query: 'Query | str | None' = None,
    llm_ids: 'list[str] | None' = None,
    offset: 'int' = 0,
    limit: 'int' = 100
) → list[CostQueryOutput]
```

Query project for costs.

* <b>`ids`</b>: The cost IDs to purge. Can be a single ID or a list of IDs.
  **Examples:**

```python theme={null}
# Fetch a specific cost object.
# Note that this still returns a collection, which is expected
# to contain zero or one item(s).
client.query_costs("1B4082A3-4EDA-4BEB-BFEB-2D16ED59AA07")

# Find all cost objects with a specific reaction.
client.query_costs(llm_ids=["gpt-4o-mini-2024-07-18"], limit=10)
```

**Args:**

* <b>`query`</b>: A mongo-style query expression. For convenience, also accepts a cost UUID string.
* <b>`llm_ids`</b>: For convenience, filter for a set of llm\_ids.
* <b>`offset`</b>: The offset to start fetching cost objects from.
* <b>`limit`</b>: The maximum number of cost objects to fetch.
  **Returns:**
  A CostQuery object.

***

<SourceLink />

### <kbd>method</kbd> `remove_aliases`

```python theme={null}
remove_aliases(obj_ref: 'ObjectRef | str', alias: 'str | list[str]') → None
```

Remove one or more aliases from an object.

**Args:**

***

<SourceLink />

### <kbd>method</kbd> `remove_tags`

```python theme={null}
remove_tags(obj_ref: 'ObjectRef | str', tags: 'list[str]') → None
```

Remove tags from an object version.

* <b>`obj_ref`</b>: Reference to the object, either an ObjectRef or a weave /// URI string (digest is not used since aliases are object-scoped).
* <b>`alias`</b>: An alias name or list of alias names to remove.
  **Args:**

***

<SourceLink />

### <kbd>method</kbd> `save`

```python theme={null}
save(val: 'Any', name: 'str', branch: 'str' = 'latest') → Any
```

Do not call directly, use weave.publish() instead.

* <b>`obj_ref`</b>: Reference to the object version, either an ObjectRef or a weave /// URI string.

* <b>`tags`</b>: List of tag strings to remove.
  **Args:**

* <b>`val`</b>: The object to save.

* <b>`name`</b>: The name to save the object under.

* <b>`branch`</b>: The branch to save the object under. Defaults to "latest".
  **Returns:**
  A deserialized version of the saved object.

***

<SourceLink />

### <kbd>method</kbd> `set_aliases`

```python theme={null}
set_aliases(obj_ref: 'ObjectRef | str', alias: 'str | list[str]') → None
```

Set one or more aliases for an object version.

**Args:**

***

<SourceLink />

### <kbd>method</kbd> `set_wandb_run_context`

```python theme={null}
set_wandb_run_context(run_id: 'str', step: 'int | None' = None) → None
```

Override wandb run\_id and step for calls created by this client.

This allows you to associate Weave calls with a specific WandB run that is not bound to the global wandb.run symbol.

* <b>`obj_ref`</b>: Reference to the object version, either an ObjectRef or a weave /// URI string.

* <b>`alias`</b>: An alias name or list of alias names to set (e.g., "production").
  **Args:**

* <b>`run_id`</b>: The run ID (not including entity/project prefix).  The client will automatically add the entity/project prefix.

* <b>`step`</b>: The step number to use for calls. If None, step will not be set.
  **Examples:**

```python theme={null}
client = weave.init("my-project")
client.set_wandb_run_context(run_id="my-run-id", step=5)
# Now all calls will be associated with entity/project/my-run-id at step 5

# Or without a step
client.set_wandb_run_context(run_id="my-run-id")
# Calls will be associated with entity/project/my-run-id with no step
```

***

<SourceLink />

### <kbd>method</kbd> `update_annotation_queue`

```python theme={null}
update_annotation_queue(
    queue_id: 'str',
    name: 'str | None' = None,
    description: 'str | None' = None,
    scorer_refs: 'list[str] | None' = None
) → AnnotationQueueSchema
```

Update annotation queue metadata.

***

<SourceLink />

### <kbd>function</kbd> `get_obj_name`

```python theme={null}
get_obj_name(val: 'Any') → str
```

***

<SourceLink />

### <kbd>function</kbd> `get_parallelism_settings`

```python theme={null}
get_parallelism_settings() → tuple[int | None, int | None]
```

***

<SourceLink />

### <kbd>function</kbd> `map_to_refs`

```python theme={null}
map_to_refs(obj: 'Any') → Any
```

***

<SourceLink />

### <kbd>function</kbd> `print_call_link`

```python theme={null}
print_call_link(call: 'Call') → None
```

***

<SourceLink />

### <kbd>function</kbd> `redact_sensitive_keys`

```python theme={null}
redact_sensitive_keys(obj: 'Any') → Any
```

***

<SourceLink />

### <kbd>function</kbd> `sanitize_object_name`

```python theme={null}
sanitize_object_name(name: 'str') → str
```


# trace_server_interface
Source: https://docs.wandb.ai/weave/reference/python-sdk/trace_server/trace_server_interface

Python SDK reference for weave.trace_server.trace_server_interface

# API Overview

***

<SourceLink />

## <kbd>class</kbd> `ActionsExecuteBatchReq`

**Pydantic Fields:**

* `project_id`: `<class 'str'>`
* `action_ref`: `<class 'str'>`
* `call_ids`: `list[str]`
* `wb_user_id`: `str | None`

***

<SourceLink />

## <kbd>class</kbd> `ActionsExecuteBatchRes`

***

<SourceLink />

## <kbd>class</kbd> `AggregationType`

Aggregation functions supported by feedback and call stats metrics.

***

<SourceLink />

## <kbd>class</kbd> `AliasesListReq`

**Pydantic Fields:**

* `project_id`: `<class 'str'>`
* `wb_user_id`: `str | None`

***

<SourceLink />

## <kbd>class</kbd> `AliasesListRes`

**Pydantic Fields:**

* `aliases`: `list[str]`

***

<SourceLink />

## <kbd>class</kbd> `AnnotationQueueAddCallsReq`

Request to add calls to an annotation queue in batch.

Extends AnnotationQueueAddCallsBody by adding queue\_id for internal API usage.

**Pydantic Fields:**

* `project_id`: `<class 'str'>`
* `call_ids`: `list[str]`
* `display_fields`: `list[str]`
* `queue_id`: `<class 'str'>`
* `wb_user_id`: `str | None`

***

<SourceLink />

## <kbd>class</kbd> `AnnotationQueueAddCallsRes`

Response from adding calls to a queue.

**Pydantic Fields:**

* `added_count`: `<class 'int'>`
* `duplicates`: `<class 'int'>`

***

<SourceLink />

## <kbd>class</kbd> `AnnotationQueueCreateReq`

Request to create a new annotation queue.

**Pydantic Fields:**

* `project_id`: `<class 'str'>`
* `name`: `<class 'str'>`
* `description`: `<class 'str'>`
* `scorer_refs`: `list[str]`
* `wb_user_id`: `str | None`

***

<SourceLink />

## <kbd>class</kbd> `AnnotationQueueCreateRes`

Response from creating an annotation queue.

**Pydantic Fields:**

* `id`: `<class 'str'>`

***

<SourceLink />

## <kbd>class</kbd> `AnnotationQueueDeleteReq`

Request to delete (soft-delete) an annotation queue.

**Pydantic Fields:**

* `project_id`: `<class 'str'>`
* `queue_id`: `<class 'str'>`
* `wb_user_id`: `str | None`

***

<SourceLink />

## <kbd>class</kbd> `AnnotationQueueDeleteRes`

Response from deleting an annotation queue.

**Pydantic Fields:**

* `queue`: `<class 'AnnotationQueueSchema'>`

***

<SourceLink />

## <kbd>class</kbd> `AnnotationQueueItemSchema`

Schema for annotation queue item responses.

**Pydantic Fields:**

* `id`: `<class 'str'>`
* `project_id`: `<class 'str'>`
* `queue_id`: `<class 'str'>`
* `call_id`: `<class 'str'>`
* `call_started_at`: `<class 'datetime.datetime'>`
* `call_ended_at`: `datetime.datetime | None`
* `call_op_name`: `<class 'str'>`
* `call_trace_id`: `<class 'str'>`
* `display_fields`: `list[str]`
* `added_by`: `str | None`
* `annotation_state`: `typing.Literal['unstarted', 'in_progress', 'completed', 'skipped']`
* `annotator_user_id`: `str | None`
* `created_at`: `<class 'datetime.datetime'>`
* `created_by`: `<class 'str'>`
* `updated_at`: `<class 'datetime.datetime'>`
* `deleted_at`: `datetime.datetime | None`
* `position_in_queue`: `int | None`

***

<SourceLink />

## <kbd>class</kbd> `AnnotationQueueItemsQueryReq`

Request to query items in an annotation queue.

Extends AnnotationQueueItemsQueryBody by adding queue\_id for internal API usage.

**Pydantic Fields:**

* `project_id`: `<class 'str'>`
* `filter`: `weave.trace_server.common_interface.AnnotationQueueItemsFilter | None`
* `sort_by`: `list[weave.trace_server.common_interface.SortBy] | None`
* `limit`: `int | None`
* `offset`: `int | None`
* `include_position`: `<class 'bool'>`
* `queue_id`: `<class 'str'>`

***

<SourceLink />

## <kbd>class</kbd> `AnnotationQueueItemsQueryRes`

Response from querying annotation queue items.

**Pydantic Fields:**

* `items`: `list[AnnotationQueueItemSchema]`

***

<SourceLink />

## <kbd>class</kbd> `AnnotationQueueReadReq`

Request to read a specific annotation queue.

**Pydantic Fields:**

* `project_id`: `<class 'str'>`
* `queue_id`: `<class 'str'>`

***

<SourceLink />

## <kbd>class</kbd> `AnnotationQueueReadRes`

Response from reading an annotation queue.

**Pydantic Fields:**

* `queue`: `<class 'AnnotationQueueSchema'>`

***

<SourceLink />

## <kbd>class</kbd> `AnnotationQueueSchema`

Schema for annotation queue responses.

**Pydantic Fields:**

* `id`: `<class 'str'>`
* `project_id`: `<class 'str'>`
* `name`: `<class 'str'>`
* `description`: `<class 'str'>`
* `scorer_refs`: `list[str]`
* `created_at`: `<class 'datetime.datetime'>`
* `created_by`: `<class 'str'>`
* `updated_at`: `<class 'datetime.datetime'>`
* `deleted_at`: `datetime.datetime | None`

***

<SourceLink />

## <kbd>class</kbd> `AnnotationQueueStatsSchema`

Statistics for a single annotation queue.

**Pydantic Fields:**

* `queue_id`: `<class 'str'>`
* `total_items`: `<class 'int'>`
* `completed_items`: `<class 'int'>`

***

<SourceLink />

## <kbd>class</kbd> `AnnotationQueueUpdateReq`

Request to update an annotation queue.

All fields except project\_id and queue\_id are optional - only provided fields will be updated.

**Pydantic Fields:**

* `project_id`: `<class 'str'>`
* `queue_id`: `<class 'str'>`
* `name`: `str | None`
* `description`: `str | None`
* `scorer_refs`: `list[str] | None`
* `wb_user_id`: `str | None`

***

<SourceLink />

## <kbd>class</kbd> `AnnotationQueueUpdateRes`

Response from updating an annotation queue.

**Pydantic Fields:**

* `queue`: `<class 'AnnotationQueueSchema'>`

***

<SourceLink />

## <kbd>class</kbd> `AnnotationQueuesQueryReq`

Request to query annotation queues for a project.

**Pydantic Fields:**

* `project_id`: `<class 'str'>`
* `name`: `str | None`
* `sort_by`: `list[weave.trace_server.common_interface.SortBy] | None`
* `limit`: `int | None`
* `offset`: `int | None`

***

<SourceLink />

## <kbd>class</kbd> `AnnotationQueuesQueryRes`

Response from querying annotation queues.

**Pydantic Fields:**

* `queues`: `list[AnnotationQueueSchema]`

***

<SourceLink />

## <kbd>class</kbd> `AnnotationQueuesStatsReq`

Request to get stats for multiple annotation queues.

**Pydantic Fields:**

* `project_id`: `<class 'str'>`
* `queue_ids`: `list[str]`

***

<SourceLink />

## <kbd>class</kbd> `AnnotationQueuesStatsRes`

Response with stats for multiple annotation queues.

**Pydantic Fields:**

* `stats`: `list[AnnotationQueueStatsSchema]`

***

<SourceLink />

## <kbd>class</kbd> `AnnotatorQueueItemsProgressUpdateReq`

Request to update the annotation state of a queue item for the current annotator.

Valid state transitions:

* (absence) -> 'in\_progress': Mark item as in progress (only when no record exists)
* (absence) -> 'completed' or 'skipped': Directly complete/skip item
* 'in\_progress' or 'unstarted' -> 'completed' or 'skipped': Complete/skip started item
* same\_state -> same\_state: Idempotent no-op (returns existing item unchanged)

**Pydantic Fields:**

* `project_id`: `<class 'str'>`
* `queue_id`: `<class 'str'>`
* `item_id`: `<class 'str'>`
* `annotation_state`: `<class 'str'>`
* `wb_user_id`: `str | None`

***

<SourceLink />

## <kbd>class</kbd> `AnnotatorQueueItemsProgressUpdateRes`

Response from updating annotation state.

**Pydantic Fields:**

* `item`: `<class 'AnnotationQueueItemSchema'>`

***

<SourceLink />

## <kbd>class</kbd> `CallBatchEndMode`

**Pydantic Fields:**

* `mode`: `<class 'str'>`
* `req`: `<class 'CallEndReq'>`

***

<SourceLink />

## <kbd>class</kbd> `CallBatchStartMode`

**Pydantic Fields:**

* `mode`: `<class 'str'>`
* `req`: `<class 'CallStartReq'>`

***

<SourceLink />

## <kbd>class</kbd> `CallCreateBatchReq`

**Pydantic Fields:**

* `batch`: `list[CallBatchStartMode | CallBatchEndMode]`

***

<SourceLink />

## <kbd>class</kbd> `CallCreateBatchRes`

**Pydantic Fields:**

* `res`: `list[CallStartRes | CallEndRes]`

***

<SourceLink />

## <kbd>class</kbd> `CallEndReq`

**Pydantic Fields:**

* `end`: `<class 'EndedCallSchemaForInsert'>`

***

<SourceLink />

## <kbd>class</kbd> `CallEndRes`

***

<SourceLink />

## <kbd>class</kbd> `CallEndV2Req`

Request for ending a single call via v2 API.

**Pydantic Fields:**

* `end`: `<class 'EndedCallSchemaForInsertWithStartedAt'>`

***

<SourceLink />

## <kbd>class</kbd> `CallEndV2Res`

Response for ending a single call via v2 API.

***

<SourceLink />

## <kbd>class</kbd> `CallMetricSpec`

Specification for a call-level metric to aggregate (not grouped by model).

**Pydantic Fields:**

* `metric`: `typing.Literal['latency_ms', 'call_count', 'error_count']`
* `aggregations`: `list[AggregationType]`
* `percentiles`: `list[float]`

***

<SourceLink />

## <kbd>class</kbd> `CallReadReq`

**Pydantic Fields:**

* `project_id`: `<class 'str'>`
* `id`: `<class 'str'>`
* `include_costs`: `bool | None`
* `include_storage_size`: `bool | None`
* `include_total_storage_size`: `bool | None`

***

<SourceLink />

## <kbd>class</kbd> `CallReadRes`

**Pydantic Fields:**

* `call`: `CallSchema | None`

***

<SourceLink />

## <kbd>class</kbd> `CallSchema`

**Pydantic Fields:**

* `id`: `<class 'str'>`
* `project_id`: `<class 'str'>`
* `op_name`: `<class 'str'>`
* `display_name`: `str | None`
* `trace_id`: `<class 'str'>`
* `parent_id`: `str | None`
* `thread_id`: `str | None`
* `turn_id`: `str | None`
* `started_at`: `<class 'datetime.datetime'>`
* `attributes`: `dict[str, typing.Any]`
* `inputs`: `dict[str, typing.Any]`
* `ended_at`: `datetime.datetime | None`
* `exception`: `str | None`
* `output`: `typing.Any | None`
* `summary`: `SummaryMap | None`
* `wb_user_id`: `str | None`
* `wb_username`: `str | None`
* `wb_run_id`: `str | None`
* `wb_run_step`: `int | None`
* `wb_run_step_end`: `int | None`
* `deleted_at`: `datetime.datetime | None`
* `expire_at`: `datetime.datetime | None`
* `storage_size_bytes`: `int | None`
* `total_storage_size_bytes`: `int | None`

<SourceLink />

### <kbd>method</kbd> `serialize_typed_dicts`

```python theme={null}
serialize_typed_dicts(v: dict[str, Any]) → dict[str, Any]
```

***

<SourceLink />

## <kbd>class</kbd> `CallStartReq`

**Pydantic Fields:**

* `start`: `<class 'StartedCallSchemaForInsert'>`

***

<SourceLink />

## <kbd>class</kbd> `CallStartRes`

**Pydantic Fields:**

* `id`: `<class 'str'>`
* `trace_id`: `<class 'str'>`

***

<SourceLink />

## <kbd>class</kbd> `CallStartV2Req`

Request for starting a single call via v2 API.

**Pydantic Fields:**

* `start`: `<class 'StartedCallSchemaForInsert'>`

***

<SourceLink />

## <kbd>class</kbd> `CallStartV2Res`

Response for starting a single call via v2 API.

**Pydantic Fields:**

* `id`: `<class 'str'>`
* `trace_id`: `<class 'str'>`

***

<SourceLink />

## <kbd>class</kbd> `CallStatsReq`

Request for aggregated call statistics over a time range.

**Pydantic Fields:**

* `project_id`: `<class 'str'>`
* `start`: `<class 'datetime.datetime'>`
* `end`: `datetime.datetime | None`
* `granularity`: `int | None`
* `usage_metrics`: `list[UsageMetricSpec] | None`
* `call_metrics`: `list[CallMetricSpec] | None`
* `filter`: `CallsFilter | None`
* `timezone`: `<class 'str'>`

<SourceLink />

### <kbd>method</kbd> `validate_date_range`

```python theme={null}
validate_date_range() → CallStatsReq
```

Ensure call stats requests are bounded to a safe date range.

***

<SourceLink />

## <kbd>class</kbd> `CallStatsRes`

Response containing time-series call statistics.

**Pydantic Fields:**

* `start`: `<class 'datetime.datetime'>`
* `end`: `<class 'datetime.datetime'>`
* `granularity`: `<class 'int'>`
* `timezone`: `<class 'str'>`
* `usage_buckets`: `list[dict[str, typing.Any]]`
* `call_buckets`: `list[dict[str, typing.Any]]`

***

<SourceLink />

## <kbd>class</kbd> `CallUpdateReq`

**Pydantic Fields:**

* `project_id`: `<class 'str'>`
* `call_id`: `<class 'str'>`
* `display_name`: `str | None`
* `wb_user_id`: `str | None`

***

<SourceLink />

## <kbd>class</kbd> `CallUpdateRes`

***

<SourceLink />

## <kbd>class</kbd> `CallsDeleteReq`

**Pydantic Fields:**

* `project_id`: `<class 'str'>`
* `call_ids`: `list[str]`
* `wb_user_id`: `str | None`

***

<SourceLink />

## <kbd>class</kbd> `CallsDeleteRes`

**Pydantic Fields:**

* `num_deleted`: `<class 'int'>`

***

<SourceLink />

## <kbd>class</kbd> `CallsFilter`

**Pydantic Fields:**

* `op_names`: `list[str] | None`
* `input_refs`: `list[str] | None`
* `output_refs`: `list[str] | None`
* `parent_ids`: `list[str] | None`
* `trace_ids`: `list[str] | None`
* `call_ids`: `list[str] | None`
* `thread_ids`: `list[str] | None`
* `turn_ids`: `list[str] | None`
* `trace_roots_only`: `bool | None`
* `wb_user_ids`: `list[str] | None`
* `wb_run_ids`: `list[str] | None`

***

<SourceLink />

## <kbd>class</kbd> `CallsQueryReq`

**Pydantic Fields:**

* `project_id`: `<class 'str'>`
* `filter`: `CallsFilter | None`
* `limit`: `int | None`
* `offset`: `int | None`
* `sort_by`: `list[weave.trace_server.common_interface.SortBy] | None`
* `query`: `weave.trace_server.interface.query.Query | None`
* `include_costs`: `bool | None`
* `include_feedback`: `bool | None`
* `include_storage_size`: `bool | None`
* `include_total_storage_size`: `bool | None`
* `include_usernames`: `bool | None`
* `columns`: `list[str] | None`
* `expand_columns`: `list[str] | None`
* `return_expanded_column_values`: `bool | None`

***

<SourceLink />

## <kbd>class</kbd> `CallsQueryRes`

**Pydantic Fields:**

* `calls`: `list[CallSchema]`

***

<SourceLink />

## <kbd>class</kbd> `CallsQueryStatsReq`

**Pydantic Fields:**

* `project_id`: `<class 'str'>`
* `filter`: `CallsFilter | None`
* `query`: `weave.trace_server.interface.query.Query | None`
* `limit`: `int | None`
* `include_total_storage_size`: `bool | None`
* `expand_columns`: `list[str] | None`

***

<SourceLink />

## <kbd>class</kbd> `CallsQueryStatsRes`

**Pydantic Fields:**

* `count`: `<class 'int'>`
* `has_more`: `<class 'bool'>`
* `total_storage_size_bytes`: `int | None`

***

<SourceLink />

## <kbd>class</kbd> `CallsScoreReq`

Request to enqueue scoring jobs for a list of calls.

Scoring is performed asynchronously by the call\_scoring\_worker, which consumes messages from Kafka and applies each scorer\_ref to each call\_id.

**Pydantic Fields:**

* `project_id`: `<class 'str'>`
* `call_ids`: `list[str]`
* `scorer_refs`: `list[str]`
* `wb_user_id`: `str | None`

***

<SourceLink />

## <kbd>class</kbd> `CallsScoreRes`

Empty response for calls\_score.

Defined as a model (rather than returning None) to follow the convention used throughout this interface and to allow fields to be added later without a breaking change.

***

<SourceLink />

## <kbd>class</kbd> `CallsUpsertCompleteReq`

Request for upserting a batch of completed calls.

**Pydantic Fields:**

* `batch`: `list[CompletedCallSchemaForInsert]`

***

<SourceLink />

## <kbd>class</kbd> `CallsUpsertCompleteRes`

Response for upserting a batch of completed calls.

***

<SourceLink />

## <kbd>class</kbd> `CallsUsageReq`

Request to compute aggregated usage for multiple root calls.

This endpoint returns usage metrics for each requested root call, where each root's metrics include the sum of its own usage plus all descendants' usage.

Note: All matching calls are loaded into memory for aggregation. For very large result sets (>10k calls), consider batching root call IDs or using narrower filters at the application layer.

**Pydantic Fields:**

* `project_id`: `<class 'str'>`
* `call_ids`: `list[str]`
* `include_costs`: `<class 'bool'>`
* `limit`: `<class 'int'>`

***

<SourceLink />

## <kbd>class</kbd> `CallsUsageRes`

Response with aggregated usage metrics per root call.

**Pydantic Fields:**

* `call_usage`: `dict[str, dict[str, LLMAggregatedUsage]]`
* `unfinished_call_ids`: `list[str]`

***

<SourceLink />

## <kbd>class</kbd> `CompletedCallSchemaForInsert`

Schema for inserting a completed call directly.

This represents a call that is already finished at insertion time, with both start and end information provided together. Used by the calls\_complete endpoint.

**Pydantic Fields:**

* `project_id`: `<class 'str'>`
* `id`: `<class 'str'>`
* `trace_id`: `<class 'str'>`
* `op_name`: `<class 'str'>`
* `started_at`: `<class 'datetime.datetime'>`
* `ended_at`: `<class 'datetime.datetime'>`
* `display_name`: `str | None`
* `parent_id`: `str | None`
* `thread_id`: `str | None`
* `turn_id`: `str | None`
* `attributes`: `dict[str, typing.Any]`
* `inputs`: `dict[str, typing.Any]`
* `output`: `typing.Any | None`
* `summary`: `<class 'SummaryInsertMap'>`
* `otel_dump`: `dict[str, typing.Any] | None`
* `exception`: `str | None`
* `wb_user_id`: `str | None`
* `wb_run_id`: `str | None`
* `wb_run_step`: `int | None`
* `wb_run_step_end`: `int | None`

<SourceLink />

### <kbd>method</kbd> `serialize_typed_dicts`

```python theme={null}
serialize_typed_dicts(v: dict[str, Any]) → dict[str, Any]
```

***

<SourceLink />

## <kbd>class</kbd> `CompletionsCreateReq`

**Pydantic Fields:**

* `project_id`: `<class 'str'>`
* `inputs`: `<class 'CompletionsCreateRequestInputs'>`
* `wb_user_id`: `str | None`
* `track_llm_call`: `bool | None`
* `trace_id`: `str | None`
* `parent_id`: `str | None`

***

<SourceLink />

## <kbd>class</kbd> `CompletionsCreateRequestInputs`

**Pydantic Fields:**

* `model`: `<class 'str'>`
* `messages`: `<class 'list'>`
* `timeout`: `float | str | None`
* `temperature`: `float | None`
* `top_p`: `float | None`
* `n`: `int | None`
* `stop`: `str | list | None`
* `max_completion_tokens`: `int | None`
* `max_tokens`: `int | None`
* `modalities`: `list | None`
* `presence_penalty`: `float | None`
* `frequency_penalty`: `float | None`
* `stream`: `bool | None`
* `logit_bias`: `dict | None`
* `user`: `str | None`
* `response_format`: `dict | type[pydantic.main.BaseModel] | None`
* `seed`: `int | None`
* `tools`: `list | None`
* `tool_choice`: `str | dict | None`
* `logprobs`: `bool | None`
* `top_logprobs`: `int | None`
* `parallel_tool_calls`: `bool | None`
* `extra_headers`: `dict | None`
* `functions`: `list | None`
* `function_call`: `str | None`
* `api_version`: `str | None`
* `prompt`: `str | None`
* `template_vars`: `dict[str, typing.Any] | None`
* `vertex_credentials`: `str | None`

***

<SourceLink />

## <kbd>class</kbd> `CompletionsCreateRes`

**Pydantic Fields:**

* `response`: `dict[str, typing.Any]`
* `weave_call_id`: `str | None`

***

<SourceLink />

## <kbd>class</kbd> `CostCreateInput`

**Pydantic Fields:**

* `prompt_token_cost`: `<class 'float'>`
* `completion_token_cost`: `<class 'float'>`
* `cache_read_input_token_cost`: `<class 'float'>`
* `cache_creation_input_token_cost`: `<class 'float'>`
* `prompt_token_cost_unit`: `str | None`
* `completion_token_cost_unit`: `str | None`
* `effective_date`: `datetime.datetime | None`
* `provider_id`: `str | None`

***

<SourceLink />

## <kbd>class</kbd> `CostCreateReq`

**Pydantic Fields:**

* `project_id`: `<class 'str'>`
* `costs`: `dict[str, CostCreateInput]`
* `wb_user_id`: `str | None`

***

<SourceLink />

## <kbd>class</kbd> `CostCreateRes`

**Pydantic Fields:**

* `ids`: `list[tuple[str, str]]`

***

<SourceLink />

## <kbd>class</kbd> `CostPurgeReq`

**Pydantic Fields:**

* `project_id`: `<class 'str'>`
* `query`: `<class 'weave.trace_server.interface.query.Query'>`

***

<SourceLink />

## <kbd>class</kbd> `CostPurgeRes`

***

<SourceLink />

## <kbd>class</kbd> `CostQueryOutput`

**Pydantic Fields:**

* `id`: `str | None`
* `llm_id`: `str | None`
* `prompt_token_cost`: `float | None`
* `completion_token_cost`: `float | None`
* `prompt_token_cost_unit`: `str | None`
* `completion_token_cost_unit`: `str | None`
* `effective_date`: `datetime.datetime | None`
* `provider_id`: `str | None`

***

<SourceLink />

## <kbd>class</kbd> `CostQueryReq`

**Pydantic Fields:**

* `project_id`: `<class 'str'>`
* `fields`: `list[str] | None`
* `query`: `weave.trace_server.interface.query.Query | None`
* `sort_by`: `list[weave.trace_server.common_interface.SortBy] | None`
* `limit`: `int | None`
* `offset`: `int | None`

***

<SourceLink />

## <kbd>class</kbd> `CostQueryRes`

**Pydantic Fields:**

* `results`: `list[CostQueryOutput]`

***

<SourceLink />

## <kbd>class</kbd> `DatasetCreateBody`

**Pydantic Fields:**

* `name`: `str | None`
* `description`: `str | None`
* `rows`: `list[dict[str, typing.Any]]`

***

<SourceLink />

## <kbd>class</kbd> `DatasetCreateReq`

**Pydantic Fields:**

* `name`: `str | None`
* `description`: `str | None`
* `rows`: `list[dict[str, typing.Any]]`
* `project_id`: `<class 'str'>`
* `wb_user_id`: `str | None`

***

<SourceLink />

## <kbd>class</kbd> `DatasetCreateRes`

**Pydantic Fields:**

* `digest`: `<class 'str'>`
* `object_id`: `<class 'str'>`
* `version_index`: `<class 'int'>`

***

<SourceLink />

## <kbd>class</kbd> `DatasetDeleteReq`

**Pydantic Fields:**

* `project_id`: `<class 'str'>`
* `object_id`: `<class 'str'>`
* `digests`: `list[str] | None`
* `wb_user_id`: `str | None`

***

<SourceLink />

## <kbd>class</kbd> `DatasetDeleteRes`

**Pydantic Fields:**

* `num_deleted`: `<class 'int'>`

***

<SourceLink />

## <kbd>class</kbd> `DatasetListReq`

**Pydantic Fields:**

* `project_id`: `<class 'str'>`
* `limit`: `int | None`
* `offset`: `int | None`
* `wb_user_id`: `str | None`

***

<SourceLink />

## <kbd>class</kbd> `DatasetReadReq`

**Pydantic Fields:**

* `project_id`: `<class 'str'>`
* `object_id`: `<class 'str'>`
* `digest`: `<class 'str'>`
* `wb_user_id`: `str | None`

***

<SourceLink />

## <kbd>class</kbd> `DatasetReadRes`

**Pydantic Fields:**

* `object_id`: `<class 'str'>`
* `digest`: `<class 'str'>`
* `version_index`: `<class 'int'>`
* `created_at`: `<class 'datetime.datetime'>`
* `name`: `<class 'str'>`
* `description`: `str | None`
* `rows`: `<class 'str'>`

***

<SourceLink />

## <kbd>class</kbd> `EndedCallSchemaForInsert`

**Pydantic Fields:**

* `project_id`: `<class 'str'>`
* `id`: `<class 'str'>`
* `ended_at`: `<class 'datetime.datetime'>`
* `started_at`: `datetime.datetime | None`
* `exception`: `str | None`
* `output`: `typing.Any | None`
* `summary`: `<class 'SummaryInsertMap'>`
* `wb_run_step_end`: `int | None`

<SourceLink />

### <kbd>method</kbd> `serialize_typed_dicts`

```python theme={null}
serialize_typed_dicts(v: dict[str, Any]) → dict[str, Any]
```

***

<SourceLink />

## <kbd>class</kbd> `EndedCallSchemaForInsertWithStartedAt`

Deprecated alias. `started_at` now lives on the parent `EndedCallSchemaForInsert`; prefer that. Kept so external SDK pins on the `WithStartedAt` name keep importing. Remove once all in-tree callers migrate.

**Pydantic Fields:**

* `project_id`: `<class 'str'>`
* `id`: `<class 'str'>`
* `ended_at`: `<class 'datetime.datetime'>`
* `started_at`: `datetime.datetime | None`
* `exception`: `str | None`
* `output`: `typing.Any | None`
* `summary`: `<class 'SummaryInsertMap'>`
* `wb_run_step_end`: `int | None`

<SourceLink />

### <kbd>method</kbd> `serialize_typed_dicts`

```python theme={null}
serialize_typed_dicts(v: dict[str, Any]) → dict[str, Any]
```

***

<SourceLink />

## <kbd>class</kbd> `EvalResultsEvaluationSummary`

**Pydantic Fields:**

* `evaluation_call_id`: `<class 'str'>`
* `trial_count`: `<class 'int'>`
* `scorer_stats`: `list[EvalResultsScorerStats]`
* `evaluation_ref`: `str | None`
* `model_ref`: `str | None`
* `display_name`: `str | None`
* `trace_id`: `str | None`
* `started_at`: `str | None`

***

<SourceLink />

## <kbd>class</kbd> `EvalResultsFilter`

A filter scoped to an optional evaluation.

**Pydantic Fields:**

* `evaluation_call_id`: `str | None`
* `query`: `<class 'weave.trace_server.interface.query.Query'>`

***

<SourceLink />

## <kbd>class</kbd> `EvalResultsQueryBody`

**Pydantic Fields:**

* `evaluation_call_ids`: `list[str] | None`
* `evaluation_run_ids`: `list[str] | None`
* `require_intersection`: `<class 'bool'>`
* `include_raw_data_rows`: `<class 'bool'>`
* `resolve_row_refs`: `<class 'bool'>`
* `include_rows`: `<class 'bool'>`
* `include_summary`: `<class 'bool'>`
* `summary_require_intersection`: `bool | None`
* `include_predict_and_score_children`: `<class 'bool'>`
* `sort_by`: `list[EvalResultsSortBy] | None`
* `filters`: `list[EvalResultsFilter] | None`
* `limit`: `int | None`
* `offset`: `<class 'int'>`

<SourceLink />

### <kbd>method</kbd> `validate_identifiers`

```python theme={null}
validate_identifiers() → EvalResultsQueryBody
```

Validate that at least one evaluation identifier is provided.

***

<SourceLink />

## <kbd>class</kbd> `EvalResultsQueryReq`

**Pydantic Fields:**

* `evaluation_call_ids`: `list[str] | None`
* `evaluation_run_ids`: `list[str] | None`
* `require_intersection`: `<class 'bool'>`
* `include_raw_data_rows`: `<class 'bool'>`
* `resolve_row_refs`: `<class 'bool'>`
* `include_rows`: `<class 'bool'>`
* `include_summary`: `<class 'bool'>`
* `summary_require_intersection`: `bool | None`
* `include_predict_and_score_children`: `<class 'bool'>`
* `sort_by`: `list[EvalResultsSortBy] | None`
* `filters`: `list[EvalResultsFilter] | None`
* `limit`: `int | None`
* `offset`: `<class 'int'>`
* `project_id`: `<class 'str'>`

<SourceLink />

### <kbd>method</kbd> `validate_identifiers`

```python theme={null}
validate_identifiers() → EvalResultsQueryBody
```

Validate that at least one evaluation identifier is provided.

***

<SourceLink />

## <kbd>class</kbd> `EvalResultsQueryRes`

**Pydantic Fields:**

* `rows`: `list[EvalResultsRow]`
* `total_rows`: `<class 'int'>`
* `summary`: `ForwardRef('EvalResultsSummaryRes | None')`
* `warnings`: `list[str]`

***

<SourceLink />

## <kbd>class</kbd> `EvalResultsRow`

**Pydantic Fields:**

* `row_digest`: `<class 'str'>`
* `raw_data_row`: `typing.Any | None`
* `evaluations`: `list[EvalResultsRowEvaluation]`

***

<SourceLink />

## <kbd>class</kbd> `EvalResultsRowEvaluation`

**Pydantic Fields:**

* `evaluation_call_id`: `<class 'str'>`
* `trials`: `list[EvalResultsTrial]`

***

<SourceLink />

## <kbd>class</kbd> `EvalResultsScorerStats`

Stats for a single flattened score dimension (scorer\_key or scorer\_key.path.to.leaf).

**Pydantic Fields:**

* `scorer_key`: `<class 'str'>`
* `path`: `str | None`
* `value_type`: `typing.Optional[typing.Literal['binary', 'continuous', 'text']]`
* `trial_count`: `<class 'int'>`
* `numeric_count`: `<class 'int'>`
* `numeric_mean`: `float | None`
* `pass_true_count`: `<class 'int'>`
* `pass_known_count`: `<class 'int'>`
* `pass_rate`: `float | None`
* `pass_signal_coverage`: `float | None`

***

<SourceLink />

## <kbd>class</kbd> `EvalResultsSortBy`

Sort specification for evaluation results, extending SortBy

**Pydantic Fields:**

* `field`: `<class 'str'>`
* `direction`: `typing.Literal['asc', 'desc']`
* `evaluation_call_id`: `str | None`
* `mode`: `typing.Literal['value', 'difference']`

***

<SourceLink />

## <kbd>class</kbd> `EvalResultsSummaryRes`

**Pydantic Fields:**

* `row_count`: `<class 'int'>`
* `evaluations`: `list[EvalResultsEvaluationSummary]`

***

<SourceLink />

## <kbd>class</kbd> `EvalResultsTrial`

**Pydantic Fields:**

* `predict_and_score_call_id`: `<class 'str'>`
* `predict_call_id`: `str | None`
* `model_output`: `typing.Any | None`
* `scores`: `dict[str, typing.Any]`
* `model_latency_seconds`: `float | None`
* `total_tokens`: `int | None`
* `scorer_call_ids`: `dict[str, str]`
* `genai_span_ref`: `list[GenAISpanRef] | None`

***

<SourceLink />

## <kbd>class</kbd> `EvaluateModelArgs`

Arguments for a full evaluate-model job (loads model + runs predictions + scores).

Moved from workers/evaluate\_model\_worker/evaluate\_model\_worker.py so both job types (EvaluateModelArgs and RescoringArgs) can be co-located in the same module for the EvalWorkerJob discriminated union.

**Pydantic Fields:**

* `job_type`: `typing.Literal['evaluate_model']`
* `project_id`: `<class 'str'>`
* `evaluation_ref`: `<class 'str'>`
* `model_ref`: `<class 'str'>`
* `wb_user_id`: `<class 'str'>`
* `evaluation_call_id`: `<class 'str'>`

***

<SourceLink />

## <kbd>class</kbd> `EvaluateModelReq`

**Pydantic Fields:**

* `project_id`: `<class 'str'>`
* `evaluation_ref`: `<class 'str'>`
* `model_ref`: `<class 'str'>`
* `wb_user_id`: `str | None`

***

<SourceLink />

## <kbd>class</kbd> `EvaluateModelRes`

**Pydantic Fields:**

* `call_id`: `<class 'str'>`

***

<SourceLink />

## <kbd>class</kbd> `EvaluationCreateBody`

**Pydantic Fields:**

* `name`: `<class 'str'>`
* `description`: `str | None`
* `dataset`: `<class 'str'>`
* `scorers`: `list[str] | None`
* `trials`: `<class 'int'>`
* `evaluation_name`: `str | None`
* `eval_attributes`: `dict[str, typing.Any] | None`

***

<SourceLink />

## <kbd>class</kbd> `EvaluationCreateReq`

**Pydantic Fields:**

* `name`: `<class 'str'>`
* `description`: `str | None`
* `dataset`: `<class 'str'>`
* `scorers`: `list[str] | None`
* `trials`: `<class 'int'>`
* `evaluation_name`: `str | None`
* `eval_attributes`: `dict[str, typing.Any] | None`
* `project_id`: `<class 'str'>`
* `wb_user_id`: `str | None`

***

<SourceLink />

## <kbd>class</kbd> `EvaluationCreateRes`

**Pydantic Fields:**

* `digest`: `<class 'str'>`
* `object_id`: `<class 'str'>`
* `version_index`: `<class 'int'>`
* `evaluation_ref`: `<class 'str'>`

***

<SourceLink />

## <kbd>class</kbd> `EvaluationDeleteReq`

**Pydantic Fields:**

* `project_id`: `<class 'str'>`
* `object_id`: `<class 'str'>`
* `digests`: `list[str] | None`
* `wb_user_id`: `str | None`

***

<SourceLink />

## <kbd>class</kbd> `EvaluationDeleteRes`

**Pydantic Fields:**

* `num_deleted`: `<class 'int'>`

***

<SourceLink />

## <kbd>class</kbd> `EvaluationListReq`

**Pydantic Fields:**

* `project_id`: `<class 'str'>`
* `limit`: `int | None`
* `offset`: `int | None`
* `wb_user_id`: `str | None`

***

<SourceLink />

## <kbd>class</kbd> `EvaluationReadReq`

**Pydantic Fields:**

* `project_id`: `<class 'str'>`
* `object_id`: `<class 'str'>`
* `digest`: `<class 'str'>`
* `wb_user_id`: `str | None`

***

<SourceLink />

## <kbd>class</kbd> `EvaluationReadRes`

**Pydantic Fields:**

* `object_id`: `<class 'str'>`
* `digest`: `<class 'str'>`
* `version_index`: `<class 'int'>`
* `created_at`: `<class 'datetime.datetime'>`
* `name`: `<class 'str'>`
* `description`: `str | None`
* `dataset`: `<class 'str'>`
* `scorers`: `list[str]`
* `trials`: `<class 'int'>`
* `evaluation_name`: `str | None`
* `evaluate_op`: `str | None`
* `predict_and_score_op`: `str | None`
* `summarize_op`: `str | None`

***

<SourceLink />

## <kbd>class</kbd> `EvaluationRunCreateBody`

**Pydantic Fields:**

* `evaluation`: `<class 'str'>`
* `model`: `<class 'str'>`
* `source_evaluation_run_id`: `str | None`

***

<SourceLink />

## <kbd>class</kbd> `EvaluationRunCreateReq`

**Pydantic Fields:**

* `evaluation`: `<class 'str'>`
* `model`: `<class 'str'>`
* `source_evaluation_run_id`: `str | None`
* `project_id`: `<class 'str'>`
* `wb_user_id`: `str | None`

***

<SourceLink />

## <kbd>class</kbd> `EvaluationRunCreateRes`

**Pydantic Fields:**

* `evaluation_run_id`: `<class 'str'>`

***

<SourceLink />

## <kbd>class</kbd> `EvaluationRunDeleteReq`

**Pydantic Fields:**

* `project_id`: `<class 'str'>`
* `evaluation_run_ids`: `list[str]`
* `wb_user_id`: `str | None`

***

<SourceLink />

## <kbd>class</kbd> `EvaluationRunDeleteRes`

**Pydantic Fields:**

* `num_deleted`: `<class 'int'>`

***

<SourceLink />

## <kbd>class</kbd> `EvaluationRunFilter`

**Pydantic Fields:**

* `evaluations`: `list[str] | None`
* `models`: `list[str] | None`
* `evaluation_run_ids`: `list[str] | None`

***

<SourceLink />

## <kbd>class</kbd> `EvaluationRunFinishBody`

Request body for finishing an evaluation run via REST API.

This model excludes project\_id and evaluation\_run\_id since they come from the URL path in RESTful endpoints.

**Pydantic Fields:**

* `summary`: `dict[str, typing.Any] | None`

***

<SourceLink />

## <kbd>class</kbd> `EvaluationRunFinishReq`

**Pydantic Fields:**

* `summary`: `dict[str, typing.Any] | None`
* `project_id`: `<class 'str'>`
* `evaluation_run_id`: `<class 'str'>`
* `wb_user_id`: `str | None`

***

<SourceLink />

## <kbd>class</kbd> `EvaluationRunFinishRes`

**Pydantic Fields:**

* `success`: `<class 'bool'>`

***

<SourceLink />

## <kbd>class</kbd> `EvaluationRunListReq`

**Pydantic Fields:**

* `project_id`: `<class 'str'>`
* `filter`: `EvaluationRunFilter | None`
* `limit`: `int | None`
* `offset`: `int | None`

***

<SourceLink />

## <kbd>class</kbd> `EvaluationRunReadReq`

**Pydantic Fields:**

* `project_id`: `<class 'str'>`
* `evaluation_run_id`: `<class 'str'>`

***

<SourceLink />

## <kbd>class</kbd> `EvaluationRunReadRes`

**Pydantic Fields:**

* `evaluation_run_id`: `<class 'str'>`
* `evaluation`: `<class 'str'>`
* `model`: `<class 'str'>`
* `status`: `str | None`
* `started_at`: `datetime.datetime | None`
* `finished_at`: `datetime.datetime | None`
* `summary`: `dict[str, typing.Any] | None`
* `source_evaluation_run_id`: `str | None`

***

<SourceLink />

## <kbd>class</kbd> `EvaluationStatusComplete`

**Pydantic Fields:**

* `code`: `typing.Literal['complete']`
* `output`: `dict[str, typing.Any]`

***

<SourceLink />

## <kbd>class</kbd> `EvaluationStatusFailed`

**Pydantic Fields:**

* `code`: `typing.Literal['failed']`
* `error`: `str | None`

***

<SourceLink />

## <kbd>class</kbd> `EvaluationStatusNotFound`

**Pydantic Fields:**

* `code`: `typing.Literal['not_found']`

***

<SourceLink />

## <kbd>class</kbd> `EvaluationStatusReq`

**Pydantic Fields:**

* `project_id`: `<class 'str'>`
* `call_id`: `<class 'str'>`

***

<SourceLink />

## <kbd>class</kbd> `EvaluationStatusRes`

**Pydantic Fields:**

* `status`: `EvaluationStatusNotFound | EvaluationStatusRunning | EvaluationStatusFailed | EvaluationStatusComplete`

***

<SourceLink />

## <kbd>class</kbd> `EvaluationStatusRunning`

**Pydantic Fields:**

* `code`: `typing.Literal['running']`
* `completed_rows`: `<class 'int'>`
* `total_rows`: `<class 'int'>`

***

<SourceLink />

## <kbd>class</kbd> `ExportTracePartialSuccess`

**Pydantic Fields:**

* `rejected_spans`: `<class 'int'>`
* `error_message`: `<class 'str'>`

***

<SourceLink />

## <kbd>class</kbd> `ExtraKeysTypedDict`

***

<SourceLink />

## <kbd>class</kbd> `Feedback`

**Pydantic Fields:**

* `id`: `<class 'str'>`
* `project_id`: `<class 'str'>`
* `weave_ref`: `<class 'str'>`
* `creator`: `str | None`
* `feedback_type`: `<class 'str'>`
* `payload`: `dict[str, typing.Any]`
* `annotation_ref`: `str | None`
* `runnable_ref`: `str | None`
* `call_ref`: `str | None`
* `trigger_ref`: `str | None`
* `queue_id`: `str | None`
* `scorer_tags`: `list[str]`
* `scorer_tag_reasons`: `dict[str, str]`
* `scorer_tag_confidences`: `dict[str, float]`
* `scorer_ratings`: `dict[str, float]`
* `scorer_rating_reasons`: `dict[str, str]`
* `scorer_rating_confidences`: `dict[str, float]`
* `wb_user_id`: `str | None`
* `created_at`: `<class 'datetime.datetime'>`

***

<SourceLink />

## <kbd>class</kbd> `FeedbackCreateBatchReq`

**Pydantic Fields:**

* `batch`: `list[FeedbackCreateReq]`

***

<SourceLink />

## <kbd>class</kbd> `FeedbackCreateBatchRes`

**Pydantic Fields:**

* `res`: `list[FeedbackCreateRes]`

***

<SourceLink />

## <kbd>class</kbd> `FeedbackCreateReq`

**Pydantic Fields:**

* `id`: `str | None`
* `project_id`: `<class 'str'>`
* `weave_ref`: `<class 'str'>`
* `creator`: `str | None`
* `feedback_type`: `<class 'str'>`
* `payload`: `dict[str, typing.Any]`
* `annotation_ref`: `str | None`
* `runnable_ref`: `str | None`
* `call_ref`: `str | None`
* `trigger_ref`: `str | None`
* `queue_id`: `str | None`
* `scorer_tags`: `list[str]`
* `scorer_tag_reasons`: `dict[str, str]`
* `scorer_tag_confidences`: `dict[str, float]`
* `scorer_ratings`: `dict[str, float]`
* `scorer_rating_reasons`: `dict[str, str]`
* `scorer_rating_confidences`: `dict[str, float]`
* `wb_user_id`: `str | None`

***

<SourceLink />

## <kbd>class</kbd> `FeedbackCreateRes`

**Pydantic Fields:**

* `id`: `<class 'str'>`
* `created_at`: `<class 'datetime.datetime'>`
* `wb_user_id`: `<class 'str'>`
* `payload`: `dict[str, typing.Any]`

***

<SourceLink />

## <kbd>class</kbd> `FeedbackDict`

***

<SourceLink />

## <kbd>class</kbd> `FeedbackMetricSpec`

Specification for a feedback payload metric to aggregate.

**Pydantic Fields:**

* `json_path`: `<class 'str'>`
* `value_type`: `typing.Literal['numeric', 'boolean', 'categorical']`
* `aggregations`: `list[AggregationType]`
* `percentiles`: `list[float]`

***

<SourceLink />

## <kbd>class</kbd> `FeedbackPayloadPath`

Discovered path in feedback payload with inferred type.

**Pydantic Fields:**

* `json_path`: `<class 'str'>`
* `value_type`: `typing.Literal['numeric', 'boolean', 'categorical']`

***

<SourceLink />

## <kbd>class</kbd> `FeedbackPayloadSchemaReq`

Request for feedback payload schema discovery.

**Pydantic Fields:**

* `project_id`: `<class 'str'>`
* `start`: `<class 'datetime.datetime'>`
* `end`: `datetime.datetime | None`
* `feedback_type`: `str | None`
* `trigger_ref`: `str | None`
* `sample_limit`: `<class 'int'>`

<SourceLink />

### <kbd>method</kbd> `validate_date_range`

```python theme={null}
validate_date_range() → _FeedbackFilterBase
```

Ensure feedback requests are bounded to a safe date range.

***

<SourceLink />

## <kbd>class</kbd> `FeedbackPayloadSchemaRes`

Response with discovered feedback payload paths and types.

**Pydantic Fields:**

* `paths`: `list[FeedbackPayloadPath]`

***

<SourceLink />

## <kbd>class</kbd> `FeedbackPurgeReq`

**Pydantic Fields:**

* `project_id`: `<class 'str'>`
* `query`: `<class 'weave.trace_server.interface.query.Query'>`

***

<SourceLink />

## <kbd>class</kbd> `FeedbackPurgeRes`

***

<SourceLink />

## <kbd>class</kbd> `FeedbackQueryReq`

**Pydantic Fields:**

* `project_id`: `<class 'str'>`
* `fields`: `list[str] | None`
* `query`: `weave.trace_server.interface.query.Query | None`
* `sort_by`: `list[weave.trace_server.common_interface.SortBy] | None`
* `limit`: `int | None`
* `offset`: `int | None`

***

<SourceLink />

## <kbd>class</kbd> `FeedbackQueryRes`

**Pydantic Fields:**

* `result`: `list[dict[str, typing.Any]]`

***

<SourceLink />

## <kbd>class</kbd> `FeedbackReplaceReq`

**Pydantic Fields:**

* `id`: `str | None`
* `project_id`: `<class 'str'>`
* `weave_ref`: `<class 'str'>`
* `creator`: `str | None`
* `feedback_type`: `<class 'str'>`
* `payload`: `dict[str, typing.Any]`
* `annotation_ref`: `str | None`
* `runnable_ref`: `str | None`
* `call_ref`: `str | None`
* `trigger_ref`: `str | None`
* `queue_id`: `str | None`
* `scorer_tags`: `list[str]`
* `scorer_tag_reasons`: `dict[str, str]`
* `scorer_tag_confidences`: `dict[str, float]`
* `scorer_ratings`: `dict[str, float]`
* `scorer_rating_reasons`: `dict[str, str]`
* `scorer_rating_confidences`: `dict[str, float]`
* `wb_user_id`: `str | None`
* `feedback_id`: `<class 'str'>`

***

<SourceLink />

## <kbd>class</kbd> `FeedbackReplaceRes`

**Pydantic Fields:**

* `id`: `<class 'str'>`
* `created_at`: `<class 'datetime.datetime'>`
* `wb_user_id`: `<class 'str'>`
* `payload`: `dict[str, typing.Any]`

***

<SourceLink />

## <kbd>class</kbd> `FeedbackStatsReq`

Request for aggregated feedback statistics over time buckets.

**Pydantic Fields:**

* `project_id`: `<class 'str'>`
* `start`: `<class 'datetime.datetime'>`
* `end`: `datetime.datetime | None`
* `feedback_type`: `str | None`
* `trigger_ref`: `str | None`
* `granularity`: `int | None`
* `timezone`: `<class 'str'>`
* `metrics`: `list[FeedbackMetricSpec]`

<SourceLink />

### <kbd>method</kbd> `validate_date_range`

```python theme={null}
validate_date_range() → _FeedbackFilterBase
```

Ensure feedback requests are bounded to a safe date range.

***

<SourceLink />

## <kbd>class</kbd> `FeedbackStatsRes`

Response with time-series feedback statistics.

**Pydantic Fields:**

* `start`: `<class 'datetime.datetime'>`
* `end`: `<class 'datetime.datetime'>`
* `granularity`: `<class 'int'>`
* `timezone`: `<class 'str'>`
* `buckets`: `list[dict[str, typing.Any]]`
* `window_stats`: `dict[str, dict[str, float | None]] | None`

***

<SourceLink />

## <kbd>class</kbd> `FileContentReadReq`

**Pydantic Fields:**

* `project_id`: `<class 'str'>`
* `digest`: `<class 'str'>`

***

<SourceLink />

## <kbd>class</kbd> `FileContentReadRes`

**Pydantic Fields:**

* `content`: `<class 'bytes'>`

***

<SourceLink />

## <kbd>class</kbd> `FileCreateReq`

**Pydantic Fields:**

* `project_id`: `<class 'str'>`
* `name`: `<class 'str'>`
* `content`: `<class 'bytes'>`
* `expected_digest`: `str | None`

***

<SourceLink />

## <kbd>class</kbd> `FileCreateRes`

**Pydantic Fields:**

* `digest`: `<class 'str'>`

***

<SourceLink />

## <kbd>class</kbd> `FilesStatsReq`

**Pydantic Fields:**

* `project_id`: `<class 'str'>`

***

<SourceLink />

## <kbd>class</kbd> `FilesStatsRes`

**Pydantic Fields:**

* `total_size_bytes`: `<class 'int'>`

***

<SourceLink />

## <kbd>class</kbd> `FullTraceServerInterface`

Complete trace server interface supporting both V1 and Object APIs.

This protocol represents a trace server implementation that supports the full set of APIs - both legacy V1 endpoints and modern Object endpoints. Use this type for implementations that need to support both API versions.

***

<SourceLink />

### <kbd>method</kbd> `actions_execute_batch`

```python theme={null}
actions_execute_batch(req: ActionsExecuteBatchReq) → ActionsExecuteBatchRes
```

***

<SourceLink />

### <kbd>method</kbd> `agent_agents_query`

```python theme={null}
agent_agents_query(req: AgentsQueryReq) → AgentsQueryRes
```

***

<SourceLink />

### <kbd>method</kbd> `agent_conversation_chat`

```python theme={null}
agent_conversation_chat(
    req: AgentConversationChatReq
) → AgentConversationChatRes
```

***

<SourceLink />

### <kbd>method</kbd> `agent_custom_attrs_schema`

```python theme={null}
agent_custom_attrs_schema(
    req: AgentCustomAttrsSchemaReq
) → AgentCustomAttrsSchemaRes
```

***

<SourceLink />

### <kbd>method</kbd> `agent_search`

```python theme={null}
agent_search(req: AgentSearchReq) → AgentSearchRes
```

***

<SourceLink />

### <kbd>method</kbd> `agent_spans_query`

```python theme={null}
agent_spans_query(req: AgentSpansQueryReq) → AgentSpansQueryRes
```

***

<SourceLink />

### <kbd>method</kbd> `agent_spans_stats`

```python theme={null}
agent_spans_stats(req: AgentSpanStatsReq) → AgentSpanStatsRes
```

***

<SourceLink />

### <kbd>method</kbd> `agent_traces_chat`

```python theme={null}
agent_traces_chat(req: AgentTraceChatReq) → AgentTraceChatRes
```

***

<SourceLink />

### <kbd>method</kbd> `agent_versions_query`

```python theme={null}
agent_versions_query(req: AgentVersionsQueryReq) → AgentVersionsQueryRes
```

***

<SourceLink />

### <kbd>method</kbd> `aliases_list`

```python theme={null}
aliases_list(req: AliasesListReq) → AliasesListRes
```

***

<SourceLink />

### <kbd>method</kbd> `annotation_queue_add_calls`

```python theme={null}
annotation_queue_add_calls(
    req: AnnotationQueueAddCallsReq
) → AnnotationQueueAddCallsRes
```

***

<SourceLink />

### <kbd>method</kbd> `annotation_queue_create`

```python theme={null}
annotation_queue_create(
    req: AnnotationQueueCreateReq
) → AnnotationQueueCreateRes
```

***

<SourceLink />

### <kbd>method</kbd> `annotation_queue_delete`

```python theme={null}
annotation_queue_delete(
    req: AnnotationQueueDeleteReq
) → AnnotationQueueDeleteRes
```

***

<SourceLink />

### <kbd>method</kbd> `annotation_queue_items_query`

```python theme={null}
annotation_queue_items_query(
    req: AnnotationQueueItemsQueryReq
) → AnnotationQueueItemsQueryRes
```

***

<SourceLink />

### <kbd>method</kbd> `annotation_queue_read`

```python theme={null}
annotation_queue_read(req: AnnotationQueueReadReq) → AnnotationQueueReadRes
```

***

<SourceLink />

### <kbd>method</kbd> `annotation_queue_update`

```python theme={null}
annotation_queue_update(
    req: AnnotationQueueUpdateReq
) → AnnotationQueueUpdateRes
```

***

<SourceLink />

### <kbd>method</kbd> `annotation_queues_query_stream`

```python theme={null}
annotation_queues_query_stream(
    req: AnnotationQueuesQueryReq
) → Iterator[AnnotationQueueSchema]
```

***

<SourceLink />

### <kbd>method</kbd> `annotation_queues_stats`

```python theme={null}
annotation_queues_stats(
    req: AnnotationQueuesStatsReq
) → AnnotationQueuesStatsRes
```

***

<SourceLink />

### <kbd>method</kbd> `annotator_queue_items_progress_update`

```python theme={null}
annotator_queue_items_progress_update(
    req: AnnotatorQueueItemsProgressUpdateReq
) → AnnotatorQueueItemsProgressUpdateRes
```

***

<SourceLink />

### <kbd>method</kbd> `call_end`

```python theme={null}
call_end(req: CallEndReq) → CallEndRes
```

***

<SourceLink />

### <kbd>method</kbd> `call_end_v2`

```python theme={null}
call_end_v2(req: CallEndV2Req) → CallEndV2Res
```

***

<SourceLink />

### <kbd>method</kbd> `call_read`

```python theme={null}
call_read(req: CallReadReq) → CallReadRes
```

***

<SourceLink />

### <kbd>method</kbd> `call_start`

```python theme={null}
call_start(req: CallStartReq) → CallStartRes
```

***

<SourceLink />

### <kbd>method</kbd> `call_start_batch`

```python theme={null}
call_start_batch(req: CallCreateBatchReq) → CallCreateBatchRes
```

***

<SourceLink />

### <kbd>method</kbd> `call_start_v2`

```python theme={null}
call_start_v2(req: CallStartV2Req) → CallStartV2Res
```

***

<SourceLink />

### <kbd>method</kbd> `call_stats`

```python theme={null}
call_stats(req: 'CallStatsReq') → CallStatsRes
```

***

<SourceLink />

### <kbd>method</kbd> `call_update`

```python theme={null}
call_update(req: CallUpdateReq) → CallUpdateRes
```

***

<SourceLink />

### <kbd>method</kbd> `calls_complete`

```python theme={null}
calls_complete(req: CallsUpsertCompleteReq) → CallsUpsertCompleteRes
```

***

<SourceLink />

### <kbd>method</kbd> `calls_delete`

```python theme={null}
calls_delete(req: CallsDeleteReq) → CallsDeleteRes
```

***

<SourceLink />

### <kbd>method</kbd> `calls_query`

```python theme={null}
calls_query(req: CallsQueryReq) → CallsQueryRes
```

***

<SourceLink />

### <kbd>method</kbd> `calls_query_stats`

```python theme={null}
calls_query_stats(req: CallsQueryStatsReq) → CallsQueryStatsRes
```

***

<SourceLink />

### <kbd>method</kbd> `calls_query_stream`

```python theme={null}
calls_query_stream(req: CallsQueryReq) → Iterator[CallSchema]
```

***

<SourceLink />

### <kbd>method</kbd> `calls_score`

```python theme={null}
calls_score(req: CallsScoreReq) → CallsScoreRes
```

***

<SourceLink />

### <kbd>method</kbd> `calls_usage`

```python theme={null}
calls_usage(req: 'CallsUsageReq') → CallsUsageRes
```

***

<SourceLink />

### <kbd>method</kbd> `completions_create`

```python theme={null}
completions_create(req: CompletionsCreateReq) → CompletionsCreateRes
```

***

<SourceLink />

### <kbd>method</kbd> `completions_create_stream`

```python theme={null}
completions_create_stream(req: CompletionsCreateReq) → Iterator[dict[str, Any]]
```

***

<SourceLink />

### <kbd>method</kbd> `cost_create`

```python theme={null}
cost_create(req: CostCreateReq) → CostCreateRes
```

***

<SourceLink />

### <kbd>method</kbd> `cost_purge`

```python theme={null}
cost_purge(req: CostPurgeReq) → CostPurgeRes
```

***

<SourceLink />

### <kbd>method</kbd> `cost_query`

```python theme={null}
cost_query(req: CostQueryReq) → CostQueryRes
```

***

<SourceLink />

### <kbd>method</kbd> `dataset_create`

```python theme={null}
dataset_create(req: DatasetCreateReq) → DatasetCreateRes
```

***

<SourceLink />

### <kbd>method</kbd> `dataset_delete`

```python theme={null}
dataset_delete(req: DatasetDeleteReq) → DatasetDeleteRes
```

***

<SourceLink />

### <kbd>method</kbd> `dataset_list`

```python theme={null}
dataset_list(req: DatasetListReq) → Iterator[DatasetReadRes]
```

***

<SourceLink />

### <kbd>method</kbd> `dataset_read`

```python theme={null}
dataset_read(req: DatasetReadReq) → DatasetReadRes
```

***

<SourceLink />

### <kbd>method</kbd> `eval_results_query`

```python theme={null}
eval_results_query(req: EvalResultsQueryReq) → EvalResultsQueryRes
```

***

<SourceLink />

### <kbd>method</kbd> `evaluate_model`

```python theme={null}
evaluate_model(req: EvaluateModelReq) → EvaluateModelRes
```

***

<SourceLink />

### <kbd>method</kbd> `evaluation_create`

```python theme={null}
evaluation_create(req: EvaluationCreateReq) → EvaluationCreateRes
```

***

<SourceLink />

### <kbd>method</kbd> `evaluation_delete`

```python theme={null}
evaluation_delete(req: EvaluationDeleteReq) → EvaluationDeleteRes
```

***

<SourceLink />

### <kbd>method</kbd> `evaluation_list`

```python theme={null}
evaluation_list(req: EvaluationListReq) → Iterator[EvaluationReadRes]
```

***

<SourceLink />

### <kbd>method</kbd> `evaluation_read`

```python theme={null}
evaluation_read(req: EvaluationReadReq) → EvaluationReadRes
```

***

<SourceLink />

### <kbd>method</kbd> `evaluation_run_create`

```python theme={null}
evaluation_run_create(req: EvaluationRunCreateReq) → EvaluationRunCreateRes
```

***

<SourceLink />

### <kbd>method</kbd> `evaluation_run_delete`

```python theme={null}
evaluation_run_delete(req: EvaluationRunDeleteReq) → EvaluationRunDeleteRes
```

***

<SourceLink />

### <kbd>method</kbd> `evaluation_run_finish`

```python theme={null}
evaluation_run_finish(req: EvaluationRunFinishReq) → EvaluationRunFinishRes
```

***

<SourceLink />

### <kbd>method</kbd> `evaluation_run_list`

```python theme={null}
evaluation_run_list(req: EvaluationRunListReq) → Iterator[EvaluationRunReadRes]
```

***

<SourceLink />

### <kbd>method</kbd> `evaluation_run_read`

```python theme={null}
evaluation_run_read(req: EvaluationRunReadReq) → EvaluationRunReadRes
```

***

<SourceLink />

### <kbd>method</kbd> `evaluation_status`

```python theme={null}
evaluation_status(req: EvaluationStatusReq) → EvaluationStatusRes
```

***

<SourceLink />

### <kbd>method</kbd> `feedback_create`

```python theme={null}
feedback_create(req: FeedbackCreateReq) → FeedbackCreateRes
```

***

<SourceLink />

### <kbd>method</kbd> `feedback_create_batch`

```python theme={null}
feedback_create_batch(req: FeedbackCreateBatchReq) → FeedbackCreateBatchRes
```

***

<SourceLink />

### <kbd>method</kbd> `feedback_payload_schema`

```python theme={null}
feedback_payload_schema(
    req: FeedbackPayloadSchemaReq
) → FeedbackPayloadSchemaRes
```

***

<SourceLink />

### <kbd>method</kbd> `feedback_purge`

```python theme={null}
feedback_purge(req: FeedbackPurgeReq) → FeedbackPurgeRes
```

***

<SourceLink />

### <kbd>method</kbd> `feedback_query`

```python theme={null}
feedback_query(req: FeedbackQueryReq) → FeedbackQueryRes
```

***

<SourceLink />

### <kbd>method</kbd> `feedback_replace`

```python theme={null}
feedback_replace(req: FeedbackReplaceReq) → FeedbackReplaceRes
```

***

<SourceLink />

### <kbd>method</kbd> `feedback_stats`

```python theme={null}
feedback_stats(req: FeedbackStatsReq) → FeedbackStatsRes
```

***

<SourceLink />

### <kbd>method</kbd> `file_content_read`

```python theme={null}
file_content_read(req: FileContentReadReq) → FileContentReadRes
```

***

<SourceLink />

### <kbd>method</kbd> `file_create`

```python theme={null}
file_create(req: FileCreateReq) → FileCreateRes
```

***

<SourceLink />

### <kbd>method</kbd> `files_stats`

```python theme={null}
files_stats(req: FilesStatsReq) → FilesStatsRes
```

***

<SourceLink />

### <kbd>method</kbd> `genai_otel_export`

```python theme={null}
genai_otel_export(req: GenAIOTelExportReq) → GenAIOTelExportRes
```

***

<SourceLink />

### <kbd>method</kbd> `image_create`

```python theme={null}
image_create(req: ImageGenerationCreateReq) → ImageGenerationCreateRes
```

***

<SourceLink />

### <kbd>method</kbd> `model_create`

```python theme={null}
model_create(req: ModelCreateReq) → ModelCreateRes
```

***

<SourceLink />

### <kbd>method</kbd> `model_delete`

```python theme={null}
model_delete(req: ModelDeleteReq) → ModelDeleteRes
```

***

<SourceLink />

### <kbd>method</kbd> `model_list`

```python theme={null}
model_list(req: ModelListReq) → Iterator[ModelReadRes]
```

***

<SourceLink />

### <kbd>method</kbd> `model_read`

```python theme={null}
model_read(req: ModelReadReq) → ModelReadRes
```

***

<SourceLink />

### <kbd>method</kbd> `obj_add_tags`

```python theme={null}
obj_add_tags(req: ObjAddTagsReq) → ObjAddTagsRes
```

***

<SourceLink />

### <kbd>method</kbd> `obj_create`

```python theme={null}
obj_create(req: ObjCreateReq) → ObjCreateRes
```

***

<SourceLink />

### <kbd>method</kbd> `obj_delete`

```python theme={null}
obj_delete(req: ObjDeleteReq) → ObjDeleteRes
```

***

<SourceLink />

### <kbd>method</kbd> `obj_read`

```python theme={null}
obj_read(req: ObjReadReq) → ObjReadRes
```

***

<SourceLink />

### <kbd>method</kbd> `obj_remove_aliases`

```python theme={null}
obj_remove_aliases(req: ObjRemoveAliasesReq) → ObjRemoveAliasesRes
```

***

<SourceLink />

### <kbd>method</kbd> `obj_remove_tags`

```python theme={null}
obj_remove_tags(req: ObjRemoveTagsReq) → ObjRemoveTagsRes
```

***

<SourceLink />

### <kbd>method</kbd> `obj_set_aliases`

```python theme={null}
obj_set_aliases(req: ObjSetAliasesReq) → ObjSetAliasesRes
```

***

<SourceLink />

### <kbd>method</kbd> `objs_query`

```python theme={null}
objs_query(req: ObjQueryReq) → ObjQueryRes
```

***

<SourceLink />

### <kbd>method</kbd> `op_create`

```python theme={null}
op_create(req: OpCreateReq) → OpCreateRes
```

***

<SourceLink />

### <kbd>method</kbd> `op_delete`

```python theme={null}
op_delete(req: OpDeleteReq) → OpDeleteRes
```

***

<SourceLink />

### <kbd>method</kbd> `op_list`

```python theme={null}
op_list(req: OpListReq) → Iterator[OpReadRes]
```

***

<SourceLink />

### <kbd>method</kbd> `op_read`

```python theme={null}
op_read(req: OpReadReq) → OpReadRes
```

***

<SourceLink />

### <kbd>method</kbd> `otel_export`

```python theme={null}
otel_export(req: OTelExportReq) → OTelExportRes
```

***

<SourceLink />

### <kbd>method</kbd> `prediction_create`

```python theme={null}
prediction_create(req: PredictionCreateReq) → PredictionCreateRes
```

***

<SourceLink />

### <kbd>method</kbd> `prediction_delete`

```python theme={null}
prediction_delete(req: PredictionDeleteReq) → PredictionDeleteRes
```

***

<SourceLink />

### <kbd>method</kbd> `prediction_finish`

```python theme={null}
prediction_finish(req: PredictionFinishReq) → PredictionFinishRes
```

***

<SourceLink />

### <kbd>method</kbd> `prediction_list`

```python theme={null}
prediction_list(req: PredictionListReq) → Iterator[PredictionReadRes]
```

***

<SourceLink />

### <kbd>method</kbd> `prediction_read`

```python theme={null}
prediction_read(req: PredictionReadReq) → PredictionReadRes
```

***

<SourceLink />

### <kbd>method</kbd> `project_stats`

```python theme={null}
project_stats(req: ProjectStatsReq) → ProjectStatsRes
```

***

<SourceLink />

### <kbd>method</kbd> `project_ttl_settings_read`

```python theme={null}
project_ttl_settings_read(
    req: ProjectTTLSettingsReadReq
) → ProjectTTLSettingsReadRes
```

***

<SourceLink />

### <kbd>method</kbd> `project_ttl_settings_update`

```python theme={null}
project_ttl_settings_update(
    req: ProjectTTLSettingsUpdateReq
) → ProjectTTLSettingsUpdateRes
```

***

<SourceLink />

### <kbd>method</kbd> `refs_read_batch`

```python theme={null}
refs_read_batch(req: RefsReadBatchReq) → RefsReadBatchRes
```

***

<SourceLink />

### <kbd>method</kbd> `rescore`

```python theme={null}
rescore(req: RescoreReq) → RescoreRes
```

***

<SourceLink />

### <kbd>method</kbd> `score_create`

```python theme={null}
score_create(req: ScoreCreateReq) → ScoreCreateRes
```

***

<SourceLink />

### <kbd>method</kbd> `score_delete`

```python theme={null}
score_delete(req: ScoreDeleteReq) → ScoreDeleteRes
```

***

<SourceLink />

### <kbd>method</kbd> `score_list`

```python theme={null}
score_list(req: ScoreListReq) → Iterator[ScoreReadRes]
```

***

<SourceLink />

### <kbd>method</kbd> `score_read`

```python theme={null}
score_read(req: ScoreReadReq) → ScoreReadRes
```

***

<SourceLink />

### <kbd>method</kbd> `scorer_create`

```python theme={null}
scorer_create(req: ScorerCreateReq) → ScorerCreateRes
```

***

<SourceLink />

### <kbd>method</kbd> `scorer_delete`

```python theme={null}
scorer_delete(req: ScorerDeleteReq) → ScorerDeleteRes
```

***

<SourceLink />

### <kbd>method</kbd> `scorer_list`

```python theme={null}
scorer_list(req: ScorerListReq) → Iterator[ScorerReadRes]
```

***

<SourceLink />

### <kbd>method</kbd> `scorer_read`

```python theme={null}
scorer_read(req: ScorerReadReq) → ScorerReadRes
```

***

<SourceLink />

### <kbd>method</kbd> `table_create`

```python theme={null}
table_create(req: TableCreateReq) → TableCreateRes
```

***

<SourceLink />

### <kbd>method</kbd> `table_create_from_digests`

```python theme={null}
table_create_from_digests(
    req: TableCreateFromDigestsReq
) → TableCreateFromDigestsRes
```

***

<SourceLink />

### <kbd>method</kbd> `table_query`

```python theme={null}
table_query(req: TableQueryReq) → TableQueryRes
```

***

<SourceLink />

### <kbd>method</kbd> `table_query_stats`

```python theme={null}
table_query_stats(req: TableQueryStatsReq) → TableQueryStatsRes
```

***

<SourceLink />

### <kbd>method</kbd> `table_query_stats_batch`

```python theme={null}
table_query_stats_batch(req: TableQueryStatsBatchReq) → TableQueryStatsBatchRes
```

***

<SourceLink />

### <kbd>method</kbd> `table_query_stream`

```python theme={null}
table_query_stream(req: TableQueryReq) → Iterator[TableRowSchema]
```

***

<SourceLink />

### <kbd>method</kbd> `table_update`

```python theme={null}
table_update(req: TableUpdateReq) → TableUpdateRes
```

***

<SourceLink />

### <kbd>method</kbd> `tags_list`

```python theme={null}
tags_list(req: TagsListReq) → TagsListRes
```

***

<SourceLink />

### <kbd>method</kbd> `threads_query_stream`

```python theme={null}
threads_query_stream(req: ThreadsQueryReq) → Iterator[ThreadSchema]
```

***

<SourceLink />

### <kbd>method</kbd> `trace_usage`

```python theme={null}
trace_usage(req: 'TraceUsageReq') → TraceUsageRes
```

***

<SourceLink />

## <kbd>class</kbd> `GenAISpanRef`

**Pydantic Fields:**

* `trace_id`: `<class 'str'>`
* `span_id`: `<class 'str'>`

***

<SourceLink />

## <kbd>class</kbd> `ImageGenerationCreateReq`

**Pydantic Fields:**

* `project_id`: `<class 'str'>`
* `inputs`: `<class 'ImageGenerationRequestInputs'>`
* `wb_user_id`: `str | None`
* `track_llm_call`: `bool | None`

***

<SourceLink />

## <kbd>class</kbd> `ImageGenerationCreateRes`

**Pydantic Fields:**

* `response`: `dict[str, typing.Any]`
* `weave_call_id`: `str | None`

***

<SourceLink />

## <kbd>class</kbd> `ImageGenerationRequestInputs`

**Pydantic Fields:**

* `model`: `<class 'str'>`
* `prompt`: `<class 'str'>`
* `n`: `int | None`

***

<SourceLink />

## <kbd>class</kbd> `LLMAggregatedUsage`

Aggregated usage metrics for a specific LLM.

**Pydantic Fields:**

* `requests`: `<class 'int'>`
* `prompt_tokens`: `<class 'int'>`
* `completion_tokens`: `<class 'int'>`
* `total_tokens`: `<class 'int'>`
* `cache_read_input_tokens`: `<class 'int'>`
* `cache_creation_input_tokens`: `<class 'int'>`
* `prompt_tokens_total_cost`: `float | None`
* `completion_tokens_total_cost`: `float | None`
* `cache_read_input_tokens_total_cost`: `float | None`
* `cache_creation_input_tokens_total_cost`: `float | None`

***

<SourceLink />

## <kbd>class</kbd> `LLMCostSchema`

***

<SourceLink />

## <kbd>class</kbd> `LLMUsageSchema`

***

<SourceLink />

## <kbd>class</kbd> `ModelCreateBody`

**Pydantic Fields:**

* `name`: `<class 'str'>`
* `description`: `str | None`
* `source_code`: `<class 'str'>`
* `attributes`: `dict[str, typing.Any] | None`

***

<SourceLink />

## <kbd>class</kbd> `ModelCreateReq`

**Pydantic Fields:**

* `name`: `<class 'str'>`
* `description`: `str | None`
* `source_code`: `<class 'str'>`
* `attributes`: `dict[str, typing.Any] | None`
* `project_id`: `<class 'str'>`
* `wb_user_id`: `str | None`

***

<SourceLink />

## <kbd>class</kbd> `ModelCreateRes`

**Pydantic Fields:**

* `digest`: `<class 'str'>`
* `object_id`: `<class 'str'>`
* `version_index`: `<class 'int'>`
* `model_ref`: `<class 'str'>`

***

<SourceLink />

## <kbd>class</kbd> `ModelDeleteReq`

**Pydantic Fields:**

* `project_id`: `<class 'str'>`
* `object_id`: `<class 'str'>`
* `digests`: `list[str] | None`
* `wb_user_id`: `str | None`

***

<SourceLink />

## <kbd>class</kbd> `ModelDeleteRes`

**Pydantic Fields:**

* `num_deleted`: `<class 'int'>`

***

<SourceLink />

## <kbd>class</kbd> `ModelListReq`

**Pydantic Fields:**

* `project_id`: `<class 'str'>`
* `limit`: `int | None`
* `offset`: `int | None`

***

<SourceLink />

## <kbd>class</kbd> `ModelReadReq`

**Pydantic Fields:**

* `project_id`: `<class 'str'>`
* `object_id`: `<class 'str'>`
* `digest`: `<class 'str'>`

***

<SourceLink />

## <kbd>class</kbd> `ModelReadRes`

**Pydantic Fields:**

* `object_id`: `<class 'str'>`
* `digest`: `<class 'str'>`
* `version_index`: `<class 'int'>`
* `created_at`: `<class 'datetime.datetime'>`
* `name`: `<class 'str'>`
* `description`: `str | None`
* `source_code`: `<class 'str'>`
* `attributes`: `dict[str, typing.Any] | None`

***

<SourceLink />

## <kbd>class</kbd> `OTelExportReq`

**Pydantic Fields:**

* `processed_spans`: `list[ProcessedResourceSpans]`
* `project_id`: `<class 'str'>`
* `wb_user_id`: `str | None`

***

<SourceLink />

## <kbd>class</kbd> `OTelExportRes`

**Pydantic Fields:**

* `partial_success`: `ExportTracePartialSuccess | None`

***

<SourceLink />

## <kbd>class</kbd> `ObjAddTagsReq`

**Pydantic Fields:**

* `project_id`: `<class 'str'>`
* `object_id`: `<class 'str'>`
* `digest`: `<class 'str'>`
* `tags`: `list[str]`
* `wb_user_id`: `str | None`

<SourceLink />

### <kbd>method</kbd> `validate_tags`

```python theme={null}
validate_tags() → ObjAddTagsReq
```

***

<SourceLink />

## <kbd>class</kbd> `ObjAddTagsRes`

***

<SourceLink />

## <kbd>class</kbd> `ObjCreateReq`

**Pydantic Fields:**

* `obj`: `<class 'ObjSchemaForInsert'>`

***

<SourceLink />

## <kbd>class</kbd> `ObjCreateRes`

**Pydantic Fields:**

* `digest`: `<class 'str'>`
* `object_id`: `str | None`

***

<SourceLink />

## <kbd>class</kbd> `ObjDeleteReq`

**Pydantic Fields:**

* `project_id`: `<class 'str'>`
* `object_id`: `<class 'str'>`
* `digests`: `list[str] | None`

***

<SourceLink />

## <kbd>class</kbd> `ObjDeleteRes`

**Pydantic Fields:**

* `num_deleted`: `<class 'int'>`

***

<SourceLink />

## <kbd>class</kbd> `ObjQueryReq`

**Pydantic Fields:**

* `project_id`: `<class 'str'>`
* `filter`: `ObjectVersionFilter | None`
* `limit`: `int | None`
* `offset`: `int | None`
* `sort_by`: `list[weave.trace_server.common_interface.SortBy] | None`
* `metadata_only`: `bool | None`
* `include_storage_size`: `bool | None`
* `include_tags_and_aliases`: `bool | None`

***

<SourceLink />

## <kbd>class</kbd> `ObjQueryRes`

**Pydantic Fields:**

* `objs`: `list[ObjSchema]`

***

<SourceLink />

## <kbd>class</kbd> `ObjReadReq`

**Pydantic Fields:**

* `project_id`: `<class 'str'>`
* `object_id`: `<class 'str'>`
* `digest`: `<class 'str'>`
* `metadata_only`: `bool | None`
* `include_tags_and_aliases`: `bool | None`

***

<SourceLink />

## <kbd>class</kbd> `ObjReadRes`

**Pydantic Fields:**

* `obj`: `<class 'ObjSchema'>`

***

<SourceLink />

## <kbd>class</kbd> `ObjRemoveAliasesReq`

**Pydantic Fields:**

* `project_id`: `<class 'str'>`
* `object_id`: `<class 'str'>`
* `aliases`: `list[str]`
* `wb_user_id`: `str | None`

<SourceLink />

### <kbd>method</kbd> `validate_aliases`

```python theme={null}
validate_aliases() → ObjRemoveAliasesReq
```

***

<SourceLink />

## <kbd>class</kbd> `ObjRemoveAliasesRes`

***

<SourceLink />

## <kbd>class</kbd> `ObjRemoveTagsReq`

**Pydantic Fields:**

* `project_id`: `<class 'str'>`
* `object_id`: `<class 'str'>`
* `digest`: `<class 'str'>`
* `tags`: `list[str]`
* `wb_user_id`: `str | None`

***

<SourceLink />

## <kbd>class</kbd> `ObjRemoveTagsRes`

***

<SourceLink />

## <kbd>class</kbd> `ObjSchema`

**Pydantic Fields:**

* `project_id`: `<class 'str'>`
* `object_id`: `<class 'str'>`
* `created_at`: `<class 'datetime.datetime'>`
* `deleted_at`: `datetime.datetime | None`
* `digest`: `<class 'str'>`
* `version_index`: `<class 'int'>`
* `is_latest`: `<class 'int'>`
* `kind`: `<class 'str'>`
* `base_object_class`: `str | None`
* `leaf_object_class`: `str | None`
* `val`: `typing.Any`
* `wb_user_id`: `str | None`
* `size_bytes`: `int | None`
* `tags`: `list[str] | None`
* `aliases`: `list[str] | None`

***

<SourceLink />

## <kbd>class</kbd> `ObjSchemaForInsert`

**Pydantic Fields:**

* `project_id`: `<class 'str'>`
* `object_id`: `<class 'str'>`
* `val`: `typing.Any`
* `builtin_object_class`: `str | None`
* `set_base_object_class`: `str | None`
* `expected_digest`: `str | None`
* `wb_user_id`: `str | None`

<SourceLink />

### <kbd>method</kbd> `model_post_init`

```python theme={null}
model_post_init(context: Any) → None
```

***

<SourceLink />

## <kbd>class</kbd> `ObjSetAliasesReq`

**Pydantic Fields:**

* `project_id`: `<class 'str'>`
* `object_id`: `<class 'str'>`
* `digest`: `<class 'str'>`
* `aliases`: `list[str]`
* `wb_user_id`: `str | None`

<SourceLink />

### <kbd>method</kbd> `validate_aliases`

```python theme={null}
validate_aliases() → ObjSetAliasesReq
```

***

<SourceLink />

## <kbd>class</kbd> `ObjSetAliasesRes`

***

<SourceLink />

## <kbd>class</kbd> `ObjectInterface`

Object API endpoints for Trace Server.

This protocol contains object management APIs that provide cleaner, more RESTful interfaces. Implementations should support both this protocol and TraceServerInterface to maintain backward compatibility.

***

<SourceLink />

### <kbd>method</kbd> `call_end_v2`

```python theme={null}
call_end_v2(req: CallEndV2Req) → CallEndV2Res
```

***

<SourceLink />

### <kbd>method</kbd> `call_start_v2`

```python theme={null}
call_start_v2(req: CallStartV2Req) → CallStartV2Res
```

***

<SourceLink />

### <kbd>method</kbd> `calls_complete`

```python theme={null}
calls_complete(req: CallsUpsertCompleteReq) → CallsUpsertCompleteRes
```

***

<SourceLink />

### <kbd>method</kbd> `dataset_create`

```python theme={null}
dataset_create(req: DatasetCreateReq) → DatasetCreateRes
```

***

<SourceLink />

### <kbd>method</kbd> `dataset_delete`

```python theme={null}
dataset_delete(req: DatasetDeleteReq) → DatasetDeleteRes
```

***

<SourceLink />

### <kbd>method</kbd> `dataset_list`

```python theme={null}
dataset_list(req: DatasetListReq) → Iterator[DatasetReadRes]
```

***

<SourceLink />

### <kbd>method</kbd> `dataset_read`

```python theme={null}
dataset_read(req: DatasetReadReq) → DatasetReadRes
```

***

<SourceLink />

### <kbd>method</kbd> `eval_results_query`

```python theme={null}
eval_results_query(req: EvalResultsQueryReq) → EvalResultsQueryRes
```

***

<SourceLink />

### <kbd>method</kbd> `evaluation_create`

```python theme={null}
evaluation_create(req: EvaluationCreateReq) → EvaluationCreateRes
```

***

<SourceLink />

### <kbd>method</kbd> `evaluation_delete`

```python theme={null}
evaluation_delete(req: EvaluationDeleteReq) → EvaluationDeleteRes
```

***

<SourceLink />

### <kbd>method</kbd> `evaluation_list`

```python theme={null}
evaluation_list(req: EvaluationListReq) → Iterator[EvaluationReadRes]
```

***

<SourceLink />

### <kbd>method</kbd> `evaluation_read`

```python theme={null}
evaluation_read(req: EvaluationReadReq) → EvaluationReadRes
```

***

<SourceLink />

### <kbd>method</kbd> `evaluation_run_create`

```python theme={null}
evaluation_run_create(req: EvaluationRunCreateReq) → EvaluationRunCreateRes
```

***

<SourceLink />

### <kbd>method</kbd> `evaluation_run_delete`

```python theme={null}
evaluation_run_delete(req: EvaluationRunDeleteReq) → EvaluationRunDeleteRes
```

***

<SourceLink />

### <kbd>method</kbd> `evaluation_run_finish`

```python theme={null}
evaluation_run_finish(req: EvaluationRunFinishReq) → EvaluationRunFinishRes
```

***

<SourceLink />

### <kbd>method</kbd> `evaluation_run_list`

```python theme={null}
evaluation_run_list(req: EvaluationRunListReq) → Iterator[EvaluationRunReadRes]
```

***

<SourceLink />

### <kbd>method</kbd> `evaluation_run_read`

```python theme={null}
evaluation_run_read(req: EvaluationRunReadReq) → EvaluationRunReadRes
```

***

<SourceLink />

### <kbd>method</kbd> `model_create`

```python theme={null}
model_create(req: ModelCreateReq) → ModelCreateRes
```

***

<SourceLink />

### <kbd>method</kbd> `model_delete`

```python theme={null}
model_delete(req: ModelDeleteReq) → ModelDeleteRes
```

***

<SourceLink />

### <kbd>method</kbd> `model_list`

```python theme={null}
model_list(req: ModelListReq) → Iterator[ModelReadRes]
```

***

<SourceLink />

### <kbd>method</kbd> `model_read`

```python theme={null}
model_read(req: ModelReadReq) → ModelReadRes
```

***

<SourceLink />

### <kbd>method</kbd> `op_create`

```python theme={null}
op_create(req: OpCreateReq) → OpCreateRes
```

***

<SourceLink />

### <kbd>method</kbd> `op_delete`

```python theme={null}
op_delete(req: OpDeleteReq) → OpDeleteRes
```

***

<SourceLink />

### <kbd>method</kbd> `op_list`

```python theme={null}
op_list(req: OpListReq) → Iterator[OpReadRes]
```

***

<SourceLink />

### <kbd>method</kbd> `op_read`

```python theme={null}
op_read(req: OpReadReq) → OpReadRes
```

***

<SourceLink />

### <kbd>method</kbd> `prediction_create`

```python theme={null}
prediction_create(req: PredictionCreateReq) → PredictionCreateRes
```

***

<SourceLink />

### <kbd>method</kbd> `prediction_delete`

```python theme={null}
prediction_delete(req: PredictionDeleteReq) → PredictionDeleteRes
```

***

<SourceLink />

### <kbd>method</kbd> `prediction_finish`

```python theme={null}
prediction_finish(req: PredictionFinishReq) → PredictionFinishRes
```

***

<SourceLink />

### <kbd>method</kbd> `prediction_list`

```python theme={null}
prediction_list(req: PredictionListReq) → Iterator[PredictionReadRes]
```

***

<SourceLink />

### <kbd>method</kbd> `prediction_read`

```python theme={null}
prediction_read(req: PredictionReadReq) → PredictionReadRes
```

***

<SourceLink />

### <kbd>method</kbd> `score_create`

```python theme={null}
score_create(req: ScoreCreateReq) → ScoreCreateRes
```

***

<SourceLink />

### <kbd>method</kbd> `score_delete`

```python theme={null}
score_delete(req: ScoreDeleteReq) → ScoreDeleteRes
```

***

<SourceLink />

### <kbd>method</kbd> `score_list`

```python theme={null}
score_list(req: ScoreListReq) → Iterator[ScoreReadRes]
```

***

<SourceLink />

### <kbd>method</kbd> `score_read`

```python theme={null}
score_read(req: ScoreReadReq) → ScoreReadRes
```

***

<SourceLink />

### <kbd>method</kbd> `scorer_create`

```python theme={null}
scorer_create(req: ScorerCreateReq) → ScorerCreateRes
```

***

<SourceLink />

### <kbd>method</kbd> `scorer_delete`

```python theme={null}
scorer_delete(req: ScorerDeleteReq) → ScorerDeleteRes
```

***

<SourceLink />

### <kbd>method</kbd> `scorer_list`

```python theme={null}
scorer_list(req: ScorerListReq) → Iterator[ScorerReadRes]
```

***

<SourceLink />

### <kbd>method</kbd> `scorer_read`

```python theme={null}
scorer_read(req: ScorerReadReq) → ScorerReadRes
```

***

<SourceLink />

## <kbd>class</kbd> `ObjectVersionFilter`

**Pydantic Fields:**

* `base_object_classes`: `list[str] | None`
* `exclude_base_object_classes`: `list[str] | None`
* `leaf_object_classes`: `list[str] | None`
* `object_ids`: `list[str] | None`
* `is_op`: `bool | None`
* `latest_only`: `bool | None`
* `tags`: `list[str] | None`
* `aliases`: `list[str] | None`

***

<SourceLink />

## <kbd>class</kbd> `OpCreateBody`

Request body for creating an Op object via REST API.

This model excludes project\_id since it comes from the URL path in RESTful endpoints.

**Pydantic Fields:**

* `name`: `str | None`
* `source_code`: `str | None`

***

<SourceLink />

## <kbd>class</kbd> `OpCreateReq`

Request model for creating an Op object.

Extends OpCreateBody by adding project\_id for internal API usage.

**Pydantic Fields:**

* `name`: `str | None`
* `source_code`: `str | None`
* `project_id`: `<class 'str'>`
* `wb_user_id`: `str | None`

***

<SourceLink />

## <kbd>class</kbd> `OpCreateRes`

Response model for creating an Op object.

**Pydantic Fields:**

* `digest`: `<class 'str'>`
* `object_id`: `<class 'str'>`
* `version_index`: `<class 'int'>`

***

<SourceLink />

## <kbd>class</kbd> `OpDeleteReq`

**Pydantic Fields:**

* `project_id`: `<class 'str'>`
* `object_id`: `<class 'str'>`
* `digests`: `list[str] | None`
* `wb_user_id`: `str | None`

***

<SourceLink />

## <kbd>class</kbd> `OpDeleteRes`

**Pydantic Fields:**

* `num_deleted`: `<class 'int'>`

***

<SourceLink />

## <kbd>class</kbd> `OpListReq`

**Pydantic Fields:**

* `project_id`: `<class 'str'>`
* `limit`: `int | None`
* `offset`: `int | None`
* `wb_user_id`: `str | None`

***

<SourceLink />

## <kbd>class</kbd> `OpReadReq`

**Pydantic Fields:**

* `project_id`: `<class 'str'>`
* `object_id`: `<class 'str'>`
* `digest`: `<class 'str'>`
* `wb_user_id`: `str | None`

***

<SourceLink />

## <kbd>class</kbd> `OpReadRes`

Response model for reading an Op object.

The code field contains the actual source code of the op.

**Pydantic Fields:**

* `object_id`: `<class 'str'>`
* `digest`: `<class 'str'>`
* `version_index`: `<class 'int'>`
* `created_at`: `<class 'datetime.datetime'>`
* `code`: `<class 'str'>`

***

<SourceLink />

## <kbd>class</kbd> `PredictionCreateBody`

Request body for creating a Prediction via REST API.

This model excludes project\_id since it comes from the URL path in RESTful endpoints.

**Pydantic Fields:**

* `model`: `<class 'str'>`
* `inputs`: `dict[str, typing.Any]`
* `output`: `typing.Any`
* `evaluation_run_id`: `str | None`
* `genai_span_ref`: `list[GenAISpanRef] | None`

***

<SourceLink />

## <kbd>class</kbd> `PredictionCreateReq`

Request model for creating a Prediction.

Extends PredictionCreateBody by adding project\_id for internal API usage.

**Pydantic Fields:**

* `model`: `<class 'str'>`
* `inputs`: `dict[str, typing.Any]`
* `output`: `typing.Any`
* `evaluation_run_id`: `str | None`
* `genai_span_ref`: `list[GenAISpanRef] | None`
* `project_id`: `<class 'str'>`
* `wb_user_id`: `str | None`

***

<SourceLink />

## <kbd>class</kbd> `PredictionCreateRes`

**Pydantic Fields:**

* `prediction_id`: `<class 'str'>`

***

<SourceLink />

## <kbd>class</kbd> `PredictionDeleteReq`

**Pydantic Fields:**

* `project_id`: `<class 'str'>`
* `prediction_ids`: `list[str]`
* `wb_user_id`: `str | None`

***

<SourceLink />

## <kbd>class</kbd> `PredictionDeleteRes`

**Pydantic Fields:**

* `num_deleted`: `<class 'int'>`

***

<SourceLink />

## <kbd>class</kbd> `PredictionFinishReq`

**Pydantic Fields:**

* `project_id`: `<class 'str'>`
* `prediction_id`: `<class 'str'>`
* `wb_user_id`: `str | None`

***

<SourceLink />

## <kbd>class</kbd> `PredictionFinishRes`

**Pydantic Fields:**

* `success`: `<class 'bool'>`

***

<SourceLink />

## <kbd>class</kbd> `PredictionListReq`

**Pydantic Fields:**

* `project_id`: `<class 'str'>`
* `evaluation_run_id`: `str | None`
* `limit`: `int | None`
* `offset`: `int | None`
* `wb_user_id`: `str | None`

***

<SourceLink />

## <kbd>class</kbd> `PredictionListRes`

**Pydantic Fields:**

* `predictions`: `list[PredictionReadRes]`

***

<SourceLink />

## <kbd>class</kbd> `PredictionReadReq`

**Pydantic Fields:**

* `project_id`: `<class 'str'>`
* `prediction_id`: `<class 'str'>`
* `wb_user_id`: `str | None`

***

<SourceLink />

## <kbd>class</kbd> `PredictionReadRes`

**Pydantic Fields:**

* `prediction_id`: `<class 'str'>`
* `model`: `<class 'str'>`
* `inputs`: `dict[str, typing.Any]`
* `output`: `typing.Any`
* `evaluation_run_id`: `str | None`
* `wb_user_id`: `str | None`

***

<SourceLink />

## <kbd>class</kbd> `ProcessedResourceSpans`

**Pydantic Fields:**

* `entity`: `<class 'str'>`
* `project`: `<class 'str'>`
* `run_id`: `str | None`
* `resource_spans`: `<class 'opentelemetry.proto.trace.v1.trace_pb2.ResourceSpans'>`

***

<SourceLink />

## <kbd>class</kbd> `ProjectStatsReq`

**Pydantic Fields:**

* `project_id`: `<class 'str'>`
* `include_trace_storage_size`: `bool | None`
* `include_object_storage_size`: `bool | None`
* `include_table_storage_size`: `bool | None`
* `include_file_storage_size`: `bool | None`

***

<SourceLink />

## <kbd>class</kbd> `ProjectStatsRes`

**Pydantic Fields:**

* `trace_storage_size_bytes`: `<class 'int'>`
* `objects_storage_size_bytes`: `<class 'int'>`
* `tables_storage_size_bytes`: `<class 'int'>`
* `files_storage_size_bytes`: `<class 'int'>`

***

<SourceLink />

## <kbd>class</kbd> `ProjectTTLSettingsReadReq`

**Pydantic Fields:**

* `project_id`: `<class 'str'>`

***

<SourceLink />

## <kbd>class</kbd> `ProjectTTLSettingsReadRes`

**Pydantic Fields:**

* `retention_days`: `int | None`

***

<SourceLink />

## <kbd>class</kbd> `ProjectTTLSettingsUpdateReq`

**Pydantic Fields:**

* `project_id`: `<class 'str'>`
* `retention_days`: `int | None`
* `wb_user_id`: `str | None`

***

<SourceLink />

## <kbd>class</kbd> `ProjectTTLSettingsUpdateRes`

**Pydantic Fields:**

* `retention_days`: `int | None`

***

<SourceLink />

## <kbd>class</kbd> `RefsReadBatchReq`

**Pydantic Fields:**

* `refs`: `list[str]`

***

<SourceLink />

## <kbd>class</kbd> `RefsReadBatchRes`

**Pydantic Fields:**

* `vals`: `list[typing.Any]`

***

<SourceLink />

## <kbd>class</kbd> `RescoreBody`

Request body for rescoring via REST API (excludes server-set fields).

**Pydantic Fields:**

* `source_evaluation_run_id`: `<class 'str'>`
* `scorer_refs`: `list[str]`

***

<SourceLink />

## <kbd>class</kbd> `RescoreReq`

Full rescore request including server-set fields.

**Pydantic Fields:**

* `source_evaluation_run_id`: `<class 'str'>`
* `scorer_refs`: `list[str]`
* `project_id`: `<class 'str'>`
* `wb_user_id`: `str | None`

***

<SourceLink />

## <kbd>class</kbd> `RescoreRes`

Response for a rescore request.

**Pydantic Fields:**

* `call_id`: `<class 'str'>`
* `evaluation_run_id`: `<class 'str'>`

***

<SourceLink />

## <kbd>class</kbd> `RescoringArgs`

Arguments for a rescore job dispatched to the evaluate-model worker.

Differs from EvaluateModelArgs: no model is loaded, no predictions are run. Only scorer(s) are applied to existing predictions from source\_evaluation\_run\_id.

**Pydantic Fields:**

* `job_type`: `typing.Literal['rescore']`
* `project_id`: `<class 'str'>`
* `source_evaluation_run_id`: `<class 'str'>`
* `scorer_refs`: `list[str]`
* `wb_user_id`: `str | None`
* `new_evaluation_run_id`: `<class 'str'>`

***

<SourceLink />

## <kbd>class</kbd> `ScoreCreateBody`

Request body for creating a Score via REST API.

This model excludes project\_id since it comes from the URL path in RESTful endpoints.

**Pydantic Fields:**

* `prediction_id`: `<class 'str'>`
* `scorer`: `<class 'str'>`
* `value`: `typing.Any`
* `evaluation_run_id`: `str | None`

***

<SourceLink />

## <kbd>class</kbd> `ScoreCreateReq`

Request model for creating a Score.

Extends ScoreCreateBody by adding project\_id for internal API usage.

**Pydantic Fields:**

* `prediction_id`: `<class 'str'>`
* `scorer`: `<class 'str'>`
* `value`: `typing.Any`
* `evaluation_run_id`: `str | None`
* `project_id`: `<class 'str'>`
* `wb_user_id`: `str | None`

***

<SourceLink />

## <kbd>class</kbd> `ScoreCreateRes`

**Pydantic Fields:**

* `score_id`: `<class 'str'>`

***

<SourceLink />

## <kbd>class</kbd> `ScoreDeleteReq`

**Pydantic Fields:**

* `project_id`: `<class 'str'>`
* `score_ids`: `list[str]`
* `wb_user_id`: `str | None`

***

<SourceLink />

## <kbd>class</kbd> `ScoreDeleteRes`

**Pydantic Fields:**

* `num_deleted`: `<class 'int'>`

***

<SourceLink />

## <kbd>class</kbd> `ScoreListReq`

**Pydantic Fields:**

* `project_id`: `<class 'str'>`
* `evaluation_run_id`: `str | None`
* `limit`: `int | None`
* `offset`: `int | None`
* `wb_user_id`: `str | None`

***

<SourceLink />

## <kbd>class</kbd> `ScoreReadReq`

**Pydantic Fields:**

* `project_id`: `<class 'str'>`
* `score_id`: `<class 'str'>`
* `wb_user_id`: `str | None`

***

<SourceLink />

## <kbd>class</kbd> `ScoreReadRes`

**Pydantic Fields:**

* `score_id`: `<class 'str'>`
* `scorer`: `<class 'str'>`
* `value`: `typing.Any`
* `evaluation_run_id`: `str | None`
* `wb_user_id`: `str | None`

***

<SourceLink />

## <kbd>class</kbd> `ScorerCreateBody`

**Pydantic Fields:**

* `name`: `<class 'str'>`
* `description`: `str | None`
* `op_source_code`: `<class 'str'>`

***

<SourceLink />

## <kbd>class</kbd> `ScorerCreateReq`

**Pydantic Fields:**

* `name`: `<class 'str'>`
* `description`: `str | None`
* `op_source_code`: `<class 'str'>`
* `project_id`: `<class 'str'>`
* `wb_user_id`: `str | None`

***

<SourceLink />

## <kbd>class</kbd> `ScorerCreateRes`

**Pydantic Fields:**

* `digest`: `<class 'str'>`
* `object_id`: `<class 'str'>`
* `version_index`: `<class 'int'>`
* `scorer`: `<class 'str'>`

***

<SourceLink />

## <kbd>class</kbd> `ScorerDeleteReq`

**Pydantic Fields:**

* `project_id`: `<class 'str'>`
* `object_id`: `<class 'str'>`
* `digests`: `list[str] | None`
* `wb_user_id`: `str | None`

***

<SourceLink />

## <kbd>class</kbd> `ScorerDeleteRes`

**Pydantic Fields:**

* `num_deleted`: `<class 'int'>`

***

<SourceLink />

## <kbd>class</kbd> `ScorerListReq`

**Pydantic Fields:**

* `project_id`: `<class 'str'>`
* `limit`: `int | None`
* `offset`: `int | None`
* `wb_user_id`: `str | None`

***

<SourceLink />

## <kbd>class</kbd> `ScorerReadReq`

**Pydantic Fields:**

* `project_id`: `<class 'str'>`
* `object_id`: `<class 'str'>`
* `digest`: `<class 'str'>`
* `wb_user_id`: `str | None`

***

<SourceLink />

## <kbd>class</kbd> `ScorerReadRes`

**Pydantic Fields:**

* `object_id`: `<class 'str'>`
* `digest`: `<class 'str'>`
* `version_index`: `<class 'int'>`
* `created_at`: `<class 'datetime.datetime'>`
* `name`: `<class 'str'>`
* `description`: `str | None`
* `score_op`: `<class 'str'>`

***

<SourceLink />

## <kbd>class</kbd> `StartedCallSchemaForInsert`

**Pydantic Fields:**

* `project_id`: `<class 'str'>`
* `id`: `str | None`
* `op_name`: `<class 'str'>`
* `display_name`: `str | None`
* `trace_id`: `str | None`
* `parent_id`: `str | None`
* `thread_id`: `str | None`
* `turn_id`: `str | None`
* `started_at`: `<class 'datetime.datetime'>`
* `attributes`: `dict[str, typing.Any]`
* `inputs`: `dict[str, typing.Any]`
* `otel_dump`: `dict[str, typing.Any] | None`
* `wb_user_id`: `str | None`
* `wb_run_id`: `str | None`
* `wb_run_step`: `int | None`

***

<SourceLink />

## <kbd>class</kbd> `SummaryInsertMap`

***

<SourceLink />

## <kbd>class</kbd> `SummaryMap`

***

<SourceLink />

## <kbd>class</kbd> `TableAppendSpec`

**Pydantic Fields:**

* `append`: `<class 'TableAppendSpecPayload'>`

***

<SourceLink />

## <kbd>class</kbd> `TableAppendSpecPayload`

**Pydantic Fields:**

* `row`: `dict[str, typing.Any]`

***

<SourceLink />

## <kbd>class</kbd> `TableCreateFromDigestsReq`

**Pydantic Fields:**

* `project_id`: `<class 'str'>`
* `row_digests`: `list[str]`
* `expected_digest`: `str | None`

***

<SourceLink />

## <kbd>class</kbd> `TableCreateFromDigestsRes`

**Pydantic Fields:**

* `digest`: `<class 'str'>`

***

<SourceLink />

## <kbd>class</kbd> `TableCreateReq`

**Pydantic Fields:**

* `table`: `<class 'TableSchemaForInsert'>`

***

<SourceLink />

## <kbd>class</kbd> `TableCreateRes`

**Pydantic Fields:**

* `digest`: `<class 'str'>`
* `row_digests`: `list[str]`

***

<SourceLink />

## <kbd>class</kbd> `TableInsertSpec`

**Pydantic Fields:**

* `insert`: `<class 'TableInsertSpecPayload'>`

***

<SourceLink />

## <kbd>class</kbd> `TableInsertSpecPayload`

**Pydantic Fields:**

* `index`: `<class 'int'>`
* `row`: `dict[str, typing.Any]`

***

<SourceLink />

## <kbd>class</kbd> `TablePopSpec`

**Pydantic Fields:**

* `pop`: `<class 'TablePopSpecPayload'>`

***

<SourceLink />

## <kbd>class</kbd> `TablePopSpecPayload`

**Pydantic Fields:**

* `index`: `<class 'int'>`

***

<SourceLink />

## <kbd>class</kbd> `TableQueryReq`

**Pydantic Fields:**

* `project_id`: `<class 'str'>`
* `digest`: `<class 'str'>`
* `filter`: `TableRowFilter | None`
* `limit`: `int | None`
* `offset`: `int | None`
* `sort_by`: `list[weave.trace_server.common_interface.SortBy] | None`

***

<SourceLink />

## <kbd>class</kbd> `TableQueryRes`

**Pydantic Fields:**

* `rows`: `list[TableRowSchema]`

***

<SourceLink />

## <kbd>class</kbd> `TableQueryStatsBatchReq`

**Pydantic Fields:**

* `project_id`: `<class 'str'>`
* `digests`: `list[str] | None`
* `include_storage_size`: `bool | None`

***

<SourceLink />

## <kbd>class</kbd> `TableQueryStatsBatchRes`

**Pydantic Fields:**

* `tables`: `list[TableStatsRow]`

***

<SourceLink />

## <kbd>class</kbd> `TableQueryStatsReq`

**Pydantic Fields:**

* `project_id`: `<class 'str'>`
* `digest`: `<class 'str'>`

***

<SourceLink />

## <kbd>class</kbd> `TableQueryStatsRes`

**Pydantic Fields:**

* `count`: `<class 'int'>`

***

<SourceLink />

## <kbd>class</kbd> `TableRowFilter`

**Pydantic Fields:**

* `row_digests`: `list[str] | None`

***

<SourceLink />

## <kbd>class</kbd> `TableRowSchema`

**Pydantic Fields:**

* `digest`: `<class 'str'>`
* `val`: `typing.Any`
* `original_index`: `int | None`

***

<SourceLink />

## <kbd>class</kbd> `TableSchemaForInsert`

**Pydantic Fields:**

* `project_id`: `<class 'str'>`
* `rows`: `list[dict[str, typing.Any]]`
* `expected_digest`: `str | None`

***

<SourceLink />

## <kbd>class</kbd> `TableStatsRow`

**Pydantic Fields:**

* `count`: `<class 'int'>`
* `digest`: `<class 'str'>`
* `storage_size_bytes`: `int | None`

***

<SourceLink />

## <kbd>class</kbd> `TableUpdateReq`

**Pydantic Fields:**

* `project_id`: `<class 'str'>`
* `base_digest`: `<class 'str'>`
* `updates`: `list[TableAppendSpec | TablePopSpec | TableInsertSpec]`

***

<SourceLink />

## <kbd>class</kbd> `TableUpdateRes`

**Pydantic Fields:**

* `digest`: `<class 'str'>`
* `updated_row_digests`: `list[str]`

***

<SourceLink />

## <kbd>class</kbd> `TagsListReq`

**Pydantic Fields:**

* `project_id`: `<class 'str'>`
* `wb_user_id`: `str | None`

***

<SourceLink />

## <kbd>class</kbd> `TagsListRes`

**Pydantic Fields:**

* `tags`: `list[str]`

***

<SourceLink />

## <kbd>class</kbd> `ThreadSchema`

**Pydantic Fields:**

* `thread_id`: `<class 'str'>`
* `turn_count`: `<class 'int'>`
* `start_time`: `<class 'datetime.datetime'>`
* `last_updated`: `<class 'datetime.datetime'>`
* `first_turn_id`: `str | None`
* `last_turn_id`: `str | None`
* `p50_turn_duration_ms`: `float | None`
* `p99_turn_duration_ms`: `float | None`

***

<SourceLink />

## <kbd>class</kbd> `ThreadsQueryFilter`

**Pydantic Fields:**

* `after_datetime`: `datetime.datetime | None`
* `before_datetime`: `datetime.datetime | None`
* `thread_ids`: `list[str] | None`

***

<SourceLink />

## <kbd>class</kbd> `ThreadsQueryReq`

Query threads with aggregated statistics based on turn calls only.

Turn calls are the immediate children of thread contexts (where call.id == turn\_id). This provides meaningful conversation-level statistics rather than including all nested implementation details.

**Pydantic Fields:**

* `project_id`: `<class 'str'>`
* `filter`: `ThreadsQueryFilter | None`
* `limit`: `int | None`
* `offset`: `int | None`
* `sort_by`: `list[weave.trace_server.common_interface.SortBy] | None`

***

<SourceLink />

## <kbd>class</kbd> `TraceServerInterface`

***

<SourceLink />

### <kbd>method</kbd> `actions_execute_batch`

```python theme={null}
actions_execute_batch(req: ActionsExecuteBatchReq) → ActionsExecuteBatchRes
```

***

<SourceLink />

### <kbd>method</kbd> `agent_agents_query`

```python theme={null}
agent_agents_query(req: AgentsQueryReq) → AgentsQueryRes
```

***

<SourceLink />

### <kbd>method</kbd> `agent_conversation_chat`

```python theme={null}
agent_conversation_chat(
    req: AgentConversationChatReq
) → AgentConversationChatRes
```

***

<SourceLink />

### <kbd>method</kbd> `agent_custom_attrs_schema`

```python theme={null}
agent_custom_attrs_schema(
    req: AgentCustomAttrsSchemaReq
) → AgentCustomAttrsSchemaRes
```

***

<SourceLink />

### <kbd>method</kbd> `agent_search`

```python theme={null}
agent_search(req: AgentSearchReq) → AgentSearchRes
```

***

<SourceLink />

### <kbd>method</kbd> `agent_spans_query`

```python theme={null}
agent_spans_query(req: AgentSpansQueryReq) → AgentSpansQueryRes
```

***

<SourceLink />

### <kbd>method</kbd> `agent_spans_stats`

```python theme={null}
agent_spans_stats(req: AgentSpanStatsReq) → AgentSpanStatsRes
```

***

<SourceLink />

### <kbd>method</kbd> `agent_traces_chat`

```python theme={null}
agent_traces_chat(req: AgentTraceChatReq) → AgentTraceChatRes
```

***

<SourceLink />

### <kbd>method</kbd> `agent_versions_query`

```python theme={null}
agent_versions_query(req: AgentVersionsQueryReq) → AgentVersionsQueryRes
```

***

<SourceLink />

### <kbd>method</kbd> `aliases_list`

```python theme={null}
aliases_list(req: AliasesListReq) → AliasesListRes
```

***

<SourceLink />

### <kbd>method</kbd> `annotation_queue_add_calls`

```python theme={null}
annotation_queue_add_calls(
    req: AnnotationQueueAddCallsReq
) → AnnotationQueueAddCallsRes
```

***

<SourceLink />

### <kbd>method</kbd> `annotation_queue_create`

```python theme={null}
annotation_queue_create(
    req: AnnotationQueueCreateReq
) → AnnotationQueueCreateRes
```

***

<SourceLink />

### <kbd>method</kbd> `annotation_queue_delete`

```python theme={null}
annotation_queue_delete(
    req: AnnotationQueueDeleteReq
) → AnnotationQueueDeleteRes
```

***

<SourceLink />

### <kbd>method</kbd> `annotation_queue_items_query`

```python theme={null}
annotation_queue_items_query(
    req: AnnotationQueueItemsQueryReq
) → AnnotationQueueItemsQueryRes
```

***

<SourceLink />

### <kbd>method</kbd> `annotation_queue_read`

```python theme={null}
annotation_queue_read(req: AnnotationQueueReadReq) → AnnotationQueueReadRes
```

***

<SourceLink />

### <kbd>method</kbd> `annotation_queue_update`

```python theme={null}
annotation_queue_update(
    req: AnnotationQueueUpdateReq
) → AnnotationQueueUpdateRes
```

***

<SourceLink />

### <kbd>method</kbd> `annotation_queues_query_stream`

```python theme={null}
annotation_queues_query_stream(
    req: AnnotationQueuesQueryReq
) → Iterator[AnnotationQueueSchema]
```

***

<SourceLink />

### <kbd>method</kbd> `annotation_queues_stats`

```python theme={null}
annotation_queues_stats(
    req: AnnotationQueuesStatsReq
) → AnnotationQueuesStatsRes
```

***

<SourceLink />

### <kbd>method</kbd> `annotator_queue_items_progress_update`

```python theme={null}
annotator_queue_items_progress_update(
    req: AnnotatorQueueItemsProgressUpdateReq
) → AnnotatorQueueItemsProgressUpdateRes
```

***

<SourceLink />

### <kbd>method</kbd> `call_end`

```python theme={null}
call_end(req: CallEndReq) → CallEndRes
```

***

<SourceLink />

### <kbd>method</kbd> `call_read`

```python theme={null}
call_read(req: CallReadReq) → CallReadRes
```

***

<SourceLink />

### <kbd>method</kbd> `call_start`

```python theme={null}
call_start(req: CallStartReq) → CallStartRes
```

***

<SourceLink />

### <kbd>method</kbd> `call_start_batch`

```python theme={null}
call_start_batch(req: CallCreateBatchReq) → CallCreateBatchRes
```

***

<SourceLink />

### <kbd>method</kbd> `call_stats`

```python theme={null}
call_stats(req: 'CallStatsReq') → CallStatsRes
```

***

<SourceLink />

### <kbd>method</kbd> `call_update`

```python theme={null}
call_update(req: CallUpdateReq) → CallUpdateRes
```

***

<SourceLink />

### <kbd>method</kbd> `calls_delete`

```python theme={null}
calls_delete(req: CallsDeleteReq) → CallsDeleteRes
```

***

<SourceLink />

### <kbd>method</kbd> `calls_query`

```python theme={null}
calls_query(req: CallsQueryReq) → CallsQueryRes
```

***

<SourceLink />

### <kbd>method</kbd> `calls_query_stats`

```python theme={null}
calls_query_stats(req: CallsQueryStatsReq) → CallsQueryStatsRes
```

***

<SourceLink />

### <kbd>method</kbd> `calls_query_stream`

```python theme={null}
calls_query_stream(req: CallsQueryReq) → Iterator[CallSchema]
```

***

<SourceLink />

### <kbd>method</kbd> `calls_score`

```python theme={null}
calls_score(req: CallsScoreReq) → CallsScoreRes
```

***

<SourceLink />

### <kbd>method</kbd> `calls_usage`

```python theme={null}
calls_usage(req: 'CallsUsageReq') → CallsUsageRes
```

***

<SourceLink />

### <kbd>method</kbd> `completions_create`

```python theme={null}
completions_create(req: CompletionsCreateReq) → CompletionsCreateRes
```

***

<SourceLink />

### <kbd>method</kbd> `completions_create_stream`

```python theme={null}
completions_create_stream(req: CompletionsCreateReq) → Iterator[dict[str, Any]]
```

***

<SourceLink />

### <kbd>method</kbd> `cost_create`

```python theme={null}
cost_create(req: CostCreateReq) → CostCreateRes
```

***

<SourceLink />

### <kbd>method</kbd> `cost_purge`

```python theme={null}
cost_purge(req: CostPurgeReq) → CostPurgeRes
```

***

<SourceLink />

### <kbd>method</kbd> `cost_query`

```python theme={null}
cost_query(req: CostQueryReq) → CostQueryRes
```

***

<SourceLink />

### <kbd>method</kbd> `evaluate_model`

```python theme={null}
evaluate_model(req: EvaluateModelReq) → EvaluateModelRes
```

***

<SourceLink />

### <kbd>method</kbd> `evaluation_status`

```python theme={null}
evaluation_status(req: EvaluationStatusReq) → EvaluationStatusRes
```

***

<SourceLink />

### <kbd>method</kbd> `feedback_create`

```python theme={null}
feedback_create(req: FeedbackCreateReq) → FeedbackCreateRes
```

***

<SourceLink />

### <kbd>method</kbd> `feedback_create_batch`

```python theme={null}
feedback_create_batch(req: FeedbackCreateBatchReq) → FeedbackCreateBatchRes
```

***

<SourceLink />

### <kbd>method</kbd> `feedback_payload_schema`

```python theme={null}
feedback_payload_schema(
    req: FeedbackPayloadSchemaReq
) → FeedbackPayloadSchemaRes
```

***

<SourceLink />

### <kbd>method</kbd> `feedback_purge`

```python theme={null}
feedback_purge(req: FeedbackPurgeReq) → FeedbackPurgeRes
```

***

<SourceLink />

### <kbd>method</kbd> `feedback_query`

```python theme={null}
feedback_query(req: FeedbackQueryReq) → FeedbackQueryRes
```

***

<SourceLink />

### <kbd>method</kbd> `feedback_replace`

```python theme={null}
feedback_replace(req: FeedbackReplaceReq) → FeedbackReplaceRes
```

***

<SourceLink />

### <kbd>method</kbd> `feedback_stats`

```python theme={null}
feedback_stats(req: FeedbackStatsReq) → FeedbackStatsRes
```

***

<SourceLink />

### <kbd>method</kbd> `file_content_read`

```python theme={null}
file_content_read(req: FileContentReadReq) → FileContentReadRes
```

***

<SourceLink />

### <kbd>method</kbd> `file_create`

```python theme={null}
file_create(req: FileCreateReq) → FileCreateRes
```

***

<SourceLink />

### <kbd>method</kbd> `files_stats`

```python theme={null}
files_stats(req: FilesStatsReq) → FilesStatsRes
```

***

<SourceLink />

### <kbd>method</kbd> `genai_otel_export`

```python theme={null}
genai_otel_export(req: GenAIOTelExportReq) → GenAIOTelExportRes
```

***

<SourceLink />

### <kbd>method</kbd> `image_create`

```python theme={null}
image_create(req: ImageGenerationCreateReq) → ImageGenerationCreateRes
```

***

<SourceLink />

### <kbd>method</kbd> `obj_add_tags`

```python theme={null}
obj_add_tags(req: ObjAddTagsReq) → ObjAddTagsRes
```

***

<SourceLink />

### <kbd>method</kbd> `obj_create`

```python theme={null}
obj_create(req: ObjCreateReq) → ObjCreateRes
```

***

<SourceLink />

### <kbd>method</kbd> `obj_delete`

```python theme={null}
obj_delete(req: ObjDeleteReq) → ObjDeleteRes
```

***

<SourceLink />

### <kbd>method</kbd> `obj_read`

```python theme={null}
obj_read(req: ObjReadReq) → ObjReadRes
```

***

<SourceLink />

### <kbd>method</kbd> `obj_remove_aliases`

```python theme={null}
obj_remove_aliases(req: ObjRemoveAliasesReq) → ObjRemoveAliasesRes
```

***

<SourceLink />

### <kbd>method</kbd> `obj_remove_tags`

```python theme={null}
obj_remove_tags(req: ObjRemoveTagsReq) → ObjRemoveTagsRes
```

***

<SourceLink />

### <kbd>method</kbd> `obj_set_aliases`

```python theme={null}
obj_set_aliases(req: ObjSetAliasesReq) → ObjSetAliasesRes
```

***

<SourceLink />

### <kbd>method</kbd> `objs_query`

```python theme={null}
objs_query(req: ObjQueryReq) → ObjQueryRes
```

***

<SourceLink />

### <kbd>method</kbd> `otel_export`

```python theme={null}
otel_export(req: OTelExportReq) → OTelExportRes
```

***

<SourceLink />

### <kbd>method</kbd> `project_stats`

```python theme={null}
project_stats(req: ProjectStatsReq) → ProjectStatsRes
```

***

<SourceLink />

### <kbd>method</kbd> `project_ttl_settings_read`

```python theme={null}
project_ttl_settings_read(
    req: ProjectTTLSettingsReadReq
) → ProjectTTLSettingsReadRes
```

***

<SourceLink />

### <kbd>method</kbd> `project_ttl_settings_update`

```python theme={null}
project_ttl_settings_update(
    req: ProjectTTLSettingsUpdateReq
) → ProjectTTLSettingsUpdateRes
```

***

<SourceLink />

### <kbd>method</kbd> `refs_read_batch`

```python theme={null}
refs_read_batch(req: RefsReadBatchReq) → RefsReadBatchRes
```

***

<SourceLink />

### <kbd>method</kbd> `rescore`

```python theme={null}
rescore(req: RescoreReq) → RescoreRes
```

***

<SourceLink />

### <kbd>method</kbd> `table_create`

```python theme={null}
table_create(req: TableCreateReq) → TableCreateRes
```

***

<SourceLink />

### <kbd>method</kbd> `table_create_from_digests`

```python theme={null}
table_create_from_digests(
    req: TableCreateFromDigestsReq
) → TableCreateFromDigestsRes
```

***

<SourceLink />

### <kbd>method</kbd> `table_query`

```python theme={null}
table_query(req: TableQueryReq) → TableQueryRes
```

***

<SourceLink />

### <kbd>method</kbd> `table_query_stats`

```python theme={null}
table_query_stats(req: TableQueryStatsReq) → TableQueryStatsRes
```

***

<SourceLink />

### <kbd>method</kbd> `table_query_stats_batch`

```python theme={null}
table_query_stats_batch(req: TableQueryStatsBatchReq) → TableQueryStatsBatchRes
```

***

<SourceLink />

### <kbd>method</kbd> `table_query_stream`

```python theme={null}
table_query_stream(req: TableQueryReq) → Iterator[TableRowSchema]
```

***

<SourceLink />

### <kbd>method</kbd> `table_update`

```python theme={null}
table_update(req: TableUpdateReq) → TableUpdateRes
```

***

<SourceLink />

### <kbd>method</kbd> `tags_list`

```python theme={null}
tags_list(req: TagsListReq) → TagsListRes
```

***

<SourceLink />

### <kbd>method</kbd> `threads_query_stream`

```python theme={null}
threads_query_stream(req: ThreadsQueryReq) → Iterator[ThreadSchema]
```

***

<SourceLink />

### <kbd>method</kbd> `trace_usage`

```python theme={null}
trace_usage(req: 'TraceUsageReq') → TraceUsageRes
```

***

<SourceLink />

## <kbd>class</kbd> `TraceStatus`

***

<SourceLink />

## <kbd>class</kbd> `TraceUsageReq`

Request to compute per-call usage for a trace, with descendant rollup.

This endpoint returns usage metrics for each call in the trace, where each call's metrics include the sum of its own usage plus all descendants' usage. Use this for trace view where you want to see rolled-up metrics per call.

Note: All matching calls are loaded into memory for aggregation. For very large result sets (>10k calls), consider using more specific filters or pagination at the application layer.

**Pydantic Fields:**

* `project_id`: `<class 'str'>`
* `filter`: `CallsFilter | None`
* `query`: `weave.trace_server.interface.query.Query | None`
* `include_costs`: `<class 'bool'>`
* `limit`: `<class 'int'>`

***

<SourceLink />

## <kbd>class</kbd> `TraceUsageRes`

Response with per-call usage metrics (each includes descendant contributions).

**Pydantic Fields:**

* `call_usage`: `dict[str, dict[str, LLMAggregatedUsage]]`
* `unfinished_call_ids`: `list[str]`

***

<SourceLink />

## <kbd>class</kbd> `UsageMetricSpec`

Specification for a usage metric to aggregate (grouped by model).

**Pydantic Fields:**

* `metric`: `typing.Literal['input_tokens', 'output_tokens', 'total_tokens', 'cache_read_input_tokens', 'cache_creation_input_tokens', 'input_cost', 'output_cost', 'total_cost']`
* `aggregations`: `list[AggregationType]`
* `percentiles`: `list[float]`

***

<SourceLink />

## <kbd>class</kbd> `WeaveSummarySchema`


# remote_http_trace_server
Source: https://docs.wandb.ai/weave/reference/python-sdk/trace_server_bindings/remote_http_trace_server

Python SDK reference for weave.trace_server_bindings.remote_http_trace_server

# API Overview

***

<SourceLink />

## <kbd>class</kbd> `RemoteHTTPTraceServer`

<SourceLink />

### <kbd>method</kbd> `__init__`

```python theme={null}
__init__(
    trace_server_url: str,
    should_batch: bool = False,
    remote_request_bytes_limit: int = 32505856,
    auth: tuple[str, str] | None = None,
    extra_headers: dict[str, str] | None = None
)
```

***

<SourceLink />

### <kbd>method</kbd> `actions_execute_batch`

```python theme={null}
actions_execute_batch(req: ActionsExecuteBatchReq) → ActionsExecuteBatchRes
```

***

<SourceLink />

### <kbd>method</kbd> `aliases_list`

```python theme={null}
aliases_list(req: AliasesListReq) → AliasesListRes
```

***

<SourceLink />

### <kbd>method</kbd> `annotation_queue_add_calls`

```python theme={null}
annotation_queue_add_calls(
    req: AnnotationQueueAddCallsReq
) → AnnotationQueueAddCallsRes
```

***

<SourceLink />

### <kbd>method</kbd> `annotation_queue_create`

```python theme={null}
annotation_queue_create(
    req: AnnotationQueueCreateReq
) → AnnotationQueueCreateRes
```

***

<SourceLink />

### <kbd>method</kbd> `annotation_queue_delete`

```python theme={null}
annotation_queue_delete(
    req: AnnotationQueueDeleteReq
) → AnnotationQueueDeleteRes
```

***

<SourceLink />

### <kbd>method</kbd> `annotation_queue_items_query`

```python theme={null}
annotation_queue_items_query(
    req: AnnotationQueueItemsQueryReq
) → AnnotationQueueItemsQueryRes
```

***

<SourceLink />

### <kbd>method</kbd> `annotation_queue_read`

```python theme={null}
annotation_queue_read(req: AnnotationQueueReadReq) → AnnotationQueueReadRes
```

***

<SourceLink />

### <kbd>method</kbd> `annotation_queue_update`

```python theme={null}
annotation_queue_update(
    req: AnnotationQueueUpdateReq
) → AnnotationQueueUpdateRes
```

***

<SourceLink />

### <kbd>method</kbd> `annotation_queues_query_stream`

```python theme={null}
annotation_queues_query_stream(
    req: AnnotationQueuesQueryReq
) → Iterator[AnnotationQueueSchema]
```

***

<SourceLink />

### <kbd>method</kbd> `annotation_queues_stats`

```python theme={null}
annotation_queues_stats(
    req: AnnotationQueuesStatsReq
) → AnnotationQueuesStatsRes
```

***

<SourceLink />

### <kbd>method</kbd> `annotator_queue_items_progress_update`

```python theme={null}
annotator_queue_items_progress_update(
    req: AnnotatorQueueItemsProgressUpdateReq
) → AnnotatorQueueItemsProgressUpdateRes
```

***

<SourceLink />

### <kbd>method</kbd> `call_end`

```python theme={null}
call_end(req: CallEndReq) → CallEndRes
```

***

<SourceLink />

### <kbd>method</kbd> `call_end_v2`

```python theme={null}
call_end_v2(req: CallEndV2Req) → CallEndV2Res
```

Single call end endpoint (v2).

This endpoint is used for eager ops that need their end sent separately.

***

<SourceLink />

### <kbd>method</kbd> `call_read`

```python theme={null}
call_read(req: CallReadReq) → CallReadRes
```

***

<SourceLink />

### <kbd>method</kbd> `call_start`

```python theme={null}
call_start(req: CallStartReq) → CallStartRes
```

***

<SourceLink />

### <kbd>method</kbd> `call_start_batch`

```python theme={null}
call_start_batch(req: CallCreateBatchReq) → CallCreateBatchRes
```

***

<SourceLink />

### <kbd>method</kbd> `call_start_v2`

```python theme={null}
call_start_v2(req: CallStartV2Req) → CallStartV2Res
```

Single call start endpoint (v2).

This endpoint is used for eager ops that need their start visible immediately.

***

<SourceLink />

### <kbd>method</kbd> `call_update`

```python theme={null}
call_update(req: CallUpdateReq) → CallUpdateRes
```

***

<SourceLink />

### <kbd>method</kbd> `calls_complete`

```python theme={null}
calls_complete(req: CallsUpsertCompleteReq) → CallsUpsertCompleteRes
```

Batch complete calls endpoint (v2).

This endpoint is used when use\_calls\_complete is enabled to send complete calls (with both start and end information) in batches.

***

<SourceLink />

### <kbd>method</kbd> `calls_delete`

```python theme={null}
calls_delete(req: CallsDeleteReq) → CallsDeleteRes
```

***

<SourceLink />

### <kbd>method</kbd> `calls_query`

```python theme={null}
calls_query(req: CallsQueryReq) → CallsQueryRes
```

***

<SourceLink />

### <kbd>method</kbd> `calls_query_stats`

```python theme={null}
calls_query_stats(req: CallsQueryStatsReq) → CallsQueryStatsRes
```

***

<SourceLink />

### <kbd>method</kbd> `calls_query_stream`

```python theme={null}
calls_query_stream(req: CallsQueryReq) → Iterator[CallSchema]
```

***

<SourceLink />

### <kbd>method</kbd> `calls_usage`

```python theme={null}
calls_usage(req: CallsUsageReq) → CallsUsageRes
```

***

<SourceLink />

### <kbd>method</kbd> `completions_create`

```python theme={null}
completions_create(req: CompletionsCreateReq) → CompletionsCreateRes
```

***

<SourceLink />

### <kbd>method</kbd> `completions_create_stream`

```python theme={null}
completions_create_stream(req: CompletionsCreateReq) → Iterator[dict[str, Any]]
```

***

<SourceLink />

### <kbd>method</kbd> `cost_create`

```python theme={null}
cost_create(req: CostCreateReq) → CostCreateRes
```

***

<SourceLink />

### <kbd>method</kbd> `cost_purge`

```python theme={null}
cost_purge(req: CostPurgeReq) → CostPurgeRes
```

***

<SourceLink />

### <kbd>method</kbd> `cost_query`

```python theme={null}
cost_query(req: CostQueryReq) → CostQueryRes
```

***

<SourceLink />

### <kbd>method</kbd> `dataset_create`

```python theme={null}
dataset_create(req: DatasetCreateReq) → DatasetCreateRes
```

***

<SourceLink />

### <kbd>method</kbd> `dataset_delete`

```python theme={null}
dataset_delete(req: DatasetDeleteReq) → DatasetDeleteRes
```

***

<SourceLink />

### <kbd>method</kbd> `dataset_list`

```python theme={null}
dataset_list(req: DatasetListReq) → Iterator[DatasetReadRes]
```

***

<SourceLink />

### <kbd>method</kbd> `dataset_read`

```python theme={null}
dataset_read(req: DatasetReadReq) → DatasetReadRes
```

***

<SourceLink />

### <kbd>method</kbd> `delete`

```python theme={null}
delete(url: str, *args: Any, **kwargs: Any) → Response
```

***

<SourceLink />

### <kbd>method</kbd> `ensure_project_exists`

```python theme={null}
ensure_project_exists(entity: str, project: str) → EnsureProjectExistsRes
```

***

<SourceLink />

### <kbd>method</kbd> `evaluate_model`

```python theme={null}
evaluate_model(req: EvaluateModelReq) → EvaluateModelRes
```

***

<SourceLink />

### <kbd>method</kbd> `evaluation_create`

```python theme={null}
evaluation_create(req: EvaluationCreateReq) → EvaluationCreateRes
```

***

<SourceLink />

### <kbd>method</kbd> `evaluation_delete`

```python theme={null}
evaluation_delete(req: EvaluationDeleteReq) → EvaluationDeleteRes
```

***

<SourceLink />

### <kbd>method</kbd> `evaluation_list`

```python theme={null}
evaluation_list(req: EvaluationListReq) → Iterator[EvaluationReadRes]
```

***

<SourceLink />

### <kbd>method</kbd> `evaluation_read`

```python theme={null}
evaluation_read(req: EvaluationReadReq) → EvaluationReadRes
```

***

<SourceLink />

### <kbd>method</kbd> `evaluation_run_create`

```python theme={null}
evaluation_run_create(req: EvaluationRunCreateReq) → EvaluationRunCreateRes
```

***

<SourceLink />

### <kbd>method</kbd> `evaluation_run_delete`

```python theme={null}
evaluation_run_delete(req: EvaluationRunDeleteReq) → EvaluationRunDeleteRes
```

***

<SourceLink />

### <kbd>method</kbd> `evaluation_run_finish`

```python theme={null}
evaluation_run_finish(req: EvaluationRunFinishReq) → EvaluationRunFinishRes
```

***

<SourceLink />

### <kbd>method</kbd> `evaluation_run_list`

```python theme={null}
evaluation_run_list(req: EvaluationRunListReq) → Iterator[EvaluationRunReadRes]
```

***

<SourceLink />

### <kbd>method</kbd> `evaluation_run_read`

```python theme={null}
evaluation_run_read(req: EvaluationRunReadReq) → EvaluationRunReadRes
```

***

<SourceLink />

### <kbd>method</kbd> `evaluation_status`

```python theme={null}
evaluation_status(req: EvaluationStatusReq) → EvaluationStatusRes
```

***

<SourceLink />

### <kbd>method</kbd> `feedback_create`

```python theme={null}
feedback_create(req: FeedbackCreateReq) → FeedbackCreateRes
```

***

<SourceLink />

### <kbd>method</kbd> `feedback_create_batch`

```python theme={null}
feedback_create_batch(req: FeedbackCreateBatchReq) → FeedbackCreateBatchRes
```

***

<SourceLink />

### <kbd>method</kbd> `feedback_payload_schema`

```python theme={null}
feedback_payload_schema(
    req: FeedbackPayloadSchemaReq
) → FeedbackPayloadSchemaRes
```

***

<SourceLink />

### <kbd>method</kbd> `feedback_purge`

```python theme={null}
feedback_purge(req: FeedbackPurgeReq) → FeedbackPurgeRes
```

***

<SourceLink />

### <kbd>method</kbd> `feedback_query`

```python theme={null}
feedback_query(req: FeedbackQueryReq) → FeedbackQueryRes
```

***

<SourceLink />

### <kbd>method</kbd> `feedback_replace`

```python theme={null}
feedback_replace(req: FeedbackReplaceReq) → FeedbackReplaceRes
```

***

<SourceLink />

### <kbd>method</kbd> `feedback_stats`

```python theme={null}
feedback_stats(req: FeedbackStatsReq) → FeedbackStatsRes
```

***

<SourceLink />

### <kbd>method</kbd> `file_content_read`

```python theme={null}
file_content_read(req: FileContentReadReq) → FileContentReadRes
```

***

<SourceLink />

### <kbd>method</kbd> `file_create`

```python theme={null}
file_create(req: FileCreateReq) → FileCreateRes
```

***

<SourceLink />

### <kbd>method</kbd> `files_stats`

```python theme={null}
files_stats(req: FilesStatsReq) → FilesStatsRes
```

***

<SourceLink />

### <kbd>classmethod</kbd> `from_env`

```python theme={null}
from_env(should_batch: bool = False) → Self
```

***

<SourceLink />

### <kbd>method</kbd> `get`

```python theme={null}
get(url: str, *args: Any, **kwargs: Any) → Response
```

***

<SourceLink />

### <kbd>method</kbd> `get_call_processor`

```python theme={null}
get_call_processor() → AsyncBatchProcessor | CallBatchProcessor | None
```

Custom method not defined on the formal TraceServerInterface to expose the underlying call processor. Should be formalized in a client-side interface.

***

<SourceLink />

### <kbd>method</kbd> `get_feedback_processor`

```python theme={null}
get_feedback_processor() → AsyncBatchProcessor | None
```

Custom method not defined on the formal TraceServerInterface to expose the underlying feedback processor. Should be formalized in a client-side interface.

***

<SourceLink />

### <kbd>method</kbd> `image_create`

```python theme={null}
image_create(req: ImageGenerationCreateReq) → ImageGenerationCreateRes
```

***

<SourceLink />

### <kbd>method</kbd> `model_create`

```python theme={null}
model_create(req: ModelCreateReq) → ModelCreateRes
```

***

<SourceLink />

### <kbd>method</kbd> `model_delete`

```python theme={null}
model_delete(req: ModelDeleteReq) → ModelDeleteRes
```

***

<SourceLink />

### <kbd>method</kbd> `model_list`

```python theme={null}
model_list(req: ModelListReq) → Iterator[ModelReadRes]
```

***

<SourceLink />

### <kbd>method</kbd> `model_read`

```python theme={null}
model_read(req: ModelReadReq) → ModelReadRes
```

***

<SourceLink />

### <kbd>method</kbd> `obj_add_tags`

```python theme={null}
obj_add_tags(req: ObjAddTagsReq) → ObjAddTagsRes
```

***

<SourceLink />

### <kbd>method</kbd> `obj_create`

```python theme={null}
obj_create(req: ObjCreateReq) → ObjCreateRes
```

***

<SourceLink />

### <kbd>method</kbd> `obj_delete`

```python theme={null}
obj_delete(req: ObjDeleteReq) → ObjDeleteRes
```

***

<SourceLink />

### <kbd>method</kbd> `obj_read`

```python theme={null}
obj_read(req: ObjReadReq) → ObjReadRes
```

***

<SourceLink />

### <kbd>method</kbd> `obj_remove_aliases`

```python theme={null}
obj_remove_aliases(req: ObjRemoveAliasesReq) → ObjRemoveAliasesRes
```

***

<SourceLink />

### <kbd>method</kbd> `obj_remove_tags`

```python theme={null}
obj_remove_tags(req: ObjRemoveTagsReq) → ObjRemoveTagsRes
```

***

<SourceLink />

### <kbd>method</kbd> `obj_set_aliases`

```python theme={null}
obj_set_aliases(req: ObjSetAliasesReq) → ObjSetAliasesRes
```

***

<SourceLink />

### <kbd>method</kbd> `objs_query`

```python theme={null}
objs_query(req: ObjQueryReq) → ObjQueryRes
```

***

<SourceLink />

### <kbd>method</kbd> `op_create`

```python theme={null}
op_create(req: OpCreateReq) → OpCreateRes
```

***

<SourceLink />

### <kbd>method</kbd> `op_delete`

```python theme={null}
op_delete(req: OpDeleteReq) → OpDeleteRes
```

***

<SourceLink />

### <kbd>method</kbd> `op_list`

```python theme={null}
op_list(req: OpListReq) → Iterator[OpReadRes]
```

***

<SourceLink />

### <kbd>method</kbd> `op_read`

```python theme={null}
op_read(req: OpReadReq) → OpReadRes
```

***

<SourceLink />

### <kbd>method</kbd> `otel_export`

```python theme={null}
otel_export(req: OTelExportReq) → OTelExportRes
```

***

<SourceLink />

### <kbd>method</kbd> `post`

```python theme={null}
post(url: str, *args: Any, **kwargs: Any) → Response
```

***

<SourceLink />

### <kbd>method</kbd> `prediction_create`

```python theme={null}
prediction_create(req: PredictionCreateReq) → PredictionCreateRes
```

***

<SourceLink />

### <kbd>method</kbd> `prediction_delete`

```python theme={null}
prediction_delete(req: PredictionDeleteReq) → PredictionDeleteRes
```

***

<SourceLink />

### <kbd>method</kbd> `prediction_finish`

```python theme={null}
prediction_finish(req: PredictionFinishReq) → PredictionFinishRes
```

***

<SourceLink />

### <kbd>method</kbd> `prediction_list`

```python theme={null}
prediction_list(req: PredictionListReq) → Iterator[PredictionReadRes]
```

***

<SourceLink />

### <kbd>method</kbd> `prediction_read`

```python theme={null}
prediction_read(req: PredictionReadReq) → PredictionReadRes
```

***

<SourceLink />

### <kbd>method</kbd> `project_stats`

```python theme={null}
project_stats(req: ProjectStatsReq) → ProjectStatsRes
```

***

<SourceLink />

### <kbd>method</kbd> `project_ttl_settings_read`

```python theme={null}
project_ttl_settings_read(
    req: ProjectTTLSettingsReadReq
) → ProjectTTLSettingsReadRes
```

***

<SourceLink />

### <kbd>method</kbd> `project_ttl_settings_update`

```python theme={null}
project_ttl_settings_update(
    req: ProjectTTLSettingsUpdateReq
) → ProjectTTLSettingsUpdateRes
```

***

<SourceLink />

### <kbd>method</kbd> `projects_info`

```python theme={null}
projects_info(req: ProjectsInfoReq) → list[ProjectsInfoRes]
```

***

<SourceLink />

### <kbd>method</kbd> `put`

```python theme={null}
put(url: str, *args: Any, **kwargs: Any) → Response
```

***

<SourceLink />

### <kbd>method</kbd> `refs_read_batch`

```python theme={null}
refs_read_batch(req: RefsReadBatchReq) → RefsReadBatchRes
```

***

<SourceLink />

### <kbd>method</kbd> `rescore`

```python theme={null}
rescore(req: RescoreReq) → RescoreRes
```

***

<SourceLink />

### <kbd>method</kbd> `score_create`

```python theme={null}
score_create(req: ScoreCreateReq) → ScoreCreateRes
```

***

<SourceLink />

### <kbd>method</kbd> `score_delete`

```python theme={null}
score_delete(req: ScoreDeleteReq) → ScoreDeleteRes
```

***

<SourceLink />

### <kbd>method</kbd> `score_list`

```python theme={null}
score_list(req: ScoreListReq) → Iterator[ScoreReadRes]
```

***

<SourceLink />

### <kbd>method</kbd> `score_read`

```python theme={null}
score_read(req: ScoreReadReq) → ScoreReadRes
```

***

<SourceLink />

### <kbd>method</kbd> `scorer_create`

```python theme={null}
scorer_create(req: ScorerCreateReq) → ScorerCreateRes
```

***

<SourceLink />

### <kbd>method</kbd> `scorer_delete`

```python theme={null}
scorer_delete(req: ScorerDeleteReq) → ScorerDeleteRes
```

***

<SourceLink />

### <kbd>method</kbd> `scorer_list`

```python theme={null}
scorer_list(req: ScorerListReq) → Iterator[ScorerReadRes]
```

***

<SourceLink />

### <kbd>method</kbd> `scorer_read`

```python theme={null}
scorer_read(req: ScorerReadReq) → ScorerReadRes
```

***

<SourceLink />

### <kbd>method</kbd> `server_info`

```python theme={null}
server_info() → ServerInfoRes
```

***

<SourceLink />

### <kbd>method</kbd> `set_auth`

```python theme={null}
set_auth(auth: tuple[str, str]) → None
```

***

<SourceLink />

### <kbd>method</kbd> `table_create`

```python theme={null}
table_create(req: TableCreateReq) → TableCreateRes
```

***

<SourceLink />

### <kbd>method</kbd> `table_create_from_digests`

```python theme={null}
table_create_from_digests(
    req: TableCreateFromDigestsReq
) → TableCreateFromDigestsRes
```

Create a table by specifying row digests instead of actual rows.

***

<SourceLink />

### <kbd>method</kbd> `table_query`

```python theme={null}
table_query(req: TableQueryReq) → TableQueryRes
```

***

<SourceLink />

### <kbd>method</kbd> `table_query_stats`

```python theme={null}
table_query_stats(req: TableQueryStatsReq) → TableQueryStatsRes
```

***

<SourceLink />

### <kbd>method</kbd> `table_query_stats_batch`

```python theme={null}
table_query_stats_batch(req: TableQueryStatsReq) → TableQueryStatsRes
```

***

<SourceLink />

### <kbd>method</kbd> `table_query_stream`

```python theme={null}
table_query_stream(req: TableQueryReq) → Iterator[TableRowSchema]
```

***

<SourceLink />

### <kbd>method</kbd> `table_update`

```python theme={null}
table_update(req: TableUpdateReq) → TableUpdateRes
```

Similar to `calls/batch_upsert`, we can dynamically adjust the payload size due to the property that table updates can be decomposed into a series of updates.

***

<SourceLink />

### <kbd>method</kbd> `tags_list`

```python theme={null}
tags_list(req: TagsListReq) → TagsListRes
```

***

<SourceLink />

### <kbd>method</kbd> `threads_query_stream`

```python theme={null}
threads_query_stream(req: ThreadsQueryReq) → Iterator[ThreadSchema]
```

***

<SourceLink />

### <kbd>method</kbd> `trace_usage`

```python theme={null}
trace_usage(req: TraceUsageReq) → TraceUsageRes
```


# weave
Source: https://docs.wandb.ai/weave/reference/typescript-sdk

TypeScript SDK reference

weave

## Table of contents

### Classes

* [Dataset](./typescript-sdk/classes/dataset)
* [Evaluation](./typescript-sdk/classes/evaluation)
* [EvaluationLogger](./typescript-sdk/classes/evaluationlogger)
* [LLM](./typescript-sdk/classes/llm)
* [MessagesPrompt](./typescript-sdk/classes/messagesprompt)
* [ObjectRef](./typescript-sdk/classes/objectref)
* [ScoreLogger](./typescript-sdk/classes/scorelogger)
* [Session](./typescript-sdk/classes/session)
* [StringPrompt](./typescript-sdk/classes/stringprompt)
* [SubAgent](./typescript-sdk/classes/subagent)
* [Tool](./typescript-sdk/classes/tool)
* [Turn](./typescript-sdk/classes/turn)
* [WeaveClient](./typescript-sdk/classes/weaveclient)
* [WeaveObject](./typescript-sdk/classes/weaveobject)

### Interfaces

* [CallSchema](./typescript-sdk/interfaces/callschema)
* [CallsFilter](./typescript-sdk/interfaces/callsfilter)
* [GetCallsOptions](./typescript-sdk/interfaces/getcallsoptions)
* [LLMInit](./typescript-sdk/interfaces/llminit)
* [Message](./typescript-sdk/interfaces/message)
* [Query](./typescript-sdk/interfaces/query)
* [Reasoning](./typescript-sdk/interfaces/reasoning)
* [SessionInit](./typescript-sdk/interfaces/sessioninit)
* [SortBy](./typescript-sdk/interfaces/sortby)
* [SubAgentInit](./typescript-sdk/interfaces/subagentinit)
* [ToolInit](./typescript-sdk/interfaces/toolinit)
* [TurnInit](./typescript-sdk/interfaces/turninit)
* [Usage](./typescript-sdk/interfaces/usage)
* [WeaveAudio](./typescript-sdk/interfaces/weaveaudio)
* [WeaveImage](./typescript-sdk/interfaces/weaveimage)

### Type Aliases

* [MessagePart](./typescript-sdk/type-aliases/messagepart)
* [Modality](./typescript-sdk/type-aliases/modality)
* [OpDecorator](./typescript-sdk/type-aliases/opdecorator)
* [Op](./typescript-sdk/type-aliases/op)
* [Role](./typescript-sdk/type-aliases/role)

### Functions

* [createOpenAIAgentsTracingProcessor](./typescript-sdk/functions/createopenaiagentstracingprocessor)
* [createOtelExtension](./typescript-sdk/functions/createotelextension)
* [endLLM](./typescript-sdk/functions/endllm)
* [endSession](./typescript-sdk/functions/endsession)
* [endTurn](./typescript-sdk/functions/endturn)
* [flushOTel](./typescript-sdk/functions/flushotel)
* [getCurrentLLM](./typescript-sdk/functions/getcurrentllm)
* [getCurrentSession](./typescript-sdk/functions/getcurrentsession)
* [getCurrentTurn](./typescript-sdk/functions/getcurrentturn)
* [init](./typescript-sdk/functions/init)
* [instrumentOpenAIAgents](./typescript-sdk/functions/instrumentopenaiagents)
* [login](./typescript-sdk/functions/login)
* [op](./typescript-sdk/functions/op)
* [patchRealtimeSession](./typescript-sdk/functions/patchrealtimesession)
* [requireCurrentCallStackEntry](./typescript-sdk/functions/requirecurrentcallstackentry)
* [requireCurrentChildSummary](./typescript-sdk/functions/requirecurrentchildsummary)
* [runIsolated](./typescript-sdk/functions/runisolated)
* [startLLM](./typescript-sdk/functions/startllm)
* [startSession](./typescript-sdk/functions/startsession)
* [startSubagent](./typescript-sdk/functions/startsubagent)
* [startTool](./typescript-sdk/functions/starttool)
* [startTurn](./typescript-sdk/functions/startturn)
* [weaveAudio](./typescript-sdk/functions/weaveaudio)
* [weaveImage](./typescript-sdk/functions/weaveimage)
* [withAttributes](./typescript-sdk/functions/withattributes)
* [wrapGoogleGenAI](./typescript-sdk/functions/wrapgooglegenai)
* [wrapOpenAI](./typescript-sdk/functions/wrapopenai)

## Type Aliases


# Class: Dataset<R>
Source: https://docs.wandb.ai/weave/reference/typescript-sdk/classes/dataset

TypeScript SDK reference

[weave](../) / Dataset

Dataset object with easy saving and automatic versioning

`Example`

```ts theme={null}
// Create a dataset
const dataset = new Dataset({
  id: 'grammar-dataset',
  rows: [
    { id: '0', sentence: "He no likes ice cream.", correction: "He doesn't like ice cream." },
    { id: '1', sentence: "She goed to the store.", correction: "She went to the store." },
    { id: '2', sentence: "They plays video games all day.", correction: "They play video games all day." }
  ]
})

// Access a specific example
const exampleLabel = dataset.getRow(2).sentence;

// Save the dataset
const ref = await dataset.save()
```

## Type parameters

| Name | Type                 |
| :--- | :------------------- |
| `R`  | extends `DatasetRow` |

## Hierarchy

* [`WeaveObject`](./weaveobject)

  ↳ `Dataset`

## Table of contents

### Constructors

* [constructor](./dataset#constructor)

### Properties

* [\_\_savedRef](./dataset#__savedref)
* [rows](./dataset#rows)

### Accessors

* [description](./dataset#description)
* [length](./dataset#length)
* [name](./dataset#name)

### Methods

* [\[asyncIterator\]](./dataset#\[asynciterator])
* [getRow](./dataset#getrow)
* [save](./dataset#save)
* [saveAttrs](./dataset#saveattrs)

## Constructors

### constructor

• **new Dataset**\<`R`>(`parameters`): [`Dataset`](./dataset)\<`R`>

#### Type parameters

| Name | Type                 |
| :--- | :------------------- |
| `R`  | extends `DatasetRow` |

#### Parameters

| Name         | Type                      |
| :----------- | :------------------------ |
| `parameters` | `DatasetParameters`\<`R`> |

#### Returns

[`Dataset`](./dataset)\<`R`>

#### Overrides

[WeaveObject](./weaveobject).[constructor](./weaveobject#constructor)

#### Defined in

[dataset.ts:51](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/dataset.ts#L51)

## Properties

### \_\_savedRef

• `Optional` **\_\_savedRef**: [`ObjectRef`](./objectref) | `Promise`\<[`ObjectRef`](./objectref)>

#### Inherited from

[WeaveObject](./weaveobject).[\_\_savedRef](./weaveobject#__savedref)

#### Defined in

[weaveObject.ts:73](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/weaveObject.ts#L73)

***

### rows

• **rows**: `Table`\<`R`>

#### Defined in

[dataset.ts:49](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/dataset.ts#L49)

## Accessors

### description

• `get` **description**(): `undefined` | `string`

#### Returns

`undefined` | `string`

#### Inherited from

WeaveObject.description

#### Defined in

[weaveObject.ts:100](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/weaveObject.ts#L100)

***

### length

• `get` **length**(): `number`

#### Returns

`number`

#### Defined in

[dataset.ts:64](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/dataset.ts#L64)

***

### name

• `get` **name**(): `string`

#### Returns

`string`

#### Inherited from

WeaveObject.name

#### Defined in

[weaveObject.ts:96](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/weaveObject.ts#L96)

## Methods

### \[asyncIterator]

▸ **\[asyncIterator]**(): `AsyncIterator`\<`any`, `any`, `undefined`>

#### Returns

`AsyncIterator`\<`any`, `any`, `undefined`>

#### Defined in

[dataset.ts:68](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/dataset.ts#L68)

***

### getRow

▸ **getRow**(`index`): `R`

#### Parameters

| Name    | Type     |
| :------ | :------- |
| `index` | `number` |

#### Returns

`R`

#### Defined in

[dataset.ts:74](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/dataset.ts#L74)

***

### save

▸ **save**(): `Promise`\<[`ObjectRef`](./objectref)>

#### Returns

`Promise`\<[`ObjectRef`](./objectref)>

#### Defined in

[dataset.ts:60](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/dataset.ts#L60)

***

### saveAttrs

▸ **saveAttrs**(): `Object`

#### Returns

`Object`

#### Inherited from

[WeaveObject](./weaveobject).[saveAttrs](./weaveobject#saveattrs)

#### Defined in

[weaveObject.ts:77](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/weaveObject.ts#L77)


# Class: Evaluation<R, E, M>
Source: https://docs.wandb.ai/weave/reference/typescript-sdk/classes/evaluation

TypeScript SDK reference

[weave](../) / Evaluation

Sets up an evaluation which includes a set of scorers and a dataset.

Calling evaluation.evaluate(model) will pass in rows form a dataset into a model matching
the names of the columns of the dataset to the argument names in model.predict.

Then it will call all of the scorers and save the results in weave.

`Example`

```ts theme={null}
// Collect your examples into a dataset
const dataset = new weave.Dataset({
  id: 'my-dataset',
  rows: [
    { question: 'What is the capital of France?', expected: 'Paris' },
    { question: 'Who wrote "To Kill a Mockingbird"?', expected: 'Harper Lee' },
    { question: 'What is the square root of 64?', expected: '8' },
  ],
});

// Define any custom scoring function
const scoringFunction = weave.op(function isEqual({ modelOutput, datasetRow }) {
  return modelOutput == datasetRow.expected;
});

// Define the function to evaluate
const model = weave.op(async function alwaysParisModel({ question }) {
  return 'Paris';
});

// Start evaluating
const evaluation = new weave.Evaluation({
  id: 'my-evaluation',
  dataset: dataset,
  scorers: [scoringFunction],
});

const results = await evaluation.evaluate({ model });
```

## Type parameters

| Name | Type                 |
| :--- | :------------------- |
| `R`  | extends `DatasetRow` |
| `E`  | extends `DatasetRow` |
| `M`  | `M`                  |

## Hierarchy

* [`WeaveObject`](./weaveobject)

  ↳ `Evaluation`

## Table of contents

### Constructors

* [constructor](./evaluation#constructor)

### Properties

* [\_\_savedRef](./evaluation#__savedref)

### Accessors

* [description](./evaluation#description)
* [name](./evaluation#name)

### Methods

* [evaluate](./evaluation#evaluate)
* [predictAndScore](./evaluation#predictandscore)
* [saveAttrs](./evaluation#saveattrs)

## Constructors

### constructor

• **new Evaluation**\<`R`, `E`, `M`>(`parameters`): [`Evaluation`](./evaluation)\<`R`, `E`, `M`>

#### Type parameters

| Name | Type                 |
| :--- | :------------------- |
| `R`  | extends `DatasetRow` |
| `E`  | extends `DatasetRow` |
| `M`  | `M`                  |

#### Parameters

| Name         | Type                                   |
| :----------- | :------------------------------------- |
| `parameters` | `EvaluationParameters`\<`R`, `E`, `M`> |

#### Returns

[`Evaluation`](./evaluation)\<`R`, `E`, `M`>

#### Overrides

[WeaveObject](./weaveobject).[constructor](./weaveobject#constructor)

#### Defined in

[evaluation.ts:148](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/evaluation.ts#L148)

## Properties

### \_\_savedRef

• `Optional` **\_\_savedRef**: [`ObjectRef`](./objectref) | `Promise`\<[`ObjectRef`](./objectref)>

#### Inherited from

[WeaveObject](./weaveobject).[\_\_savedRef](./weaveobject#__savedref)

#### Defined in

[weaveObject.ts:73](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/weaveObject.ts#L73)

## Accessors

### description

• `get` **description**(): `undefined` | `string`

#### Returns

`undefined` | `string`

#### Inherited from

WeaveObject.description

#### Defined in

[weaveObject.ts:100](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/weaveObject.ts#L100)

***

### name

• `get` **name**(): `string`

#### Returns

`string`

#### Inherited from

WeaveObject.name

#### Defined in

[weaveObject.ts:96](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/weaveObject.ts#L96)

## Methods

### evaluate

▸ **evaluate**(`«destructured»`): `Promise`\<`Record`\<`string`, `any`>>

#### Parameters

| Name                | Type                                                                         | Default value |
| :------------------ | :--------------------------------------------------------------------------- | :------------ |
| `«destructured»`    | `Object`                                                                     | `undefined`   |
| › `maxConcurrency?` | `number`                                                                     | `5`           |
| › `model`           | `WeaveCallable`\<(...`args`: \[\{ `datasetRow`: `R`  }]) => `Promise`\<`M`>> | `undefined`   |
| › `nTrials?`        | `number`                                                                     | `1`           |

#### Returns

`Promise`\<`Record`\<`string`, `any`>>

#### Defined in

[evaluation.ts:163](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/evaluation.ts#L163)

***

### predictAndScore

▸ **predictAndScore**(`«destructured»`): `Promise`\<\{ `model_latency`: `number` = modelLatency; `model_output`: `any` = modelOutput; `model_success`: `boolean` = !modelError; `scores`: \{ `[key: string]`: `any`;  }  }>

#### Parameters

| Name               | Type                                                                         |
| :----------------- | :--------------------------------------------------------------------------- |
| `«destructured»`   | `Object`                                                                     |
| › `columnMapping?` | `ColumnMapping`\<`R`, `E`>                                                   |
| › `example`        | `R`                                                                          |
| › `model`          | `WeaveCallable`\<(...`args`: \[\{ `datasetRow`: `E`  }]) => `Promise`\<`M`>> |

#### Returns

`Promise`\<\{ `model_latency`: `number` = modelLatency; `model_output`: `any` = modelOutput; `model_success`: `boolean` = !modelError; `scores`: \{ `[key: string]`: `any`;  }  }>

#### Defined in

[evaluation.ts:231](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/evaluation.ts#L231)

***

### saveAttrs

▸ **saveAttrs**(): `Object`

#### Returns

`Object`

#### Inherited from

[WeaveObject](./weaveobject).[saveAttrs](./weaveobject#saveattrs)

#### Defined in

[weaveObject.ts:77](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/weaveObject.ts#L77)


# Class: EvaluationLogger
Source: https://docs.wandb.ai/weave/reference/typescript-sdk/classes/evaluationlogger

TypeScript SDK reference

[weave](../) / EvaluationLogger

EvaluationLogger enables incremental logging of predictions and scores.

Unlike the traditional Evaluation class which requires upfront dataset and batch processing,
EvaluationLogger allows you to log predictions as they happen, with flexible scoring.

`Example`

```ts theme={null}
const ev = new EvaluationLogger({name: 'my-eval', dataset: 'my-dataset'});

for (const example of streamingData) {
  const output = await myModel.predict(example);
  const pred = ev.logPrediction(example, output);

  if (shouldScore(output)) {
    pred.logScore("accuracy", calculateAccuracy(output));
  }
  pred.finish();
}

await ev.logSummary();
```

## Table of contents

### Constructors

* [constructor](./evaluationlogger#constructor)

### Methods

* [logPrediction](./evaluationlogger#logprediction)
* [logPredictionAsync](./evaluationlogger#logpredictionasync)
* [logSummary](./evaluationlogger#logsummary)

## Constructors

### constructor

• **new EvaluationLogger**(`options`): [`EvaluationLogger`](./evaluationlogger)

#### Parameters

| Name      | Type                      |
| :-------- | :------------------------ |
| `options` | `EvaluationLoggerOptions` |

#### Returns

[`EvaluationLogger`](./evaluationlogger)

#### Defined in

[evaluationLogger.ts:570](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/evaluationLogger.ts#L570)

## Methods

### logPrediction

▸ **logPrediction**(`inputs`, `output`): [`ScoreLogger`](./scorelogger)

Log a prediction with its input and output (synchronous version).
Creates a predict\_and\_score call (with child predict call).
Returns a ScoreLogger immediately for adding scores.

This method returns the ScoreLogger synchronously. Operations on the
ScoreLogger (logScore, finish) will be queued and executed when initialization completes.

#### Parameters

| Name     | Type                       |
| :------- | :------------------------- |
| `inputs` | `Record`\<`string`, `any`> |
| `output` | `any`                      |

#### Returns

[`ScoreLogger`](./scorelogger)

`Example`

```ts theme={null}
// Fire-and-forget style
const scoreLogger = evalLogger.logPrediction({input: 'test'}, 'output');
scoreLogger.logScore('accuracy', 0.95);
scoreLogger.finish();
await evalLogger.logSummary(); // Waits for everything
```

#### Defined in

[evaluationLogger.ts:657](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/evaluationLogger.ts#L657)

***

### logPredictionAsync

▸ **logPredictionAsync**(`inputs`, `output`): `Promise`\<[`ScoreLogger`](./scorelogger)>

Log a prediction with its input and output (async version).
Like logPrediction() but returns a Promise that resolves when
the prediction call is fully initialized.

Use this if you need to await the initialization before proceeding.

#### Parameters

| Name     | Type                       |
| :------- | :------------------------- |
| `inputs` | `Record`\<`string`, `any`> |
| `output` | `any`                      |

#### Returns

`Promise`\<[`ScoreLogger`](./scorelogger)>

`Example`

```ts theme={null}
// Awaitable style
const scoreLogger = await evalLogger.logPredictionAsync({input: 'test'}, 'output');
await scoreLogger.logScore('accuracy', 0.95);
await scoreLogger.finish();
```

#### Defined in

[evaluationLogger.ts:682](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/evaluationLogger.ts#L682)

***

### logSummary

▸ **logSummary**(`summary?`): `Promise`\<`void`>

Log a summary and finalize the evaluation.
Creates a summarize call and finishes the evaluate call.

This method can be called without await (fire-and-forget), but internally
it will wait for all pending operations to complete.

#### Parameters

| Name       | Type                       |
| :--------- | :------------------------- |
| `summary?` | `Record`\<`string`, `any`> |

#### Returns

`Promise`\<`void`>

#### Defined in

[evaluationLogger.ts:783](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/evaluationLogger.ts#L783)


# Class: LLM
Source: https://docs.wandb.ai/weave/reference/typescript-sdk/classes/llm

TypeScript SDK reference

[weave](../) / LLM

An LLM call. Emits a `chat` span with `gen_ai.*` attributes.

Created by `weave.startLLM()` (or `turn.startLLM()`) and terminated with
`end()`. Only one LLM may be active in an async context at a time; nest
tool/subagent calls under it via `startTool` / `startSubagent`.

Populate `inputMessages` / `outputMessages` / `usage` / `reasoning` directly,
or via the helper functions (`output`, `think`, `attachMedia`, `record`).

All recorded data is flushed to the span at `end()`.

`Example`

```ts theme={null}
const llm = weave.startLLM({model: 'gpt-4o-mini', providerName: 'openai'});
try {
  llm.inputMessages = [{role: 'user', content: prompt}];
  const resp = await openai.chat.completions.create({...});
  llm.output(resp.choices[0].message.content ?? '');
  llm.record({usage: {inputTokens: resp.usage?.prompt_tokens}});
} finally {
  llm.end();
}
```

## Table of contents

### Properties

* [inputMessages](./llm#inputmessages)
* [model](./llm#model)
* [outputMessages](./llm#outputmessages)
* [providerName](./llm#providername)
* [reasoning](./llm#reasoning)
* [usage](./llm#usage)

### Methods

* [attachMedia](./llm#attachmedia)
* [attachMediaUrl](./llm#attachmediaurl)
* [end](./llm#end)
* [output](./llm#output)
* [record](./llm#record)
* [startSubagent](./llm#startsubagent)
* [startTool](./llm#starttool)
* [think](./llm#think)
* [create](./llm#create)

## Properties

### inputMessages

• **inputMessages**: [`Message`](../interfaces/message)\[] = `[]`

Input messages sent to the model. Flushed to `gen_ai.input.messages` on
`end()`.

#### Defined in

[genai/llm.ts:78](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/genai/llm.ts#L78)

***

### model

• `Readonly` **model**: `string`

#### Defined in

[genai/llm.ts:98](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/genai/llm.ts#L98)

***

### outputMessages

• **outputMessages**: [`Message`](../interfaces/message)\[] = `[]`

Assistant messages returned by the model. Flushed to
`gen_ai.output.messages` on `end()`.

#### Defined in

[genai/llm.ts:83](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/genai/llm.ts#L83)

***

### providerName

• `Readonly` **providerName**: `string`

#### Defined in

[genai/llm.ts:99](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/genai/llm.ts#L99)

***

### reasoning

• `Optional` **reasoning**: [`Reasoning`](../interfaces/reasoning)

Chain-of-thought content. Folded into the last assistant message as a
ReasoningPart at serialization time.

#### Defined in

[genai/llm.ts:90](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/genai/llm.ts#L90)

***

### usage

• **usage**: [`Usage`](../interfaces/usage) = `{}`

Token counts and cache stats. Flushed to `gen_ai.usage.*` on `end()`.

#### Defined in

[genai/llm.ts:85](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/genai/llm.ts#L85)

## Methods

### attachMedia

▸ **attachMedia**(`opts`): `this`

Attach a media part to the last input message. Pick exactly one of
`content` (inline base64 bytes), `uri` (URI reference), or `fileId`
(pre-uploaded file id).

#### Parameters

| Name   | Type              |
| :----- | :---------------- |
| `opts` | `AttachMediaOpts` |

#### Returns

`this`

#### Defined in

[genai/llm.ts:168](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/genai/llm.ts#L168)

***

### attachMediaUrl

▸ **attachMediaUrl**(`url`, `opts`): `this`

Convenience for `attachMedia({uri, modality})`.

#### Parameters

| Name            | Type                       |
| :-------------- | :------------------------- |
| `url`           | `string`                   |
| `opts`          | `Object`                   |
| `opts.modality` | [`Modality`](../#modality) |

#### Returns

`this`

#### Defined in

[genai/llm.ts:186](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/genai/llm.ts#L186)

***

### end

▸ **end**(`opts?`): `void`

Flush accumulated state to the span and close it. Idempotent. Pass
`error` to mark the span as failed.

#### Parameters

| Name          | Type     |
| :------------ | :------- |
| `opts?`       | `Object` |
| `opts.error?` | `Error`  |

#### Returns

`void`

#### Defined in

[genai/llm.ts:246](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/genai/llm.ts#L246)

***

### output

▸ **output**(`content`): `this`

Append an assistant message to the response.

#### Parameters

| Name      | Type     |
| :-------- | :------- |
| `content` | `string` |

#### Returns

`this`

#### Defined in

[genai/llm.ts:141](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/genai/llm.ts#L141)

***

### record

▸ **record**(`opts`): `this`

Bulk-set any subset of the mutable fields. Replaces (does not merge).
Useful for assigning everything at once after a provider call returns.

#### Parameters

| Name   | Type            |
| :----- | :-------------- |
| `opts` | `LLMRecordOpts` |

#### Returns

`this`

#### Defined in

[genai/llm.ts:197](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/genai/llm.ts#L197)

***

### startSubagent

▸ **startSubagent**(`opts`): [`SubAgent`](./subagent)

Start a child SubAgent span nested under this LLM.

#### Parameters

| Name   | Type                                         |
| :----- | :------------------------------------------- |
| `opts` | [`SubAgentInit`](../interfaces/subagentinit) |

#### Returns

[`SubAgent`](./subagent)

#### Defined in

[genai/llm.ts:230](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/genai/llm.ts#L230)

***

### startTool

▸ **startTool**(`opts`): [`Tool`](./tool)

Start a child Tool span nested under this LLM.

#### Parameters

| Name   | Type                                 |
| :----- | :----------------------------------- |
| `opts` | [`ToolInit`](../interfaces/toolinit) |

#### Returns

[`Tool`](./tool)

#### Defined in

[genai/llm.ts:221](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/genai/llm.ts#L221)

***

### think

▸ **think**(`content`): `this`

Set or extend the model's reasoning/chain-of-thought content. Accumulates
into `this.reasoning.content`. Folded into the last assistant message as
a `ReasoningPart` at serialization time, matching the Python SDK's
on-the-wire shape.

#### Parameters

| Name      | Type     |
| :-------- | :------- |
| `content` | `string` |

#### Returns

`this`

#### Defined in

[genai/llm.ts:153](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/genai/llm.ts#L153)

***

### create

▸ **create**(`opts`): [`LLM`](./llm)

#### Parameters

| Name   | Type                                                    |
| :----- | :------------------------------------------------------ |
| `opts` | [`LLMInit`](../interfaces/llminit) & `ChildSpanContext` |

#### Returns

[`LLM`](./llm)

#### Defined in

[genai/llm.ts:102](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/genai/llm.ts#L102)


# Class: MessagesPrompt
Source: https://docs.wandb.ai/weave/reference/typescript-sdk/classes/messagesprompt

TypeScript SDK reference

[weave](../) / MessagesPrompt

## Hierarchy

* `Prompt`

  ↳ `MessagesPrompt`

## Table of contents

### Constructors

* [constructor](./messagesprompt#constructor)

### Properties

* [\_\_savedRef](./messagesprompt#__savedref)
* [messages](./messagesprompt#messages)

### Accessors

* [description](./messagesprompt#description)
* [name](./messagesprompt#name)

### Methods

* [format](./messagesprompt#format)
* [saveAttrs](./messagesprompt#saveattrs)
* [get](./messagesprompt#get)

## Constructors

### constructor

• **new MessagesPrompt**(`parameters`): [`MessagesPrompt`](./messagesprompt)

#### Parameters

| Name         | Type                       |
| :----------- | :------------------------- |
| `parameters` | `MessagesPromptParameters` |

#### Returns

[`MessagesPrompt`](./messagesprompt)

#### Overrides

Prompt.constructor

#### Defined in

[prompt.ts:40](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/prompt.ts#L40)

## Properties

### \_\_savedRef

• `Optional` **\_\_savedRef**: [`ObjectRef`](./objectref) | `Promise`\<[`ObjectRef`](./objectref)>

#### Inherited from

Prompt.\_\_savedRef

#### Defined in

[weaveObject.ts:73](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/weaveObject.ts#L73)

***

### messages

• **messages**: `Record`\<`string`, `any`>\[]

#### Defined in

[prompt.ts:38](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/prompt.ts#L38)

## Accessors

### description

• `get` **description**(): `undefined` | `string`

#### Returns

`undefined` | `string`

#### Inherited from

Prompt.description

#### Defined in

[weaveObject.ts:100](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/weaveObject.ts#L100)

***

### name

• `get` **name**(): `string`

#### Returns

`string`

#### Inherited from

Prompt.name

#### Defined in

[weaveObject.ts:96](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/weaveObject.ts#L96)

## Methods

### format

▸ **format**(`values?`): `Record`\<`string`, `any`>\[]

#### Parameters

| Name     | Type                       |
| :------- | :------------------------- |
| `values` | `Record`\<`string`, `any`> |

#### Returns

`Record`\<`string`, `any`>\[]

#### Defined in

[prompt.ts:67](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/prompt.ts#L67)

***

### saveAttrs

▸ **saveAttrs**(): `Object`

#### Returns

`Object`

#### Inherited from

Prompt.saveAttrs

#### Defined in

[weaveObject.ts:77](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/weaveObject.ts#L77)

***

### get

▸ **get**(`client`, `uri`): `Promise`\<[`MessagesPrompt`](./messagesprompt)>

#### Parameters

| Name     | Type                           |
| :------- | :----------------------------- |
| `client` | [`WeaveClient`](./weaveclient) |
| `uri`    | `string`                       |

#### Returns

`Promise`\<[`MessagesPrompt`](./messagesprompt)>

#### Defined in

[prompt.ts:73](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/prompt.ts#L73)


# Class: ObjectRef
Source: https://docs.wandb.ai/weave/reference/typescript-sdk/classes/objectref

TypeScript SDK reference

[weave](../) / ObjectRef

Represents a reference to a saved Weave object.

Generally, end users will not need to interact with this class directly.

An ObjectRef contains the project ID, object ID, and digest that uniquely identify
a saved object in Weave's storage system.

`Example`

```ts theme={null}
const ref = new ObjectRef('my-project', 'abc123', 'def456');
const uri = ref.uri(); // weave:///my-project/object/abc123:def456
```

## Table of contents

### Constructors

* [constructor](./objectref#constructor)

### Properties

* [digest](./objectref#digest)
* [objectId](./objectref#objectid)
* [projectId](./objectref#projectid)

### Methods

* [get](./objectref#get)
* [ui\_url](./objectref#ui_url)
* [uri](./objectref#uri)
* [fromUri](./objectref#fromuri)

## Constructors

### constructor

• **new ObjectRef**(`projectId`, `objectId`, `digest`): [`ObjectRef`](./objectref)

#### Parameters

| Name        | Type     |
| :---------- | :------- |
| `projectId` | `string` |
| `objectId`  | `string` |
| `digest`    | `string` |

#### Returns

[`ObjectRef`](./objectref)

#### Defined in

[weaveObject.ts:26](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/weaveObject.ts#L26)

## Properties

### digest

• **digest**: `string`

#### Defined in

[weaveObject.ts:29](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/weaveObject.ts#L29)

***

### objectId

• **objectId**: `string`

#### Defined in

[weaveObject.ts:28](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/weaveObject.ts#L28)

***

### projectId

• **projectId**: `string`

#### Defined in

[weaveObject.ts:27](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/weaveObject.ts#L27)

## Methods

### get

▸ **get**(): `Promise`\<`any`>

#### Returns

`Promise`\<`any`>

#### Defined in

[weaveObject.ts:66](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/weaveObject.ts#L66)

***

### ui\_url

▸ **ui\_url**(): `string`

#### Returns

`string`

#### Defined in

[weaveObject.ts:61](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/weaveObject.ts#L61)

***

### uri

▸ **uri**(): `string`

#### Returns

`string`

#### Defined in

[weaveObject.ts:57](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/weaveObject.ts#L57)

***

### fromUri

▸ **fromUri**(`uri`): [`ObjectRef`](./objectref)

Creates an ObjectRef from a Weave URI string.

#### Parameters

| Name  | Type     | Description                                                           |
| :---- | :------- | :-------------------------------------------------------------------- |
| `uri` | `string` | A Weave URI in the format: weave:///entity/project/object/name:digest |

#### Returns

[`ObjectRef`](./objectref)

A new ObjectRef instance

`Throws`

Error if the URI format is invalid or not an object ref

`Example`

```ts theme={null}
const ref = ObjectRef.fromUri('weave:///my-entity/my-project/object/my-dataset:abc123');
```

#### Defined in

[weaveObject.ts:44](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/weaveObject.ts#L44)


# Class: ScoreLogger
Source: https://docs.wandb.ai/weave/reference/typescript-sdk/classes/scorelogger

TypeScript SDK reference

[weave](../) / ScoreLogger

ScoreLogger manages scoring for a single prediction.
Returned from EvaluationLogger.logPrediction().

`Example`

```ts theme={null}
const pred = await ev.logPrediction(example, output);
await pred.logScore("accuracy", 0.95);
await pred.logScore("relevance", 0.8);
await pred.finish(); // Finalizes the prediction
```

## Table of contents

### Constructors

* [constructor](./scorelogger#constructor)

### Accessors

* [isFinishCalled](./scorelogger#isfinishcalled)

### Methods

* [finish](./scorelogger#finish)
* [logScore](./scorelogger#logscore)

## Constructors

### constructor

• **new ScoreLogger**(`evalLogger`): [`ScoreLogger`](./scorelogger)

#### Parameters

| Name         | Type                                     |
| :----------- | :--------------------------------------- |
| `evalLogger` | [`EvaluationLogger`](./evaluationlogger) |

#### Returns

[`ScoreLogger`](./scorelogger)

#### Defined in

[evaluationLogger.ts:335](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/evaluationLogger.ts#L335)

## Accessors

### isFinishCalled

• `get` **isFinishCalled**(): `boolean`

Check if finish() has been called.
Used by EvaluationLogger to detect unfinished predictions.

#### Returns

`boolean`

#### Defined in

[evaluationLogger.ts:365](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/evaluationLogger.ts#L365)

## Methods

### finish

▸ **finish**(): `Promise`\<`void`>

Finish the scoring process for the prediction.
Finalizes the predict\_and\_score call with accumulated scores.
Updates incremental aggregates and frees memory.

#### Returns

`Promise`\<`void`>

#### Defined in

[evaluationLogger.ts:467](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/evaluationLogger.ts#L467)

***

### logScore

▸ **logScore**(`scorerName`, `score`): `Promise`\<`void`>

Log a score for this prediction.
Creates a scorer call as a child of predict\_and\_score.

#### Parameters

| Name         | Type     | Description                                        |
| :----------- | :------- | :------------------------------------------------- |
| `scorerName` | `string` | Name of the scorer (e.g., "accuracy", "f1\_score") |
| `score`      | `any`    | The score value                                    |

#### Returns

`Promise`\<`void`>

#### Defined in

[evaluationLogger.ts:376](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/evaluationLogger.ts#L376)


# Class: Session
Source: https://docs.wandb.ai/weave/reference/typescript-sdk/classes/session

TypeScript SDK reference

[weave](../) / Session

A Session groups Turns under a single `gen_ai.conversation.id`. It is not
itself an OTel span — children stamp the conversation id onto theirs.

## Table of contents

### Properties

* [agentName](./session#agentname)
* [model](./session#model)
* [sessionId](./session#sessionid)

### Methods

* [end](./session#end)
* [startTurn](./session#startturn)
* [create](./session#create)

## Properties

### agentName

• `Readonly` **agentName**: `string`

#### Defined in

[genai/session.ts:22](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/genai/session.ts#L22)

***

### model

• `Readonly` **model**: `string`

#### Defined in

[genai/session.ts:23](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/genai/session.ts#L23)

***

### sessionId

• `Readonly` **sessionId**: `string`

#### Defined in

[genai/session.ts:24](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/genai/session.ts#L24)

## Methods

### end

▸ **end**(): `void`

#### Returns

`void`

#### Defined in

[genai/session.ts:51](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/genai/session.ts#L51)

***

### startTurn

▸ **startTurn**(`opts?`): [`Turn`](./turn)

#### Parameters

| Name   | Type                                 |
| :----- | :----------------------------------- |
| `opts` | [`TurnInit`](../interfaces/turninit) |

#### Returns

[`Turn`](./turn)

#### Defined in

[genai/session.ts:43](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/genai/session.ts#L43)

***

### create

▸ **create**(`opts?`): [`Session`](./session)

#### Parameters

| Name   | Type                                       |
| :----- | :----------------------------------------- |
| `opts` | [`SessionInit`](../interfaces/sessioninit) |

#### Returns

[`Session`](./session)

#### Defined in

[genai/session.ts:27](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/genai/session.ts#L27)


# Class: StringPrompt
Source: https://docs.wandb.ai/weave/reference/typescript-sdk/classes/stringprompt

TypeScript SDK reference

[weave](../) / StringPrompt

## Hierarchy

* `Prompt`

  ↳ `StringPrompt`

## Table of contents

### Constructors

* [constructor](./stringprompt#constructor)

### Properties

* [\_\_savedRef](./stringprompt#__savedref)
* [content](./stringprompt#content)

### Accessors

* [description](./stringprompt#description)
* [name](./stringprompt#name)

### Methods

* [format](./stringprompt#format)
* [saveAttrs](./stringprompt#saveattrs)
* [get](./stringprompt#get)

## Constructors

### constructor

• **new StringPrompt**(`parameters`): [`StringPrompt`](./stringprompt)

#### Parameters

| Name         | Type                     |
| :----------- | :----------------------- |
| `parameters` | `StringPromptParameters` |

#### Returns

[`StringPrompt`](./stringprompt)

#### Overrides

Prompt.constructor

#### Defined in

[prompt.ts:17](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/prompt.ts#L17)

## Properties

### \_\_savedRef

• `Optional` **\_\_savedRef**: [`ObjectRef`](./objectref) | `Promise`\<[`ObjectRef`](./objectref)>

#### Inherited from

Prompt.\_\_savedRef

#### Defined in

[weaveObject.ts:73](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/weaveObject.ts#L73)

***

### content

• **content**: `string`

#### Defined in

[prompt.ts:15](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/prompt.ts#L15)

## Accessors

### description

• `get` **description**(): `undefined` | `string`

#### Returns

`undefined` | `string`

#### Inherited from

Prompt.description

#### Defined in

[weaveObject.ts:100](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/weaveObject.ts#L100)

***

### name

• `get` **name**(): `string`

#### Returns

`string`

#### Inherited from

Prompt.name

#### Defined in

[weaveObject.ts:96](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/weaveObject.ts#L96)

## Methods

### format

▸ **format**(`values?`): `string`

#### Parameters

| Name     | Type                       |
| :------- | :------------------------- |
| `values` | `Record`\<`string`, `any`> |

#### Returns

`string`

#### Defined in

[prompt.ts:22](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/prompt.ts#L22)

***

### saveAttrs

▸ **saveAttrs**(): `Object`

#### Returns

`Object`

#### Inherited from

Prompt.saveAttrs

#### Defined in

[weaveObject.ts:77](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/weaveObject.ts#L77)

***

### get

▸ **get**(`client`, `uri`): `Promise`\<[`StringPrompt`](./stringprompt)>

#### Parameters

| Name     | Type                           |
| :------- | :----------------------------- |
| `client` | [`WeaveClient`](./weaveclient) |
| `uri`    | `string`                       |

#### Returns

`Promise`\<[`StringPrompt`](./stringprompt)>

#### Defined in

[prompt.ts:26](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/prompt.ts#L26)


# Class: SubAgent
Source: https://docs.wandb.ai/weave/reference/typescript-sdk/classes/subagent

TypeScript SDK reference

[weave](../) / SubAgent

A nested agent invocation — used when the current agent hands work to
another named agent (e.g. a planner calling a researcher). Emits an
`invoke_agent` span tagged with the sub-agent's name and (optionally)
its model.

Created by `weave.startSubagent()` (or `turn.startAgent()`, or
`llm.startAgent()`) and terminated with `end()`.

`Example`

```ts theme={null}
const sub = weave.startSubagent({name: 'researcher', model: 'gpt-4o'});
try {
  // ... orchestrate the sub-agent's LLM/Tool calls ...
} finally {
  sub.end();
}
```

## Table of contents

### Properties

* [model](./subagent#model)
* [name](./subagent#name)

### Methods

* [end](./subagent#end)
* [create](./subagent#create)

## Properties

### model

• `Readonly` **model**: `string`

#### Defined in

[genai/subagent.ts:41](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/genai/subagent.ts#L41)

***

### name

• `Readonly` **name**: `string`

#### Defined in

[genai/subagent.ts:40](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/genai/subagent.ts#L40)

## Methods

### end

▸ **end**(`opts?`): `void`

Close the SubAgent span. Idempotent. Pass `error` to mark it as failed.

#### Parameters

| Name          | Type     |
| :------------ | :------- |
| `opts?`       | `Object` |
| `opts.error?` | `Error`  |

#### Returns

`void`

#### Defined in

[genai/subagent.ts:65](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/genai/subagent.ts#L65)

***

### create

▸ **create**(`opts`): [`SubAgent`](./subagent)

#### Parameters

| Name   | Type                                                              |
| :----- | :---------------------------------------------------------------- |
| `opts` | [`SubAgentInit`](../interfaces/subagentinit) & `ChildSpanContext` |

#### Returns

[`SubAgent`](./subagent)

#### Defined in

[genai/subagent.ts:44](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/genai/subagent.ts#L44)


# Class: Tool
Source: https://docs.wandb.ai/weave/reference/typescript-sdk/classes/tool

TypeScript SDK reference

[weave](../) / Tool

A tool invocation. Emits an `execute_tool` span carrying the tool name,
the JSON-encoded arguments, the tool-call id, and the result.

Created by `weave.startTool()` (or `turn.startTool()`, or
`llm.startTool()`) and terminated with `end()`. Assign `result` before
calling `end()` to record the tool's output on the span.

`Example`

```ts theme={null}
const tool = weave.startTool({
  name: tc.function.name,
  args: tc.function.arguments,
  toolCallId: tc.id,
});
try {
  tool.result = await wikipediaSearch(JSON.parse(tc.function.arguments));
} finally {
  tool.end();
}
```

## Table of contents

### Properties

* [args](./tool#args)
* [name](./tool#name)
* [result](./tool#result)
* [toolCallId](./tool#toolcallid)

### Methods

* [end](./tool#end)
* [create](./tool#create)

## Properties

### args

• `Readonly` **args**: `string`

#### Defined in

[genai/tool.ts:52](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/genai/tool.ts#L52)

***

### name

• `Readonly` **name**: `string`

#### Defined in

[genai/tool.ts:51](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/genai/tool.ts#L51)

***

### result

• `Optional` **result**: `string`

Tool output as a string. Recorded on `gen_ai.tool.call.result` at `end()`.

#### Defined in

[genai/tool.ts:45](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/genai/tool.ts#L45)

***

### toolCallId

• `Readonly` **toolCallId**: `string`

#### Defined in

[genai/tool.ts:53](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/genai/tool.ts#L53)

## Methods

### end

▸ **end**(`opts?`): `void`

Flush `result` to the span and close it. Idempotent. Pass `error` to
mark the span as failed.

#### Parameters

| Name          | Type     |
| :------------ | :------- |
| `opts?`       | `Object` |
| `opts.error?` | `Error`  |

#### Returns

`void`

#### Defined in

[genai/tool.ts:83](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/genai/tool.ts#L83)

***

### create

▸ **create**(`opts`): [`Tool`](./tool)

#### Parameters

| Name   | Type                                                      |
| :----- | :-------------------------------------------------------- |
| `opts` | [`ToolInit`](../interfaces/toolinit) & `ChildSpanContext` |

#### Returns

[`Tool`](./tool)

#### Defined in

[genai/tool.ts:56](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/genai/tool.ts#L56)


# Class: Turn
Source: https://docs.wandb.ai/weave/reference/typescript-sdk/classes/turn

TypeScript SDK reference

[weave](../) / Turn

An agent invocation. Typically wraps the work to respond to a single
user message. Emits an `invoke_agent` span and acts as the root of the
trace for that turn: it is always started under `ROOT_CONTEXT` so it
never accidentally inherits a parent from another OTel-instrumented
library.

Created by `weave.startTurn()` (or `session.startTurn()`) and
terminated with `end()`. Only one Turn may be active in an async chain.
Children (LLM, Tool, SubAgent) attach via the `startLLM`, `startTool`,
`startSubagent` methods.

`Example`

```ts theme={null}
const turn = weave.startTurn({agentName: 'research-bot', model: MODEL});
try {
  const llm = turn.startLLM({model: MODEL, providerName: 'openai'});
  // ...
  llm.end();
} finally {
  turn.end();
}
```

## Table of contents

### Properties

* [agentName](./turn#agentname)
* [model](./turn#model)

### Methods

* [addEvent](./turn#addevent)
* [end](./turn#end)
* [setAttribute](./turn#setattribute)
* [startLLM](./turn#startllm)
* [startSubagent](./turn#startsubagent)
* [startTool](./turn#starttool)
* [create](./turn#create)

## Properties

### agentName

• `Readonly` **agentName**: `string`

#### Defined in

[genai/turn.ts:60](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/genai/turn.ts#L60)

***

### model

• `Readonly` **model**: `string`

#### Defined in

[genai/turn.ts:61](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/genai/turn.ts#L61)

## Methods

### addEvent

▸ **addEvent**(`name`, `attributes?`, `startTime?`): `this`

Add a named event to the Turn span. Useful for marking non-span moments
such as context compaction, tool-loop detection, or guardrail trips.
No-op after `end()`. Mirrors OTel `Span.addEvent`.

#### Parameters

| Name          | Type         |
| :------------ | :----------- |
| `name`        | `string`     |
| `attributes?` | `Attributes` |
| `startTime?`  | `TimeInput`  |

#### Returns

`this`

`Example`

```ts theme={null}
turn.addEvent('context_compacted', {removedMessages: 12});
```

#### Defined in

[genai/turn.ts:152](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/genai/turn.ts#L152)

***

### end

▸ **end**(`opts?`): `void`

Close the Turn span. Idempotent. Pass `error` to mark it as failed.

#### Parameters

| Name          | Type     |
| :------------ | :------- |
| `opts?`       | `Object` |
| `opts.error?` | `Error`  |

#### Returns

`void`

#### Defined in

[genai/turn.ts:159](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/genai/turn.ts#L159)

***

### setAttribute

▸ **setAttribute**(`key`, `value`): `this`

Set a single attribute on the Turn span. Useful for stamping running
totals (e.g. cumulative cost, token usage) or other metadata that becomes
known mid-turn. No-op after `end()`. Mirrors OTel `Span.setAttribute`.

#### Parameters

| Name    | Type             |
| :------ | :--------------- |
| `key`   | `string`         |
| `value` | `AttributeValue` |

#### Returns

`this`

`Example`

```ts theme={null}
turn.setAttribute('gen_ai.usage.input_tokens', totalInputTokens);
```

#### Defined in

[genai/turn.ts:138](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/genai/turn.ts#L138)

***

### startLLM

▸ **startLLM**(`opts`): [`LLM`](./llm)

Start a child LLM span under this Turn.

#### Parameters

| Name   | Type                               |
| :----- | :--------------------------------- |
| `opts` | [`LLMInit`](../interfaces/llminit) |

#### Returns

[`LLM`](./llm)

#### Defined in

[genai/turn.ts:104](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/genai/turn.ts#L104)

***

### startSubagent

▸ **startSubagent**(`opts`): [`SubAgent`](./subagent)

Start a child SubAgent span under this Turn.

#### Parameters

| Name   | Type                                         |
| :----- | :------------------------------------------- |
| `opts` | [`SubAgentInit`](../interfaces/subagentinit) |

#### Returns

[`SubAgent`](./subagent)

#### Defined in

[genai/turn.ts:122](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/genai/turn.ts#L122)

***

### startTool

▸ **startTool**(`opts`): [`Tool`](./tool)

Start a child Tool span under this Turn.

#### Parameters

| Name   | Type                                 |
| :----- | :----------------------------------- |
| `opts` | [`ToolInit`](../interfaces/toolinit) |

#### Returns

[`Tool`](./tool)

#### Defined in

[genai/turn.ts:113](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/genai/turn.ts#L113)

***

### create

▸ **create**(`opts?`): [`Turn`](./turn)

#### Parameters

| Name   | Type                                                                     |
| :----- | :----------------------------------------------------------------------- |
| `opts` | [`TurnInit`](../interfaces/turninit) & \{ `conversationId?`: `string`  } |

#### Returns

[`Turn`](./turn)

#### Defined in

[genai/turn.ts:64](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/genai/turn.ts#L64)


# Class: WeaveClient
Source: https://docs.wandb.ai/weave/reference/typescript-sdk/classes/weaveclient

TypeScript SDK reference

[weave](../) / WeaveClient

## Table of contents

### Constructors

* [constructor](./weaveclient#constructor)

### Properties

* [projectId](./weaveclient#projectid)
* [settings](./weaveclient#settings)
* [traceServerApi](./weaveclient#traceserverapi)

### Methods

* [addScore](./weaveclient#addscore)
* [createCall](./weaveclient#createcall)
* [finishCall](./weaveclient#finishcall)
* [finishCallWithException](./weaveclient#finishcallwithexception)
* [get](./weaveclient#get)
* [getCall](./weaveclient#getcall)
* [getCallStack](./weaveclient#getcallstack)
* [getCalls](./weaveclient#getcalls)
* [getCallsIterator](./weaveclient#getcallsiterator)
* [getCurrentAttributes](./weaveclient#getcurrentattributes)
* [linkPromptToRegistry](./weaveclient#linkprompttoregistry)
* [publish](./weaveclient#publish)
* [pushNewCall](./weaveclient#pushnewcall)
* [runWithAttributes](./weaveclient#runwithattributes)
* [runWithCallStack](./weaveclient#runwithcallstack)
* [saveCallEnd](./weaveclient#savecallend)
* [saveCallStart](./weaveclient#savecallstart)
* [saveOp](./weaveclient#saveop)
* [serializeAudio](./weaveclient#serializeaudio)
* [updateCall](./weaveclient#updatecall)
* [waitForBatchProcessing](./weaveclient#waitforbatchprocessing)

## Constructors

### constructor

• **new WeaveClient**(`traceServerApi`, `wandbServerApi`, `projectId`, `settings?`): [`WeaveClient`](./weaveclient)

#### Parameters

| Name             | Type             |
| :--------------- | :--------------- |
| `traceServerApi` | `Api`\<`any`>    |
| `wandbServerApi` | `WandbServerApi` |
| `projectId`      | `string`         |
| `settings`       | `Settings`       |

#### Returns

[`WeaveClient`](./weaveclient)

#### Defined in

[weaveClient.ts:190](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/weaveClient.ts#L190)

## Properties

### projectId

• **projectId**: `string`

#### Defined in

[weaveClient.ts:193](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/weaveClient.ts#L193)

***

### settings

• **settings**: `Settings`

#### Defined in

[weaveClient.ts:194](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/weaveClient.ts#L194)

***

### traceServerApi

• **traceServerApi**: `Api`\<`any`>

#### Defined in

[weaveClient.ts:191](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/weaveClient.ts#L191)

## Methods

### addScore

▸ **addScore**(`predictCallId`, `scorerCallId`, `runnableRefUri`, `scorerOutput`): `Promise`\<`string`>

Add a scorer result (e.g., scorer output) to a call.
Used in imperative evaluation to attach scorer results to predict calls.

#### Parameters

| Name             | Type     | Description                                       |
| :--------------- | :------- | :------------------------------------------------ |
| `predictCallId`  | `string` | ID of the predict call to attach feedback to      |
| `scorerCallId`   | `string` | ID of the scorer call that generated the feedback |
| `runnableRefUri` | `string` | URI of the scorer (Op or Object ref)              |
| `scorerOutput`   | `any`    | Output of the scorer                              |

#### Returns

`Promise`\<`string`>

#### Defined in

[weaveClient.ts:1132](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/weaveClient.ts#L1132)

***

### createCall

▸ **createCall**(`internalCall`, `opRef`, `params`, `parameterNames`, `thisArg`, `currentCall`, `parentCall`, `startTime`, `displayName?`, `attributes?`): `Promise`\<`void`>

#### Parameters

| Name             | Type                            |
| :--------------- | :------------------------------ |
| `internalCall`   | `InternalCall`                  |
| `opRef`          | `any`                           |
| `params`         | `any`\[]                        |
| `parameterNames` | `ParameterNamesOption`          |
| `thisArg`        | `any`                           |
| `currentCall`    | `CallStackEntry`                |
| `parentCall`     | `undefined` \| `CallStackEntry` |
| `startTime`      | `Date`                          |
| `displayName?`   | `string`                        |
| `attributes?`    | `Record`\<`string`, `any`>      |

#### Returns

`Promise`\<`void`>

#### Defined in

[weaveClient.ts:979](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/weaveClient.ts#L979)

***

### finishCall

▸ **finishCall**(`call`, `result`, `currentCall`, `parentCall`, `summarize`, `endTime`, `startCallPromise`): `Promise`\<`void`>

#### Parameters

| Name               | Type                                                           |
| :----------------- | :------------------------------------------------------------- |
| `call`             | `InternalCall`                                                 |
| `result`           | `any`                                                          |
| `currentCall`      | `CallStackEntry`                                               |
| `parentCall`       | `undefined` \| `CallStackEntry`                                |
| `summarize`        | `undefined` \| (`result`: `any`) => `Record`\<`string`, `any`> |
| `endTime`          | `Date`                                                         |
| `startCallPromise` | `Promise`\<`void`>                                             |

#### Returns

`Promise`\<`void`>

#### Defined in

[weaveClient.ts:1039](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/weaveClient.ts#L1039)

***

### finishCallWithException

▸ **finishCallWithException**(`call`, `error`, `currentCall`, `parentCall`, `endTime`, `startCallPromise`): `Promise`\<`void`>

#### Parameters

| Name               | Type                            |
| :----------------- | :------------------------------ |
| `call`             | `InternalCall`                  |
| `error`            | `any`                           |
| `currentCall`      | `CallStackEntry`                |
| `parentCall`       | `undefined` \| `CallStackEntry` |
| `endTime`          | `Date`                          |
| `startCallPromise` | `Promise`\<`void`>              |

#### Returns

`Promise`\<`void`>

#### Defined in

[weaveClient.ts:1079](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/weaveClient.ts#L1079)

***

### get

▸ **get**(`ref`): `Promise`\<`any`>

#### Parameters

| Name  | Type                       |
| :---- | :------------------------- |
| `ref` | [`ObjectRef`](./objectref) |

#### Returns

`Promise`\<`any`>

#### Defined in

[weaveClient.ts:435](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/weaveClient.ts#L435)

***

### getCall

▸ **getCall**(`callId`, `includeCosts?`): `Promise`\<`Call`>

#### Parameters

| Name           | Type      | Default value |
| :------------- | :-------- | :------------ |
| `callId`       | `string`  | `undefined`   |
| `includeCosts` | `boolean` | `false`       |

#### Returns

`Promise`\<`Call`>

#### Defined in

[weaveClient.ts:312](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/weaveClient.ts#L312)

***

### getCallStack

▸ **getCallStack**(): `CallStack`

#### Returns

`CallStack`

#### Defined in

[weaveClient.ts:890](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/weaveClient.ts#L890)

***

### getCalls

▸ **getCalls**(`options?`): `Promise`\<`Call`\[]>

#### Parameters

| Name       | Type                                               |
| :--------- | :------------------------------------------------- |
| `options?` | [`GetCallsOptions`](../interfaces/getcallsoptions) |

#### Returns

`Promise`\<`Call`\[]>

#### Defined in

[weaveClient.ts:345](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/weaveClient.ts#L345)

▸ **getCalls**(`options?`, `includeCosts?`, `limit?`): `Promise`\<`Call`\[]>

#### Parameters

| Name            | Type                                       |
| :-------------- | :----------------------------------------- |
| `options?`      | [`CallsFilter`](../interfaces/callsfilter) |
| `includeCosts?` | `boolean`                                  |
| `limit?`        | `number`                                   |

#### Returns

`Promise`\<`Call`\[]>

#### Defined in

[weaveClient.ts:346](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/weaveClient.ts#L346)

***

### getCallsIterator

▸ **getCallsIterator**(`options?`, `includeCosts?`, `limit?`): `AsyncIterableIterator`\<[`CallSchema`](../interfaces/callschema)>

#### Parameters

| Name            | Type                                       |
| :-------------- | :----------------------------------------- |
| `options?`      | [`CallsFilter`](../interfaces/callsfilter) |
| `includeCosts?` | `boolean`                                  |
| `limit?`        | `number`                                   |

#### Returns

`AsyncIterableIterator`\<[`CallSchema`](../interfaces/callschema)>

#### Defined in

[weaveClient.ts:367](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/weaveClient.ts#L367)

▸ **getCallsIterator**(`options?`): `AsyncIterableIterator`\<[`CallSchema`](../interfaces/callschema)>

#### Parameters

| Name       | Type                                               |
| :--------- | :------------------------------------------------- |
| `options?` | [`GetCallsOptions`](../interfaces/getcallsoptions) |

#### Returns

`AsyncIterableIterator`\<[`CallSchema`](../interfaces/callschema)>

#### Defined in

[weaveClient.ts:372](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/weaveClient.ts#L372)

***

### getCurrentAttributes

▸ **getCurrentAttributes**(): `Record`\<`string`, `any`>

#### Returns

`Record`\<`string`, `any`>

#### Defined in

[weaveClient.ts:894](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/weaveClient.ts#L894)

***

### linkPromptToRegistry

▸ **linkPromptToRegistry**(`prompt`, `options`): `Promise`\<`LinkAssetToRegistryRes`>

Link a published prompt version into a registry portfolio.

#### Parameters

| Name      | Type                          |
| :-------- | :---------------------------- |
| `prompt`  | `RegistryLinkable`            |
| `options` | `LinkPromptToRegistryOptions` |

#### Returns

`Promise`\<`LinkAssetToRegistryRes`>

#### Defined in

[weaveClient.ts:591](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/weaveClient.ts#L591)

***

### publish

▸ **publish**(`obj`, `objId?`): `Promise`\<[`ObjectRef`](./objectref)>

#### Parameters

| Name     | Type     |
| :------- | :------- |
| `obj`    | `any`    |
| `objId?` | `string` |

#### Returns

`Promise`\<[`ObjectRef`](./objectref)>

#### Defined in

[weaveClient.ts:300](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/weaveClient.ts#L300)

***

### pushNewCall

▸ **pushNewCall**(): `Object`

#### Returns

`Object`

| Name          | Type             |
| :------------ | :--------------- |
| `currentCall` | `CallStackEntry` |
| `newStack`    | `CallStack`      |
| `parentCall?` | `CallStackEntry` |

#### Defined in

[weaveClient.ts:898](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/weaveClient.ts#L898)

***

### runWithAttributes

▸ **runWithAttributes**\<`T`>(`attributes`, `fn`): `T`

#### Type parameters

| Name |
| :--- |
| `T`  |

#### Parameters

| Name         | Type                       |
| :----------- | :------------------------- |
| `attributes` | `Record`\<`string`, `any`> |
| `fn`         | () => `T`                  |

#### Returns

`T`

#### Defined in

[weaveClient.ts:906](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/weaveClient.ts#L906)

***

### runWithCallStack

▸ **runWithCallStack**\<`T`>(`callStack`, `fn`): `T`

#### Type parameters

| Name |
| :--- |
| `T`  |

#### Parameters

| Name        | Type        |
| :---------- | :---------- |
| `callStack` | `CallStack` |
| `fn`        | () => `T`   |

#### Returns

`T`

#### Defined in

[weaveClient.ts:902](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/weaveClient.ts#L902)

***

### saveCallEnd

▸ **saveCallEnd**(`callEnd`): `void`

#### Parameters

| Name      | Type                       |
| :-------- | :------------------------- |
| `callEnd` | `EndedCallSchemaForInsert` |

#### Returns

`void`

#### Defined in

[weaveClient.ts:885](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/weaveClient.ts#L885)

***

### saveCallStart

▸ **saveCallStart**(`callStart`): `void`

#### Parameters

| Name        | Type                         |
| :---------- | :--------------------------- |
| `callStart` | `StartedCallSchemaForInsert` |

#### Returns

`void`

#### Defined in

[weaveClient.ts:880](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/weaveClient.ts#L880)

***

### saveOp

▸ **saveOp**(`op`, `objId?`): `Promise`\<`OpRef`>

#### Parameters

| Name     | Type                                            |
| :------- | :---------------------------------------------- |
| `op`     | [`Op`](../#op)\<(...`args`: `any`\[]) => `any`> |
| `objId?` | `string`                                        |

#### Returns

`Promise`\<`OpRef`>

#### Defined in

[weaveClient.ts:945](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/weaveClient.ts#L945)

***

### serializeAudio

▸ **serializeAudio**(`data`, `audioType?`): `Promise`\<`SerializedFileBlob`>

Upload raw audio bytes to the Weave content store and return the
`CustomWeaveType` placeholder that can be embedded in a call output.

Use this when building call outputs manually (e.g. via `saveCallEnd`)
where the automatic serialization pipeline from `finishCall` is not used.

#### Parameters

| Name        | Type     | Default value        | Description                                          |
| :---------- | :------- | :------------------- | :--------------------------------------------------- |
| `data`      | `Buffer` | `undefined`          | Raw audio bytes (WAV for best browser compatibility) |
| `audioType` | `"wav"`  | `DEFAULT_AUDIO_TYPE` | File format — currently only 'wav' is supported      |

#### Returns

`Promise`\<`SerializedFileBlob`>

#### Defined in

[weaveClient.ts:832](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/weaveClient.ts#L832)

***

### updateCall

▸ **updateCall**(`callId`, `displayName`): `Promise`\<`void`>

#### Parameters

| Name          | Type     |
| :------------ | :------- |
| `callId`      | `string` |
| `displayName` | `string` |

#### Returns

`Promise`\<`void`>

#### Defined in

[weaveClient.ts:1115](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/weaveClient.ts#L1115)

***

### waitForBatchProcessing

▸ **waitForBatchProcessing**(): `Promise`\<`void`>

#### Returns

`Promise`\<`void`>

#### Defined in

[weaveClient.ts:211](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/weaveClient.ts#L211)


# Class: WeaveObject
Source: https://docs.wandb.ai/weave/reference/typescript-sdk/classes/weaveobject

TypeScript SDK reference

[weave](../) / WeaveObject

## Hierarchy

* `WeaveObject`

  ↳ [`Dataset`](./dataset)

  ↳ [`Evaluation`](./evaluation)

## Table of contents

### Constructors

* [constructor](./weaveobject#constructor)

### Properties

* [\_\_savedRef](./weaveobject#__savedref)

### Accessors

* [description](./weaveobject#description)
* [name](./weaveobject#name)

### Methods

* [saveAttrs](./weaveobject#saveattrs)

## Constructors

### constructor

• **new WeaveObject**(`_baseParameters`): [`WeaveObject`](./weaveobject)

#### Parameters

| Name              | Type                    |
| :---------------- | :---------------------- |
| `_baseParameters` | `WeaveObjectParameters` |

#### Returns

[`WeaveObject`](./weaveobject)

#### Defined in

[weaveObject.ts:75](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/weaveObject.ts#L75)

## Properties

### \_\_savedRef

• `Optional` **\_\_savedRef**: [`ObjectRef`](./objectref) | `Promise`\<[`ObjectRef`](./objectref)>

#### Defined in

[weaveObject.ts:73](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/weaveObject.ts#L73)

## Accessors

### description

• `get` **description**(): `undefined` | `string`

#### Returns

`undefined` | `string`

#### Defined in

[weaveObject.ts:100](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/weaveObject.ts#L100)

***

### name

• `get` **name**(): `string`

#### Returns

`string`

#### Defined in

[weaveObject.ts:96](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/weaveObject.ts#L96)

## Methods

### saveAttrs

▸ **saveAttrs**(): `Object`

#### Returns

`Object`

#### Defined in

[weaveObject.ts:77](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/weaveObject.ts#L77)


# createOpenAIAgentsTracingProcessor
Source: https://docs.wandb.ai/weave/reference/typescript-sdk/functions/createopenaiagentstracingprocessor

TypeScript SDK reference

# createOpenAIAgentsTracingProcessor

▸ **createOpenAIAgentsTracingProcessor**(): `TracingProcessor`

Create a new Weave tracing processor for OpenAI Agents.

#### Returns

`TracingProcessor`

A TracingProcessor instance that can be registered with OpenAI Agents

`Example`

```typescript theme={null}
import { addTraceProcessor } from '@openai/agents';
import { createOpenAIAgentsTracingProcessor } from 'weave';

const processor = createOpenAIAgentsTracingProcessor();
addTraceProcessor(processor);
```

#### Defined in

[integrations/openai.agent.ts:665](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/integrations/openai.agent.ts#L665)

***


# createOtelExtension
Source: https://docs.wandb.ai/weave/reference/typescript-sdk/functions/createotelextension

TypeScript SDK reference

# createOtelExtension

▸ **createOtelExtension**(`opts?`): (`pi`: `PiExtensionApi`) => `void`

Creates a pi coding agent extension that emits OTEL spans for the full
agent lifecycle, conforming to the GenAI semantic conventions.

When `weave.init(...)` has been called, spans are automatically exported
to the Weave trace server at `/agents/otel/v1/traces`. Otherwise, pass a
custom `tracer` in `opts`.

#### Parameters

| Name   | Type                   |
| :----- | :--------------------- |
| `opts` | `OtelExtensionOptions` |

#### Returns

`fn`

▸ (`pi`): `void`

##### Parameters

| Name | Type             |
| :--- | :--------------- |
| `pi` | `PiExtensionApi` |

##### Returns

`void`

`Example`

```typescript theme={null}
const resourceLoader = new DefaultResourceLoader({
  extensionFactories: [createOtelExtension()],
});
```

#### Defined in

[integrations/piCodingAgent.ts:644](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/integrations/piCodingAgent.ts#L644)

***


# endLLM
Source: https://docs.wandb.ai/weave/reference/typescript-sdk/functions/endllm

TypeScript SDK reference

# endLLM

▸ **endLLM**(): `void`

End the current LLM. No-op if no LLM is active.

#### Returns

`void`

#### Defined in

[genai/api.ts:102](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/genai/api.ts#L102)

***


# endSession
Source: https://docs.wandb.ai/weave/reference/typescript-sdk/functions/endsession

TypeScript SDK reference

# endSession

▸ **endSession**(): `void`

End the current Session. No-op if no Session is active.

#### Returns

`void`

#### Defined in

[genai/api.ts:82](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/genai/api.ts#L82)

***


# endTurn
Source: https://docs.wandb.ai/weave/reference/typescript-sdk/functions/endturn

TypeScript SDK reference

# endTurn

▸ **endTurn**(): `void`

End the current Turn. No-op if no Turn is active.

#### Returns

`void`

#### Defined in

[genai/api.ts:92](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/genai/api.ts#L92)

***


# flushOTel
Source: https://docs.wandb.ai/weave/reference/typescript-sdk/functions/flushotel

TypeScript SDK reference

# flushOTel

▸ **flushOTel**(): `Promise`\<`void`>

Force-flush any GenAI spans buffered by the active span processor.

Resolves immediately if `weave.init()` has not been called. Otherwise
delegates to `BasicTracerProvider.forceFlush()`, which waits for the
underlying span processor to drain its queue and complete its OTLP export
round-trip.

Call before process exit when using `'simple'` or any other processor that
may have in-flight work, or in tests that need to observe exported spans
synchronously.

#### Returns

`Promise`\<`void`>

#### Defined in

[genai/flush.ts:15](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/genai/flush.ts#L15)

***


# getCurrentLLM
Source: https://docs.wandb.ai/weave/reference/typescript-sdk/functions/getcurrentllm

TypeScript SDK reference

# getCurrentLLM

▸ **getCurrentLLM**(): [`LLM`](../classes/llm) | `undefined`

Returns the current LLM, or undefined.

#### Returns

[`LLM`](../classes/llm) | `undefined`

#### Defined in

[genai/context.ts:105](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/genai/context.ts#L105)

***


# getCurrentSession
Source: https://docs.wandb.ai/weave/reference/typescript-sdk/functions/getcurrentsession

TypeScript SDK reference

# getCurrentSession

▸ **getCurrentSession**(): [`Session`](../classes/session) | `undefined`

Returns the current Session, or undefined.

#### Returns

[`Session`](../classes/session) | `undefined`

#### Defined in

[genai/context.ts:95](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/genai/context.ts#L95)

***


# getCurrentTurn
Source: https://docs.wandb.ai/weave/reference/typescript-sdk/functions/getcurrentturn

TypeScript SDK reference

# getCurrentTurn

▸ **getCurrentTurn**(): [`Turn`](../classes/turn) | `undefined`

Returns the current Turn, or undefined.

#### Returns

[`Turn`](../classes/turn) | `undefined`

#### Defined in

[genai/context.ts:100](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/genai/context.ts#L100)

***


# init
Source: https://docs.wandb.ai/weave/reference/typescript-sdk/functions/init

TypeScript SDK reference

# init

▸ **init**(`project`, `settings?`): `Promise`\<[`WeaveClient`](../classes/weaveclient)>

Initialize the Weave client, which is required for weave tracing to work.

#### Parameters

| Name        | Type           | Description                                                                                                                                                                                                                                                                                                                                                                 |
| :---------- | :------------- | :-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `project`   | `string`       | The W\&B project name (can be project or entity/project). If you don't specify a W\&B team (e.g., 'team/project'), your default entity is used. To find or update your default entity, refer to User Settings at [https://docs.wandb.ai/platform/app/settings-page/user-settings#default-team](https://docs.wandb.ai/platform/app/settings-page/user-settings#default-team) |
| `settings?` | `SettingsInit` | (Optional) Weave tracing settings                                                                                                                                                                                                                                                                                                                                           |

#### Returns

`Promise`\<[`WeaveClient`](../classes/weaveclient)>

A promise that resolves to the initialized Weave client.

`Throws`

If the initialization fails

#### Defined in

[clientApi.ts:90](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/clientApi.ts#L90)

***


# instrumentOpenAIAgents
Source: https://docs.wandb.ai/weave/reference/typescript-sdk/functions/instrumentopenaiagents

TypeScript SDK reference

# instrumentOpenAIAgents

▸ **instrumentOpenAIAgents**(): `Promise`\<`boolean`>

Manually register Weave tracing with OpenAI Agents if the package is available.

**Note: You typically don't need to call this function!** OpenAI Agents is automatically
instrumented via module loader hooks when you import Weave. This function is provided for
edge cases where automatic instrumentation doesn't work (e.g., dynamic imports, bundlers
that bypass hooks).

This function attempts to dynamically import @openai/agents from the consumer's node\_modules
and registers a TracingProcessor. If the package is not installed, it returns false without
throwing an error.

#### Returns

`Promise`\<`boolean`>

`Promise<boolean>` - `true` if registration succeeded, `false` if `@openai/agents` not available

`Example`

```typescript theme={null}
// ✅ Recommended: Just import Weave - instrumentation happens automatically!
import * as weave from 'weave';
await weave.init('my-project');

// OpenAI Agents is already instrumented via hooks - no manual setup needed
import { Agent } from '@openai/agents';
const agent = new Agent({ ... });
await agent.run(input); // Automatically traced in Weave
```

`Example`

```typescript theme={null}
// ⚠️ Only needed for edge cases where automatic hooks don't work
import { instrumentOpenAIAgents } from 'weave';

const registered = await instrumentOpenAIAgents();
if (!registered) {
  console.log('OpenAI Agents not found - install @openai/agents to enable tracing');
}
```

`Remarks`

**How automatic instrumentation works**: When you import Weave, it registers module loader
hooks via `addCJSInstrumentation()` and `addESMInstrumentation()`. When your code later
imports `@openai/agents`, these hooks intercept the import and automatically patch the
module to add Weave tracing. This happens transparently - no action required from you!

**When to use this function**: Only use this if automatic instrumentation fails, such as:

* Using dynamic imports that bypass module hooks
* Bundlers that don't support import-in-the-middle
* Need explicit control over when instrumentation happens

**Alternative for custom processor logic**: If you need custom tracing behavior,
use `createOpenAIAgentsTracingProcessor()` and register it manually:

```typescript theme={null}
import { addTraceProcessor } from '@openai/agents';
import { createOpenAIAgentsTracingProcessor } from 'weave';

const processor = createOpenAIAgentsTracingProcessor();
addTraceProcessor(processor);
```

#### Defined in

[integrations/openai.agent.ts:727](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/integrations/openai.agent.ts#L727)

***


# login
Source: https://docs.wandb.ai/weave/reference/typescript-sdk/functions/login

TypeScript SDK reference

# login

▸ **login**(`apiKey`, `host?`): `Promise`\<`void`>

Log in to Weights & Biases (W\&B) using the provided API key.
This function attempts to save the credentials to your netrc file for future use,
but will continue even if it cannot write to the file system.

#### Parameters

| Name     | Type     | Description                                                                          |
| :------- | :------- | :----------------------------------------------------------------------------------- |
| `apiKey` | `string` | Your W\&B API key.                                                                   |
| `host?`  | `string` | (Optional) The host name (usually only needed if you're using a custom W\&B server). |

#### Returns

`Promise`\<`void`>

`Throws`

If the API key is not specified or if the connection to the weave trace server cannot be verified.

#### Defined in

[clientApi.ts:30](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/clientApi.ts#L30)

***


# op
Source: https://docs.wandb.ai/weave/reference/typescript-sdk/functions/op

TypeScript SDK reference

# op

▸ **op**\<`T`>(`fn`, `options?`): [`Op`](../type-aliases/op)\<`T`>

#### Type parameters

| Name | Type                                   |
| :--- | :------------------------------------- |
| `T`  | extends (...`args`: `any`\[]) => `any` |

#### Parameters

| Name       | Type              |
| :--------- | :---------------- |
| `fn`       | `T`               |
| `options?` | `OpOptions`\<`T`> |

#### Returns

[`Op`](../type-aliases/op)\<`T`>

#### Defined in

[op.ts:381](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/op.ts#L381)

▸ **op**\<`T`>(`thisArg`, `fn`, `options?`): [`Op`](../type-aliases/op)\<`T`>

#### Type parameters

| Name | Type                                   |
| :--- | :------------------------------------- |
| `T`  | extends (...`args`: `any`\[]) => `any` |

#### Parameters

| Name       | Type              |
| :--------- | :---------------- |
| `thisArg`  | `any`             |
| `fn`       | `T`               |
| `options?` | `OpOptions`\<`T`> |

#### Returns

[`Op`](../type-aliases/op)\<`T`>

#### Defined in

[op.ts:386](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/op.ts#L386)

▸ **op**(`target`, `propertyKey`, `descriptor`): `TypedPropertyDescriptor`\<`any`>

#### Parameters

| Name          | Type                              |
| :------------ | :-------------------------------- |
| `target`      | `Object`                          |
| `propertyKey` | `string` \| `symbol`              |
| `descriptor`  | `TypedPropertyDescriptor`\<`any`> |

#### Returns

`TypedPropertyDescriptor`\<`any`>

#### Defined in

[op.ts:392](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/op.ts#L392)

▸ **op**\<`T`>(`value`, `context`): [`Op`](../type-aliases/op)\<`T`>

#### Type parameters

| Name | Type                                   |
| :--- | :------------------------------------- |
| `T`  | extends (...`args`: `any`\[]) => `any` |

#### Parameters

| Name      | Type                     |
| :-------- | :----------------------- |
| `value`   | `T`                      |
| `context` | `MethodDecoratorContext` |

#### Returns

[`Op`](../type-aliases/op)\<`T`>

#### Defined in

[op.ts:398](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/op.ts#L398)

▸ **op**(`options`): `MethodDecorator`

#### Parameters

| Name      | Type                            |
| :-------- | :------------------------------ |
| `options` | `Partial`\<`OpOptions`\<`any`>> |

#### Returns

`MethodDecorator`

#### Defined in

[op.ts:403](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/op.ts#L403)

***


# patchRealtimeSession
Source: https://docs.wandb.ai/weave/reference/typescript-sdk/functions/patchrealtimesession

TypeScript SDK reference

# patchRealtimeSession

▸ **patchRealtimeSession**(): `Promise`\<`boolean`>

Manually patch the `RealtimeSession` class from `@openai/agents-realtime` so that
every new instance is automatically traced by Weave.

**Note: You typically don't need to call this function!** `@openai/agents-realtime` is
automatically instrumented via module loader hooks when you import Weave. This function
is provided for edge cases where automatic instrumentation doesn't work (e.g. dynamic
imports, bundlers that bypass hooks).

Call this **once** at app startup, before any `RealtimeSession` is constructed.
The function is idempotent — safe to call multiple times.

#### Returns

`Promise`\<`boolean`>

`true` if the patch was applied (or was already in place), `false` if
`@openai/agents-realtime` could not be loaded.

`Example`

```typescript theme={null}
import { patchRealtimeSession } from 'weave';
await patchRealtimeSession();
// Every new RealtimeSession(...) is now auto-instrumented
```

#### Defined in

[integrations/openai.realtime.agent.ts:808](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/integrations/openai.realtime.agent.ts#L808)

***


# requireCurrentCallStackEntry
Source: https://docs.wandb.ai/weave/reference/typescript-sdk/functions/requirecurrentcallstackentry

TypeScript SDK reference

# requireCurrentCallStackEntry

▸ **requireCurrentCallStackEntry**(): `CallStackEntry`

#### Returns

`CallStackEntry`

#### Defined in

[clientApi.ts:155](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/clientApi.ts#L155)

***


# requireCurrentChildSummary
Source: https://docs.wandb.ai/weave/reference/typescript-sdk/functions/requirecurrentchildsummary

TypeScript SDK reference

# requireCurrentChildSummary

▸ **requireCurrentChildSummary**(): `Object`

#### Returns

`Object`

#### Defined in

[clientApi.ts:167](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/clientApi.ts#L167)

***


# runIsolated
Source: https://docs.wandb.ai/weave/reference/typescript-sdk/functions/runisolated

TypeScript SDK reference

# runIsolated

▸ **runIsolated**\<`T`>(`fn`): `T`

Run `fn` in a fresh, isolated GenAI state frame. Any Session / Turn / LLM
started inside `fn` lives in this frame only — it does not clash with
sibling `runIsolated` frames running concurrently, and it does not leak
to the outer async chain.

Use this to safely run parallel GenAI work:

```typescript theme={null}
await Promise.all([
  weave.runIsolated(async () => { ... }),
  weave.runIsolated(async () => { ... }),
]);
```

Sequential single-flight usage doesn't require this wrapper — the
process-wide default state handles it.

#### Type parameters

| Name |
| :--- |
| `T`  |

#### Parameters

| Name | Type      |
| :--- | :-------- |
| `fn` | () => `T` |

#### Returns

`T`

#### Defined in

[genai/context.ts:90](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/genai/context.ts#L90)

***


# startLLM
Source: https://docs.wandb.ai/weave/reference/typescript-sdk/functions/startllm

TypeScript SDK reference

# startLLM

▸ **startLLM**(`opts`): [`LLM`](../classes/llm)

Start an LLM span as a child of the current Turn. Throws if no Turn is
active.

#### Parameters

| Name   | Type                               |
| :----- | :--------------------------------- |
| `opts` | [`LLMInit`](../interfaces/llminit) |

#### Returns

[`LLM`](../classes/llm)

#### Defined in

[genai/api.ts:32](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/genai/api.ts#L32)

***


# startSession
Source: https://docs.wandb.ai/weave/reference/typescript-sdk/functions/startsession

TypeScript SDK reference

# startSession

▸ **startSession**(`opts?`): [`Session`](../classes/session)

Start a new Session and install it as the current session.
Subsequent calls to `startTurn` will pick it up automatically.

#### Parameters

| Name   | Type                                       |
| :----- | :----------------------------------------- |
| `opts` | [`SessionInit`](../interfaces/sessioninit) |

#### Returns

[`Session`](../classes/session)

#### Defined in

[genai/api.ts:12](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/genai/api.ts#L12)

***


# startSubagent
Source: https://docs.wandb.ai/weave/reference/typescript-sdk/functions/startsubagent

TypeScript SDK reference

# startSubagent

▸ **startSubagent**(`opts`): [`SubAgent`](../classes/subagent)

Start a SubAgent span. Same parent-resolution rules as `startTool`.

#### Parameters

| Name   | Type                                         |
| :----- | :------------------------------------------- |
| `opts` | [`SubAgentInit`](../interfaces/subagentinit) |

#### Returns

[`SubAgent`](../classes/subagent)

#### Defined in

[genai/api.ts:66](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/genai/api.ts#L66)

***


# startTool
Source: https://docs.wandb.ai/weave/reference/typescript-sdk/functions/starttool

TypeScript SDK reference

# startTool

▸ **startTool**(`opts`): [`Tool`](../classes/tool)

Start a Tool span. Parent resolution (matches the design's "flat by
default, hierarchical if you nest"):

* If an LLM is active, the Tool nests under it.
* Otherwise, the Tool is a sibling under the current Turn.

Throws if neither a Turn nor an LLM is active.

#### Parameters

| Name   | Type                                 |
| :----- | :----------------------------------- |
| `opts` | [`ToolInit`](../interfaces/toolinit) |

#### Returns

[`Tool`](../classes/tool)

#### Defined in

[genai/api.ts:50](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/genai/api.ts#L50)

***


# startTurn
Source: https://docs.wandb.ai/weave/reference/typescript-sdk/functions/startturn

TypeScript SDK reference

# startTurn

▸ **startTurn**(`opts?`): [`Turn`](../classes/turn)

Start a new Turn. If a Session is active, the turn inherits its
`conversationId`; otherwise the turn has no conversation id.

#### Parameters

| Name   | Type                                 |
| :----- | :----------------------------------- |
| `opts` | [`TurnInit`](../interfaces/turninit) |

#### Returns

[`Turn`](../classes/turn)

#### Defined in

[genai/api.ts:20](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/genai/api.ts#L20)

***


# weaveAudio
Source: https://docs.wandb.ai/weave/reference/typescript-sdk/functions/weaveaudio

TypeScript SDK reference

# weaveAudio

▸ **weaveAudio**(`options`): [`WeaveAudio`](../interfaces/weaveaudio)

Create a new WeaveAudio object

#### Parameters

| Name      | Type              | Description                                                                                                                                              |
| :-------- | :---------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `options` | `WeaveAudioInput` | The options for this media type - data: The raw audio data as a Buffer - audioType: (Optional) The type of audio file, currently only 'wav' is supported |

#### Returns

[`WeaveAudio`](../interfaces/weaveaudio)

`Example`

```ts theme={null}
const audioBuffer = fs.readFileSync('path/to/audio.wav');
const weaveAudio = weaveAudio({ data: audioBuffer });
```

#### Defined in

[media.ts:62](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/media.ts#L62)

***


# weaveImage
Source: https://docs.wandb.ai/weave/reference/typescript-sdk/functions/weaveimage

TypeScript SDK reference

# weaveImage

▸ **weaveImage**(`options`): [`WeaveImage`](../interfaces/weaveimage)

Create a new WeaveImage object

#### Parameters

| Name      | Type              | Description                                                                                                                                              |
| :-------- | :---------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `options` | `WeaveImageInput` | The options for this media type - data: The raw image data as a Buffer - imageType: (Optional) The type of image file, currently only 'png' is supported |

#### Returns

[`WeaveImage`](../interfaces/weaveimage)

`Example`

```ts theme={null}
const imageBuffer = fs.readFileSync('path/to/image.png');
const weaveImage = weaveImage({ data: imageBuffer });
```

#### Defined in

[media.ts:28](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/media.ts#L28)

***


# withAttributes
Source: https://docs.wandb.ai/weave/reference/typescript-sdk/functions/withattributes

TypeScript SDK reference

# withAttributes

▸ **withAttributes**\<`T`>(`attrs`, `fn`): `Promise`\<`T`> | `T`

Attach attributes to the current execution context so that any calls created
inside `fn` automatically inherit them. Attributes are written to the call
record on the trace server and surface in the Weave UI/filtering, so they’re
ideal for tagging runs with request IDs, tenants, experiments, etc.

Example:

```ts theme={null}
await withAttributes({requestId: 'abc'}, async () => {
  await myOp();
});
```

#### Type parameters

| Name |
| :--- |
| `T`  |

#### Parameters

| Name    | Type                         |
| :------ | :--------------------------- |
| `attrs` | `Record`\<`string`, `any`>   |
| `fn`    | () => `T` \| `Promise`\<`T`> |

#### Returns

`Promise`\<`T`> | `T`

#### Defined in

[clientApi.ts:201](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/clientApi.ts#L201)

***


# wrapGoogleGenAI
Source: https://docs.wandb.ai/weave/reference/typescript-sdk/functions/wrapgooglegenai

TypeScript SDK reference

# wrapGoogleGenAI

▸ **wrapGoogleGenAI**\<`T`>(`googleGenAI`): `T`

#### Type parameters

| Name | Type                     |
| :--- | :----------------------- |
| `T`  | extends `GoogleGenAIAPI` |

#### Parameters

| Name          | Type |
| :------------ | :--- |
| `googleGenAI` | `T`  |

#### Returns

`T`

#### Defined in

[integrations/googleGenAI.ts:223](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/integrations/googleGenAI.ts#L223)

***


# wrapOpenAI
Source: https://docs.wandb.ai/weave/reference/typescript-sdk/functions/wrapopenai

TypeScript SDK reference

# wrapOpenAI

▸ **wrapOpenAI**\<`T`>(`openai`): `T`

Wraps the OpenAI API to enable function tracing for OpenAI calls.

#### Type parameters

| Name | Type                |
| :--- | :------------------ |
| `T`  | extends `OpenAIAPI` |

#### Parameters

| Name     | Type |
| :------- | :--- |
| `openai` | `T`  |

#### Returns

`T`

`Example`

```ts theme={null}
const openai = wrapOpenAI(new OpenAI());
const result = await openai.chat.completions.create({
  model: 'gpt-3.5-turbo',
  messages: [{ role: 'user', content: 'Hello, world!' }]
});
```

#### Defined in

[integrations/openai.ts:703](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/integrations/openai.ts#L703)


# Interface: CallSchema
Source: https://docs.wandb.ai/weave/reference/typescript-sdk/interfaces/callschema

TypeScript SDK reference

[weave](../) / CallSchema

CallSchema

## Table of contents

### Properties

* [attributes](./callschema#attributes)
* [deleted\_at](./callschema#deleted_at)
* [display\_name](./callschema#display_name)
* [ended\_at](./callschema#ended_at)
* [exception](./callschema#exception)
* [id](./callschema#id)
* [inputs](./callschema#inputs)
* [op\_name](./callschema#op_name)
* [output](./callschema#output)
* [parent\_id](./callschema#parent_id)
* [project\_id](./callschema#project_id)
* [started\_at](./callschema#started_at)
* [summary](./callschema#summary)
* [trace\_id](./callschema#trace_id)
* [wb\_run\_id](./callschema#wb_run_id)
* [wb\_user\_id](./callschema#wb_user_id)

## Properties

### attributes

• **attributes**: `object`

Attributes

#### Defined in

[generated/traceServerApi.ts:120](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/generated/traceServerApi.ts#L120)

***

### deleted\_at

• `Optional` **deleted\_at**: `null` | `string`

Deleted At

#### Defined in

[generated/traceServerApi.ts:135](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/generated/traceServerApi.ts#L135)

***

### display\_name

• `Optional` **display\_name**: `null` | `string`

Display Name

#### Defined in

[generated/traceServerApi.ts:109](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/generated/traceServerApi.ts#L109)

***

### ended\_at

• `Optional` **ended\_at**: `null` | `string`

Ended At

#### Defined in

[generated/traceServerApi.ts:124](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/generated/traceServerApi.ts#L124)

***

### exception

• `Optional` **exception**: `null` | `string`

Exception

#### Defined in

[generated/traceServerApi.ts:126](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/generated/traceServerApi.ts#L126)

***

### id

• **id**: `string`

Id

#### Defined in

[generated/traceServerApi.ts:103](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/generated/traceServerApi.ts#L103)

***

### inputs

• **inputs**: `object`

Inputs

#### Defined in

[generated/traceServerApi.ts:122](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/generated/traceServerApi.ts#L122)

***

### op\_name

• **op\_name**: `string`

Op Name

#### Defined in

[generated/traceServerApi.ts:107](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/generated/traceServerApi.ts#L107)

***

### output

• `Optional` **output**: `null`

Output

#### Defined in

[generated/traceServerApi.ts:128](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/generated/traceServerApi.ts#L128)

***

### parent\_id

• `Optional` **parent\_id**: `null` | `string`

Parent Id

#### Defined in

[generated/traceServerApi.ts:113](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/generated/traceServerApi.ts#L113)

***

### project\_id

• **project\_id**: `string`

Project Id

#### Defined in

[generated/traceServerApi.ts:105](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/generated/traceServerApi.ts#L105)

***

### started\_at

• **started\_at**: `string`

Started At

`Format`

date-time

#### Defined in

[generated/traceServerApi.ts:118](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/generated/traceServerApi.ts#L118)

***

### summary

• `Optional` **summary**: `object`

#### Defined in

[generated/traceServerApi.ts:129](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/generated/traceServerApi.ts#L129)

***

### trace\_id

• **trace\_id**: `string`

Trace Id

#### Defined in

[generated/traceServerApi.ts:111](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/generated/traceServerApi.ts#L111)

***

### wb\_run\_id

• `Optional` **wb\_run\_id**: `null` | `string`

Wb Run Id

#### Defined in

[generated/traceServerApi.ts:133](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/generated/traceServerApi.ts#L133)

***

### wb\_user\_id

• `Optional` **wb\_user\_id**: `null` | `string`

Wb User Id

#### Defined in

[generated/traceServerApi.ts:131](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/generated/traceServerApi.ts#L131)


# Interface: CallsFilter
Source: https://docs.wandb.ai/weave/reference/typescript-sdk/interfaces/callsfilter

TypeScript SDK reference

[weave](../) / CallsFilter

CallsFilter

## Table of contents

### Properties

* [call\_ids](./callsfilter#call_ids)
* [input\_refs](./callsfilter#input_refs)
* [op\_names](./callsfilter#op_names)
* [output\_refs](./callsfilter#output_refs)
* [parent\_ids](./callsfilter#parent_ids)
* [trace\_ids](./callsfilter#trace_ids)
* [trace\_roots\_only](./callsfilter#trace_roots_only)
* [wb\_run\_ids](./callsfilter#wb_run_ids)
* [wb\_user\_ids](./callsfilter#wb_user_ids)

## Properties

### call\_ids

• `Optional` **call\_ids**: `null` | `string`\[]

Call Ids

#### Defined in

[generated/traceServerApi.ts:198](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/generated/traceServerApi.ts#L198)

***

### input\_refs

• `Optional` **input\_refs**: `null` | `string`\[]

Input Refs

#### Defined in

[generated/traceServerApi.ts:190](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/generated/traceServerApi.ts#L190)

***

### op\_names

• `Optional` **op\_names**: `null` | `string`\[]

Op Names

#### Defined in

[generated/traceServerApi.ts:188](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/generated/traceServerApi.ts#L188)

***

### output\_refs

• `Optional` **output\_refs**: `null` | `string`\[]

Output Refs

#### Defined in

[generated/traceServerApi.ts:192](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/generated/traceServerApi.ts#L192)

***

### parent\_ids

• `Optional` **parent\_ids**: `null` | `string`\[]

Parent Ids

#### Defined in

[generated/traceServerApi.ts:194](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/generated/traceServerApi.ts#L194)

***

### trace\_ids

• `Optional` **trace\_ids**: `null` | `string`\[]

Trace Ids

#### Defined in

[generated/traceServerApi.ts:196](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/generated/traceServerApi.ts#L196)

***

### trace\_roots\_only

• `Optional` **trace\_roots\_only**: `null` | `boolean`

Trace Roots Only

#### Defined in

[generated/traceServerApi.ts:200](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/generated/traceServerApi.ts#L200)

***

### wb\_run\_ids

• `Optional` **wb\_run\_ids**: `null` | `string`\[]

Wb Run Ids

#### Defined in

[generated/traceServerApi.ts:204](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/generated/traceServerApi.ts#L204)

***

### wb\_user\_ids

• `Optional` **wb\_user\_ids**: `null` | `string`\[]

Wb User Ids

#### Defined in

[generated/traceServerApi.ts:202](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/generated/traceServerApi.ts#L202)


# Use the Weave Service API to trace
Source: https://docs.wandb.ai/weave/cookbooks/weave_via_service_api

Learn how to use Weave tracing through the Service API with W&B Weave

<Note>
  This is an interactive notebook. You can run it locally or use the following links:

  * [Open in Google Colab](https://colab.research.google.com/github/wandb/docs/blob/main/weave/cookbooks/source/weave_via_service_api.ipynb)
  * [View source on GitHub](https://github.com/wandb/docs/blob/main/weave/cookbooks/source/weave_via_service_api.ipynb)
</Note>

In the following notebook, you learn how to use the W\&B Weave Service API to log traces to:

1. [Create a mock of a simple LLM call and response, and log it to Weave.](#simple-trace)
2. [Create a mock of a more complex LLM call and response, and log it to Weave.](#complex-trace)
3. [Run a sample lookup query on the logged traces.](#run-a-lookup-query)

> **View logged traces**
>
> To view all of the Weave traces created when you run the code in this guide, go to the **Traces** tab in your Weave project (specified by `team_id\project_id`), and select the name of the trace.

Before beginning, complete the [prerequisites](#prerequisites-set-variables-and-endpoints).

## Prerequisites: Set variables and endpoints

Before logging traces, you must configure the Service API endpoints and authenticate with your W\&B credentials.

The following code sets the URL endpoints used to access the Service API:

* [`https://trace.wandb.ai/call/start`](https://docs.wandb.ai/weave/reference/service-api/calls/call-start)
* [`https://trace.wandb.ai/call/end`](https://docs.wandb.ai/weave/reference/service-api/calls/call-end)
* [`https://trace.wandb.ai/calls/stream_query`](https://docs.wandb.ai/weave/reference/service-api/calls/calls-query-stream)

Additionally, you must set the following variables:

* `project_id`: the name of the W\&B project where you want to log your traces.
* `team_id`: your W\&B team name.
* `wandb_token`: your [W\&B API key](https://wandb.ai/settings).

```python lines theme={null}
import datetime
import json

import requests

# Headers and URLs
headers = {"Content-Type": "application/json"}
url_start = "https://trace.wandb.ai/call/start"
url_end = "https://trace.wandb.ai/call/end"
url_stream_query = "https://trace.wandb.ai/calls/stream_query"

# W&B variables
team_id = ""
project_id = ""
wandb_token = ""
```

## Simple trace

The following sections walk you through creating a simple trace:

1. [Start a simple trace](#start-a-simple-trace)
2. [End a simple trace](#end-a-simple-trace)

### Start a simple trace

The following code creates a sample LLM call `payload_start` and logs it to Weave using the `url_start` endpoint. The `payload_start` object mimics a call to OpenAI's `gpt-4o` with the query `Why is the sky blue?`.

On success, this code outputs a message indicating that the trace started:

```
Call started. ID: 01939cdc-38d2-7d61-940d-dcca0a56c575, Trace ID: 01939cdc-38d2-7d61-940d-dcd0e76c5f34
python
## ------------
## Start trace
## ------------
payload_start = {
    "start": {
        "project_id": f"{team_id}/{project_id}",
        "op_name": "simple_trace",
        "started_at": datetime.datetime.now().isoformat(),
        "inputs": {
            # Use this "messages" style to generate the chat UI in the expanded trace.
            "messages": [{"role": "user", "content": "Why is the sky blue?"}],
            "model": "gpt-4o",
        },
        "attributes": {},
    }
}
response = requests.post(
    url_start, headers=headers, json=payload_start, auth=("api", wandb_token)
)
if response.status_code == 200:
    data = response.json()
    call_id = data.get("id")
    trace_id = data.get("trace_id")
    print(f"Call started. ID: {call_id}, Trace ID: {trace_id}")
else:
    print("Start request failed with status:", response.status_code)
    print(response.text)
    exit()
```

### End a simple trace

To complete the simple trace, the following code creates a sample LLM call `payload_end` and logs it to Weave using the `url_end` endpoint. The `payload_end` object mimics the response from OpenAI's `gpt-4o` given the query `Why is the sky blue?`. The object is formatted so that pricing summary information and the chat completion are generated in trace view in the Weave dashboard.

On success, this code outputs a message indicating that the trace completed:

```
Call ended.
python
## ------------
## End trace
## ------------
payload_end = {
    "end": {
        "project_id": f"{team_id}/{project_id}",
        "id": call_id,
        "ended_at": datetime.datetime.now().isoformat(),
        "output": {
            # Use this "choices" style to add the completion to the chat UI in the expanded trace.
            "choices": [
                {
                    "message": {
                        "content": "It’s due to Rayleigh scattering, where shorter blue wavelengths of sunlight scatter in all directions."
                    }
                },
            ]
        },
        # Format the summary like this to generate the pricing summary information in the traces table.
        "summary": {
            "usage": {
                "gpt-4o": {
                    "prompt_tokens": 10,
                    "completion_tokens": 20,
                    "total_tokens": 30,
                    "requests": 1,
                }
            }
        },
    }
}
response = requests.post(
    url_end, headers=headers, json=payload_end, auth=("api", wandb_token)
)
if response.status_code == 200:
    print("Call ended.")
else:
    print("End request failed with status:", response.status_code)
    print(response.text)
```

## Complex trace

Now that you've logged a simple, single-step trace, the following sections walk you through creating a more complex trace with child spans, similar to a multi-operation RAG lookup:

1. [Start a complex trace](#start-a-complex-trace)
2. [Add a child span for a RAG document lookup](#add-a-child-span-for-a-rag-document-lookup)
3. [Add a child span for an LLM completion call](#add-a-child-span-for-an-llm-completion-call)
4. [End a complex trace](#end-a-complex-trace)

### Start a complex trace

The following code demonstrates how to create a more complex trace with multiple spans. An example is a Retrieval-Augmented Generation (RAG) lookup followed by an LLM call. The first part initializes a parent trace (`payload_parent_start`) that represents the operation. In this case, the operation processes the user query `Can you summarize the key points of this document?`.

The `payload_parent_start` object mimics the initial step in a multi-step workflow, logging the operation in Weave using the `url_start` endpoint.

On success, this code outputs a message indicating that the parent call is logged:

```
Parent call started. ID: 01939d26-0844-7c43-94bb-cdc471b6d65f, Trace ID: 01939d26-0844-7c43-94bb-cdd97dc296c8
python
## ------------
## Start trace (parent)
## ------------

# Parent call: Start
payload_parent_start = {
    "start": {
        "project_id": f"{team_id}/{project_id}",
        "op_name": "complex_trace",
        "started_at": datetime.datetime.now().isoformat(),
        "inputs": {"question": "Can you summarize the key points of this document?"},
        "attributes": {},
    }
}
response = requests.post(
    url_start, headers=headers, json=payload_parent_start, auth=("api", wandb_token)
)
if response.status_code == 200:
    data = response.json()
    parent_call_id = data.get("id")
    trace_id = data.get("trace_id")
    print(f"Parent call started. ID: {parent_call_id}, Trace ID: {trace_id}")
else:
    print("Parent start request failed with status:", response.status_code)
    print(response.text)
    exit()
```

### Add a child span for a RAG document lookup

The following code demonstrates how to add a child span to the complex parent trace started in the previous step. This step models the RAG document lookup sub-operation in the workflow.

The child trace is initiated with the `payload_child_start` object, which includes:

* `trace_id`: links this child span to the parent trace.
* `parent_id`: associates the child span with the parent operation.
* `inputs`: logs the search query, for example,
  `"This is a search query of the documents I'm looking for."`

On a successful call to the `url_start` endpoint, the code outputs a message indicating that the child call was started and completed:

```
Child call started. ID: 01939d32-23d6-75f2-9128-36a4a806f179
Child call ended.
python
## ------------
## Child span:
## Ex. RAG Document lookup
## ------------

# Child call: Start
payload_child_start = {
    "start": {
        "project_id": f"{team_id}/{project_id}",
        "op_name": "rag_document_lookup",
        "trace_id": trace_id,
        "parent_id": parent_call_id,
        "started_at": datetime.datetime.now().isoformat(),
        "inputs": {
            "document_search": "This is a search query of the documents I'm looking for."
        },
        "attributes": {},
    }
}
response = requests.post(
    url_start, headers=headers, json=payload_child_start, auth=("api", wandb_token)
)
if response.status_code == 200:
    data = response.json()
    child_call_id = data.get("id")
    print(f"Child call started. ID: {child_call_id}")
else:
    print("Child start request failed with status:", response.status_code)
    print(response.text)
    exit()

# Child call: End
payload_child_end = {
    "end": {
        "project_id": f"{team_id}/{project_id}",
        "id": child_call_id,
        "ended_at": datetime.datetime.now().isoformat(),
        "output": {
            "document_results": "This will be the RAG'd document text which will be returned from the search query."
        },
        "summary": {},
    }
}
response = requests.post(
    url_end, headers=headers, json=payload_child_end, auth=("api", wandb_token)
)
if response.status_code == 200:
    print("Child call ended.")
else:
    print("Child end request failed with status:", response.status_code)
    print(response.text)
```

### Add a child span for an LLM completion call

The following code demonstrates how to add another child span to the complex parent trace, representing an LLM completion call. This step models the LLM response generation based on document context retrieved in the previous RAG operation.

The LLM completion trace is initiated with the `payload_child_start` object, which includes:

* `trace_id`: links this child span to the parent trace.
* `parent_id`: associates the child span with the parent workflow.
* `inputs`: logs the input messages for the LLM, including the user query and the appended document context.
* `model`: specifies the model used for the operation (`gpt-4o`).

On success, the code outputs a message indicating the LLM child span trace has started:

```
Child call started. ID: 0245acdf-83a9-4c90-90df-dcb2b89f234a
```

After the operation completes, the `payload_child_end` object ends the trace by logging the LLM-generated response in the `output` field. The code also logs usage summary information.

On success, the code outputs a message indicating the LLM child span trace has started and ended:

```
Child call started. ID: 0245acdf-83a9-4c90-90df-dcb2b89f234a
Child call ended.
python
## ------------
## Child span:
## Create an LLM completion call
## ------------

# Child call: Start
payload_child_start = {
    "start": {
        "project_id": f"{team_id}/{project_id}",
        "op_name": "llm_completion",
        "trace_id": trace_id,
        "parent_id": parent_call_id,
        "started_at": datetime.datetime.now().isoformat(),
        "inputs": {
            "messages": [
                {
                    "role": "user",
                    "content": "With the following document context, could you help me answer:\n Can you summarize the key points of this document?\n [+ appended document context]",
                }
            ],
            "model": "gpt-4o",
        },
        "attributes": {},
    }
}
response = requests.post(
    url_start, headers=headers, json=payload_child_start, auth=("api", wandb_token)
)
if response.status_code == 200:
    data = response.json()
    child_call_id = data.get("id")
    print(f"Child call started. ID: {child_call_id}")
else:
    print("Child start request failed with status:", response.status_code)
    print(response.text)
    exit()

# Child call: End
payload_child_end = {
    "end": {
        "project_id": f"{team_id}/{project_id}",
        "id": child_call_id,
        "ended_at": datetime.datetime.now().isoformat(),
        "output": {
            "choices": [
                {"message": {"content": "This is the response generated by the LLM."}},
            ]
        },
        "summary": {
            "usage": {
                "gpt-4o": {
                    "prompt_tokens": 10,
                    "completion_tokens": 20,
                    "total_tokens": 30,
                    "requests": 1,
                }
            }
        },
    }
}
response = requests.post(
    url_end, headers=headers, json=payload_child_end, auth=("api", wandb_token)
)
if response.status_code == 200:
    print("Child call ended.")
else:
    print("Child end request failed with status:", response.status_code)
    print(response.text)
```

### End a complex trace

The following code demonstrates how to finalize the parent trace, marking the completion of the workflow. This step aggregates the results of all child spans (for example, RAG lookup and LLM completion) and logs the final output and metadata.

The trace is finalized using the `payload_parent_end` object, which includes:

* `id`: the `parent_call_id` from the initial parent trace start.
* `output`: represents the final output of the workflow.
* `summary`: consolidates usage data for the workflow.
* `prompt_tokens`: total tokens used for all prompts.
* `completion_tokens`: total tokens generated in all responses.
* `total_tokens`: combined token count for the workflow.
* `requests`: total number of requests made (in this case, `1`).

On success, the code outputs:

```
Parent call ended.
python
## ------------
## End trace
## ------------

# Parent call: End
payload_parent_end = {
    "end": {
        "project_id": f"{team_id}/{project_id}",
        "id": parent_call_id,
        "ended_at": datetime.datetime.now().isoformat(),
        "output": {
            "choices": [
                {"message": {"content": "This is the response generated by the LLM."}},
            ]
        },
        "summary": {
            "usage": {
                "gpt-4o": {
                    "prompt_tokens": 10,
                    "completion_tokens": 20,
                    "total_tokens": 30,
                    "requests": 1,
                }
            }
        },
    }
}
response = requests.post(
    url_end, headers=headers, json=payload_parent_end, auth=("api", wandb_token)
)
if response.status_code == 200:
    print("Parent call ended.")
else:
    print("Parent end request failed with status:", response.status_code)
    print(response.text)
```

## Run a lookup query

With traces logged to Weave, you can use the Service API to query them programmatically. The following code demonstrates how to query the traces created in previous examples, filtering only for traces where the `inputs.model` field is equal to `gpt-4o`.

The `query_payload` object includes:

* `project_id`: identifies the team and project to query.
* `filter`: ensures the query returns only trace roots (top-level traces).
* `query`: defines the filter logic using the `$expr` operator.
  * `$getField`: retrieves the `inputs.model` field.
  * `$literal`: matches traces where `inputs.model` equals `"gpt-4o"`.
* `limit`: limits the query to 10,000 results.
* `offset`: starts the query at the first result.
* `sort_by`: orders results by the `started_at` timestamp in descending order.
* `include_feedback`: excludes feedback data from the results.

On a successful query, the response includes trace data matching the query parameters:

```
{'id': '01939cf3-541f-76d3-ade3-50cfae068b39', 'project_id': 'cool-new-team/uncategorized', 'op_name': 'simple_trace', 'display_name': None, 'trace_id': '01939cf3-541f-76d3-ade3-50d5cfabe2db', 'parent_id': None, 'started_at': '2024-12-06T17:10:12.590000Z', 'attributes': {}, 'inputs': {'messages': [{'role': 'user', 'content': 'Why is the sky blue?'}], 'model': 'gpt-4o'}, 'ended_at': '2024-12-06T17:47:08.553000Z', 'exception': None, 'output': {'choices': [{'message': {'content': 'It’s due to Rayleigh scattering, where shorter blue wavelengths of sunlight scatter in all directions.'}}]}, 'summary': {'usage': {'gpt-4o': {'prompt_tokens': 10, 'completion_tokens': 20, 'requests': 1, 'total_tokens': 30}}, 'weave': {'status': 'success', 'trace_name': 'simple_trace', 'latency_ms': 2215963}}, 'wb_user_id': 'VXNlcjoyMDk5Njc0', 'wb_run_id': None, 'deleted_at': None}
python
query_payload = {
    "project_id": f"{team_id}/{project_id}",
    "filter": {"trace_roots_only": True},
    "query": {
        "$expr": {"$eq": [{"$getField": "inputs.model"}, {"$literal": "gpt-4o"}]}
    },
    "limit": 10000,
    "offset": 0,
    "sort_by": [{"field": "started_at", "direction": "desc"}],
    "include_feedback": False,
}
response = requests.post(
    url_stream_query, headers=headers, json=query_payload, auth=("api", wandb_token)
)
if response.status_code == 200:
    print("Query successful!")
    try:
        data = response.json()
        print(data)
    except json.JSONDecodeError as e:
        # Alternate decode
        json_objects = response.text.strip().split("\n")
        parsed_data = [json.loads(obj) for obj in json_objects]
        print(parsed_data)
else:
    print(f"Query failed with status code: {response.status_code}")
    print(response.text)
```


# Service API overview
Source: https://docs.wandb.ai/weave/reference/service-api

Use the Weave Service API, which provides REST endpoints for programmatic access to Weave functionality.

## Base URL

For [W\&B Multi-tenant Cloud](/platform/hosting/hosting-options/multi_tenant_cloud) and [W\&B Dedicated Cloud](/platform/hosting/hosting-options/dedicated-cloud), use the following base URL:

```
https://trace.wandb.ai
```

For [W\&B Self-Managed instances](/weave/guides/platform/weave-self-managed), use:

```
https://<your-subdomain>.wandb.io/traces
```

## Authentication

Include your W\&B API key in the request headers:

```bash theme={null}
curl -H "Authorization: Bearer YOUR_API_KEY" https://trace.wandb.ai/...
```

## Available Endpoints

### Calls

* **[POST `/call/end`](https://docs.wandb.ai/weave/reference/service-api/calls/call-end)** - Call End
* **[POST `/call/read`](https://docs.wandb.ai/weave/reference/service-api/calls/call-read)** - Call Read
* **[POST `/call/start`](https://docs.wandb.ai/weave/reference/service-api/calls/call-start)** - Call Start
* **[POST `/call/update`](https://docs.wandb.ai/weave/reference/service-api/calls/call-update)** - Call Update
* **[POST `/call/upsert_batch`](https://docs.wandb.ai/weave/reference/service-api/calls/call-start-batch)** - Call Start Batch
* **[POST `/calls/delete`](https://docs.wandb.ai/weave/reference/service-api/calls/calls-delete)** - Calls Delete
* **[POST `/calls/query_stats`](https://docs.wandb.ai/weave/reference/service-api/calls/calls-query-stats)** - Calls Query Stats
* **[POST `/calls/stats`](https://docs.wandb.ai/weave/reference/service-api/calls/call-stats)** - Call Stats
* **[POST `/calls/stream_query`](https://docs.wandb.ai/weave/reference/service-api/calls/calls-query-stream)** - Calls Query Stream
* **[POST `/calls/usage`](https://docs.wandb.ai/weave/reference/service-api/calls/calls-usage)** - Calls Usage
* **[POST `/trace/usage`](https://docs.wandb.ai/weave/reference/service-api/calls/trace-usage)** - Trace Usage

### Costs

* **[POST `/cost/create`](https://docs.wandb.ai/weave/reference/service-api/costs/cost-create)** - Cost Create
* **[POST `/cost/purge`](https://docs.wandb.ai/weave/reference/service-api/costs/cost-purge)** - Cost Purge
* **[POST `/cost/query`](https://docs.wandb.ai/weave/reference/service-api/costs/cost-query)** - Cost Query

### Feedback

* **[POST `/feedback/batch/create`](https://docs.wandb.ai/weave/reference/service-api/feedback/feedback-create-batch)** - Feedback Create Batch
* **[POST `/feedback/create`](https://docs.wandb.ai/weave/reference/service-api/feedback/feedback-create)** - Feedback Create
* **[POST `/feedback/payload_schema`](https://docs.wandb.ai/weave/reference/service-api/feedback/feedback-payload-schema)** - Feedback Payload Schema
* **[POST `/feedback/purge`](https://docs.wandb.ai/weave/reference/service-api/feedback/feedback-purge)** - Feedback Purge
* **[POST `/feedback/query`](https://docs.wandb.ai/weave/reference/service-api/feedback/feedback-query)** - Feedback Query
* **[POST `/feedback/replace`](https://docs.wandb.ai/weave/reference/service-api/feedback/feedback-replace)** - Feedback Replace
* **[POST `/feedback/stats`](https://docs.wandb.ai/weave/reference/service-api/feedback/feedback-stats)** - Feedback Stats

### Files

* **[POST `/file/content`](https://docs.wandb.ai/weave/reference/service-api/files/file-content)** - File Content
* **[POST `/file/create`](https://docs.wandb.ai/weave/reference/service-api/files/file-create)** - File Create
* **[POST `/files/query_stats`](https://docs.wandb.ai/weave/reference/service-api/files/files-stats)** - Files Stats

### Objects

* **[GET `/aliases`](https://docs.wandb.ai/weave/reference/service-api/objects/aliases-list)** - Aliases List
* **[POST `/obj/create`](https://docs.wandb.ai/weave/reference/service-api/objects/obj-create)** - Obj Create
* **[POST `/obj/delete`](https://docs.wandb.ai/weave/reference/service-api/objects/obj-delete)** - Obj Delete
* **[POST `/obj/read`](https://docs.wandb.ai/weave/reference/service-api/objects/obj-read)** - Obj Read
* **[POST `/objs/query`](https://docs.wandb.ai/weave/reference/service-api/objects/objs-query)** - Objs Query
* **[PUT `/objs/{object_id}/aliases`](https://docs.wandb.ai/weave/reference/service-api/objects/obj-set-aliases)** - Obj Set Aliases
* **[POST `/objs/{object_id}/aliases/remove`](https://docs.wandb.ai/weave/reference/service-api/objects/obj-remove-aliases)** - Obj Remove Aliases
* **[PUT `/objs/{object_id}/versions/{digest}/tags`](https://docs.wandb.ai/weave/reference/service-api/objects/obj-add-tags)** - Obj Add Tags
* **[POST `/objs/{object_id}/versions/{digest}/tags/remove`](https://docs.wandb.ai/weave/reference/service-api/objects/obj-remove-tags)** - Obj Remove Tags
* **[GET `/tags`](https://docs.wandb.ai/weave/reference/service-api/objects/tags-list)** - Tags List

### OpenTelemetry

* **[POST `/otel/v1/traces`](https://docs.wandb.ai/weave/reference/service-api/opentelemetry/export-trace)** - Export Trace

### Refs

* **[POST `/refs/read_batch`](https://docs.wandb.ai/weave/reference/service-api/refs/refs-read-batch)** - Refs Read Batch

### Service

* **[GET `/geolocate`](https://docs.wandb.ai/weave/reference/service-api/service/get-caller-location)** - Get Caller Location
* **[GET `/health`](https://docs.wandb.ai/weave/reference/service-api/service/read-root)** - Read Root
* **[GET `/server_info`](https://docs.wandb.ai/weave/reference/service-api/service/server-info)** - Server Info
* **[POST `/service/projects_info`](https://docs.wandb.ai/weave/reference/service-api/service/projects-info)** - Projects Info
* **[GET `/version`](https://docs.wandb.ai/weave/reference/service-api/service/read-version)** - Read Version

### Tables

* **[POST `/table/create`](https://docs.wandb.ai/weave/reference/service-api/tables/table-create)** - Table Create
* **[POST `/table/create_from_digests`](https://docs.wandb.ai/weave/reference/service-api/tables/table-create-from-digests)** - Table Create From Digests
* **[POST `/table/query`](https://docs.wandb.ai/weave/reference/service-api/tables/table-query)** - Table Query
* **[POST `/table/query_stats`](https://docs.wandb.ai/weave/reference/service-api/tables/table-query-stats)** - Table Query Stats
* **[POST `/table/query_stats_batch`](https://docs.wandb.ai/weave/reference/service-api/tables/table-query-stats-batch)** - Table Query Stats Batch
* **[POST `/table/update`](https://docs.wandb.ai/weave/reference/service-api/tables/table-update)** - Table Update

### Threads

* **[POST `/threads/stream_query`](https://docs.wandb.ai/weave/reference/service-api/threads/threads-query-stream)** - Threads Query Stream

### Inference

* **[GET `/inference/analysis/artificialanalysis/models`](https://docs.wandb.ai/weave/reference/service-api/inference/inference-analysis-artificialanalysis-models)** - Inference Analysis Artificialanalysis Models
* **[GET `/inference/catalog/models`](https://docs.wandb.ai/weave/reference/service-api/inference/inference-catalog-models)** - Inference Catalog Models
* **[GET `/inference/nvidia/v2/hardware`](https://docs.wandb.ai/weave/reference/service-api/inference/nvidia-hardware)** - Nvidia Hardware
* **[GET `/inference/router/openrouter/models`](https://docs.wandb.ai/weave/reference/service-api/inference/inference-router-openrouter-models)** - Inference Router Openrouter Models

### Agents

* **[POST `/agents/agent-versions/query`](https://docs.wandb.ai/weave/reference/service-api/agents/genai-agent-versions-query)** - Genai Agent Versions Query
* **[POST `/agents/conversations/chat`](https://docs.wandb.ai/weave/reference/service-api/agents/genai-conversation-chat)** - Genai Conversation Chat
* **[POST `/agents/otel/v1/traces`](https://docs.wandb.ai/weave/reference/service-api/agents/export-genai-trace)** - Export Genai Trace
* **[POST `/agents/query`](https://docs.wandb.ai/weave/reference/service-api/agents/genai-agents-query)** - Genai Agents Query
* **[POST `/agents/search`](https://docs.wandb.ai/weave/reference/service-api/agents/genai-search)** - Genai Search
* **[POST `/agents/spans/custom-attrs/schema`](https://docs.wandb.ai/weave/reference/service-api/agents/genai-custom-attrs-schema)** - Genai Custom Attrs Schema
* **[POST `/agents/spans/query`](https://docs.wandb.ai/weave/reference/service-api/agents/genai-spans-query)** - Genai Spans Query
* **[POST `/agents/spans/stats`](https://docs.wandb.ai/weave/reference/service-api/agents/genai-spans-stats)** - Genai Spans Stats
* **[POST `/agents/traces/chat`](https://docs.wandb.ai/weave/reference/service-api/agents/genai-traces-chat)** - Genai Traces Chat

### Annotation Queues

* **[POST `/annotation_queues`](https://docs.wandb.ai/weave/reference/service-api/annotation-queues/annotation-queue-create)** - Annotation Queue Create
* **[POST `/annotation_queues/query`](https://docs.wandb.ai/weave/reference/service-api/annotation-queues/annotation-queues-query-stream)** - Annotation Queues Query Stream
* **[POST `/annotation_queues/stats`](https://docs.wandb.ai/weave/reference/service-api/annotation-queues/annotation-queues-stats)** - Annotation Queues Stats
* **[DELETE `/annotation_queues/{queue_id}`](https://docs.wandb.ai/weave/reference/service-api/annotation-queues/annotation-queue-delete)** - Annotation Queue Delete
* **[GET `/annotation_queues/{queue_id}`](https://docs.wandb.ai/weave/reference/service-api/annotation-queues/annotation-queue-read)** - Annotation Queue Read
* **[PUT `/annotation_queues/{queue_id}`](https://docs.wandb.ai/weave/reference/service-api/annotation-queues/annotation-queue-update)** - Annotation Queue Update
* **[POST `/annotation_queues/{queue_id}/items`](https://docs.wandb.ai/weave/reference/service-api/annotation-queues/annotation-queue-add-calls)** - Annotation Queue Add Calls
* **[POST `/annotation_queues/{queue_id}/items/query`](https://docs.wandb.ai/weave/reference/service-api/annotation-queues/annotation-queue-items-query)** - Annotation Queue Items Query
* **[POST `/annotation_queues/{queue_id}/items/{item_id}/progress`](https://docs.wandb.ai/weave/reference/service-api/annotation-queues/annotation-queue-item-progress-update)** - Annotation Queue Item Progress Update

### Datasets

* **[GET `/v2/{entity}/{project}/datasets`](https://docs.wandb.ai/weave/reference/service-api/datasets/dataset-list)** - Dataset List
* **[POST `/v2/{entity}/{project}/datasets`](https://docs.wandb.ai/weave/reference/service-api/datasets/dataset-create)** - Dataset Create
* **[DELETE `/v2/{entity}/{project}/datasets/{object_id}`](https://docs.wandb.ai/weave/reference/service-api/datasets/dataset-delete)** - Dataset Delete
* **[GET `/v2/{entity}/{project}/datasets/{object_id}/versions/{digest}`](https://docs.wandb.ai/weave/reference/service-api/datasets/dataset-read)** - Dataset Read

### Eval Results

* **[POST `/v2/{entity}/{project}/eval_results/query`](https://docs.wandb.ai/weave/reference/service-api/eval-results/eval-results-query)** - Eval Results Query

### Evaluation Runs

* **[DELETE `/v2/{entity}/{project}/evaluation_runs`](https://docs.wandb.ai/weave/reference/service-api/evaluation-runs/evaluation-run-delete)** - Evaluation Run Delete
* **[GET `/v2/{entity}/{project}/evaluation_runs`](https://docs.wandb.ai/weave/reference/service-api/evaluation-runs/evaluation-run-list)** - Evaluation Run List
* **[POST `/v2/{entity}/{project}/evaluation_runs`](https://docs.wandb.ai/weave/reference/service-api/evaluation-runs/evaluation-run-create)** - Evaluation Run Create
* **[GET `/v2/{entity}/{project}/evaluation_runs/{evaluation_run_id}`](https://docs.wandb.ai/weave/reference/service-api/evaluation-runs/evaluation-run-read)** - Evaluation Run Read
* **[POST `/v2/{entity}/{project}/evaluation_runs/{evaluation_run_id}/finish`](https://docs.wandb.ai/weave/reference/service-api/evaluation-runs/evaluation-run-finish)** - Evaluation Run Finish

### Evaluations

* **[POST `/evaluations/evaluate_model`](https://docs.wandb.ai/weave/reference/service-api/evaluations/evaluate-model)** - Evaluate Model
* **[POST `/evaluations/rescore`](https://docs.wandb.ai/weave/reference/service-api/evaluations/rescore-evaluation)** - Rescore Evaluation
* **[POST `/evaluations/status`](https://docs.wandb.ai/weave/reference/service-api/evaluations/evaluation-status)** - Evaluation Status
* **[GET `/v2/{entity}/{project}/evaluations`](https://docs.wandb.ai/weave/reference/service-api/evaluations/evaluation-list)** - Evaluation List
* **[POST `/v2/{entity}/{project}/evaluations`](https://docs.wandb.ai/weave/reference/service-api/evaluations/evaluation-create)** - Evaluation Create
* **[DELETE `/v2/{entity}/{project}/evaluations/{object_id}`](https://docs.wandb.ai/weave/reference/service-api/evaluations/evaluation-delete)** - Evaluation Delete
* **[GET `/v2/{entity}/{project}/evaluations/{object_id}/versions/{digest}`](https://docs.wandb.ai/weave/reference/service-api/evaluations/evaluation-read)** - Evaluation Read

### Images

* **[POST `/image/create`](https://docs.wandb.ai/weave/reference/service-api/images/image-create)** - Image Create

### Models

* **[GET `/v2/{entity}/{project}/models`](https://docs.wandb.ai/weave/reference/service-api/models/model-list)** - Model List
* **[POST `/v2/{entity}/{project}/models`](https://docs.wandb.ai/weave/reference/service-api/models/model-create)** - Model Create
* **[DELETE `/v2/{entity}/{project}/models/{object_id}`](https://docs.wandb.ai/weave/reference/service-api/models/model-delete)** - Model Delete
* **[GET `/v2/{entity}/{project}/models/{object_id}/versions/{digest}`](https://docs.wandb.ai/weave/reference/service-api/models/model-read)** - Model Read

### Ops

* **[GET `/v2/{entity}/{project}/ops`](https://docs.wandb.ai/weave/reference/service-api/ops/op-list)** - Op List
* **[POST `/v2/{entity}/{project}/ops`](https://docs.wandb.ai/weave/reference/service-api/ops/op-create)** - Op Create
* **[DELETE `/v2/{entity}/{project}/ops/{object_id}`](https://docs.wandb.ai/weave/reference/service-api/ops/op-delete)** - Op Delete
* **[GET `/v2/{entity}/{project}/ops/{object_id}/versions/{digest}`](https://docs.wandb.ai/weave/reference/service-api/ops/op-read)** - Op Read

### Predictions

* **[DELETE `/v2/{entity}/{project}/predictions`](https://docs.wandb.ai/weave/reference/service-api/predictions/prediction-delete)** - Prediction Delete
* **[GET `/v2/{entity}/{project}/predictions`](https://docs.wandb.ai/weave/reference/service-api/predictions/prediction-list)** - Prediction List
* **[POST `/v2/{entity}/{project}/predictions`](https://docs.wandb.ai/weave/reference/service-api/predictions/prediction-create)** - Prediction Create
* **[GET `/v2/{entity}/{project}/predictions/{prediction_id}`](https://docs.wandb.ai/weave/reference/service-api/predictions/prediction-read)** - Prediction Read
* **[POST `/v2/{entity}/{project}/predictions/{prediction_id}/finish`](https://docs.wandb.ai/weave/reference/service-api/predictions/prediction-finish)** - Prediction Finish

### Registry

* **[POST `/link_to_registry`](https://docs.wandb.ai/weave/reference/service-api/registry/link-to-registry)** - Link To Registry

### Scorers

* **[GET `/v2/{entity}/{project}/scorers`](https://docs.wandb.ai/weave/reference/service-api/scorers/scorer-list)** - Scorer List
* **[POST `/v2/{entity}/{project}/scorers`](https://docs.wandb.ai/weave/reference/service-api/scorers/scorer-create)** - Scorer Create
* **[DELETE `/v2/{entity}/{project}/scorers/{object_id}`](https://docs.wandb.ai/weave/reference/service-api/scorers/scorer-delete)** - Scorer Delete
* **[GET `/v2/{entity}/{project}/scorers/{object_id}/versions/{digest}`](https://docs.wandb.ai/weave/reference/service-api/scorers/scorer-read)** - Scorer Read

### Scores

* **[POST `/calls/score`](https://docs.wandb.ai/weave/reference/service-api/scores/calls-score)** - Calls Score
* **[DELETE `/v2/{entity}/{project}/scores`](https://docs.wandb.ai/weave/reference/service-api/scores/score-delete)** - Score Delete
* **[GET `/v2/{entity}/{project}/scores`](https://docs.wandb.ai/weave/reference/service-api/scores/score-list)** - Score List
* **[POST `/v2/{entity}/{project}/scores`](https://docs.wandb.ai/weave/reference/service-api/scores/score-create)** - Score Create
* **[GET `/v2/{entity}/{project}/scores/{score_id}`](https://docs.wandb.ai/weave/reference/service-api/scores/score-read)** - Score Read


# Export Genai Trace
Source: https://docs.wandb.ai/weave/reference/service-api/agents/export-genai-trace

/weave/reference/service-api/openapi.json post /agents/otel/v1/traces
Ingest OTel spans into the GenAI observability system.



# Genai Agent Versions Query
Source: https://docs.wandb.ai/weave/reference/service-api/agents/genai-agent-versions-query

/weave/reference/service-api/openapi.json post /agents/agent-versions/query



# Genai Agents Query
Source: https://docs.wandb.ai/weave/reference/service-api/agents/genai-agents-query

/weave/reference/service-api/openapi.json post /agents/query



# Genai Conversation Chat
Source: https://docs.wandb.ai/weave/reference/service-api/agents/genai-conversation-chat

/weave/reference/service-api/openapi.json post /agents/conversations/chat



# Genai Custom Attrs Schema
Source: https://docs.wandb.ai/weave/reference/service-api/agents/genai-custom-attrs-schema

/weave/reference/service-api/openapi.json post /agents/spans/custom-attrs/schema
Discover typed custom attribute keys on matching agent spans.



# Genai Search
Source: https://docs.wandb.ai/weave/reference/service-api/agents/genai-search

/weave/reference/service-api/openapi.json post /agents/search



# Genai Spans Query
Source: https://docs.wandb.ai/weave/reference/service-api/agents/genai-spans-query

/weave/reference/service-api/openapi.json post /agents/spans/query
Query agent spans, either as raw rows or grouped aggregates.



# Genai Spans Stats
Source: https://docs.wandb.ai/weave/reference/service-api/agents/genai-spans-stats

/weave/reference/service-api/openapi.json post /agents/spans/stats
Query chart-ready aggregations over agent spans.



# Genai Traces Chat
Source: https://docs.wandb.ai/weave/reference/service-api/agents/genai-traces-chat

/weave/reference/service-api/openapi.json post /agents/traces/chat



# Annotation Queue Add Calls
Source: https://docs.wandb.ai/weave/reference/service-api/annotation-queues/annotation-queue-add-calls

/weave/reference/service-api/openapi.json post /annotation_queues/{queue_id}/items
Add calls to an annotation queue.



# Annotation Queue Create
Source: https://docs.wandb.ai/weave/reference/service-api/annotation-queues/annotation-queue-create

/weave/reference/service-api/openapi.json post /annotation_queues
Create a new annotation queue.



# Annotation Queue Delete
Source: https://docs.wandb.ai/weave/reference/service-api/annotation-queues/annotation-queue-delete

/weave/reference/service-api/openapi.json delete /annotation_queues/{queue_id}
Delete (soft-delete) an annotation queue.



# Annotation Queue Item Progress Update
Source: https://docs.wandb.ai/weave/reference/service-api/annotation-queues/annotation-queue-item-progress-update

/weave/reference/service-api/openapi.json post /annotation_queues/{queue_id}/items/{item_id}/progress
Update the annotation state of a queue item for the current annotator.



# Annotation Queue Items Query
Source: https://docs.wandb.ai/weave/reference/service-api/annotation-queues/annotation-queue-items-query

/weave/reference/service-api/openapi.json post /annotation_queues/{queue_id}/items/query
Query items in an annotation queue with pagination and sorting.



# Annotation Queue Read
Source: https://docs.wandb.ai/weave/reference/service-api/annotation-queues/annotation-queue-read

/weave/reference/service-api/openapi.json get /annotation_queues/{queue_id}
Read a specific annotation queue.



# Annotation Queue Update
Source: https://docs.wandb.ai/weave/reference/service-api/annotation-queues/annotation-queue-update

/weave/reference/service-api/openapi.json put /annotation_queues/{queue_id}
Update an annotation queue's metadata (name, description, scorer_refs).



# Annotation Queues Query Stream
Source: https://docs.wandb.ai/weave/reference/service-api/annotation-queues/annotation-queues-query-stream

/weave/reference/service-api/openapi.json post /annotation_queues/query
Query annotation queues for a project (streaming NDJSON response).



# Annotation Queues Stats
Source: https://docs.wandb.ai/weave/reference/service-api/annotation-queues/annotation-queues-stats

/weave/reference/service-api/openapi.json post /annotation_queues/stats
Get stats for multiple annotation queues.



# Call End
Source: https://docs.wandb.ai/weave/reference/service-api/calls/call-end

/weave/reference/service-api/openapi.json post /call/end



# Call Read
Source: https://docs.wandb.ai/weave/reference/service-api/calls/call-read

/weave/reference/service-api/openapi.json post /call/read



# Call Start
Source: https://docs.wandb.ai/weave/reference/service-api/calls/call-start

/weave/reference/service-api/openapi.json post /call/start



# Call Start Batch
Source: https://docs.wandb.ai/weave/reference/service-api/calls/call-start-batch

/weave/reference/service-api/openapi.json post /call/upsert_batch



# Call Stats
Source: https://docs.wandb.ai/weave/reference/service-api/calls/call-stats

/weave/reference/service-api/openapi.json post /calls/stats



# Call Update
Source: https://docs.wandb.ai/weave/reference/service-api/calls/call-update

/weave/reference/service-api/openapi.json post /call/update



# Calls Delete
Source: https://docs.wandb.ai/weave/reference/service-api/calls/calls-delete

/weave/reference/service-api/openapi.json post /calls/delete



# Calls Query Stats
Source: https://docs.wandb.ai/weave/reference/service-api/calls/calls-query-stats

/weave/reference/service-api/openapi.json post /calls/query_stats



# Calls Query Stream
Source: https://docs.wandb.ai/weave/reference/service-api/calls/calls-query-stream

/weave/reference/service-api/openapi.json post /calls/stream_query



# Calls Usage
Source: https://docs.wandb.ai/weave/reference/service-api/calls/calls-usage

/weave/reference/service-api/openapi.json post /calls/usage
Compute aggregated usage for multiple root calls, with descendant rollup.



# Trace Usage
Source: https://docs.wandb.ai/weave/reference/service-api/calls/trace-usage

/weave/reference/service-api/openapi.json post /trace/usage
Compute per-call usage for a trace, with descendant rollup.



# Cost Create
Source: https://docs.wandb.ai/weave/reference/service-api/costs/cost-create

/weave/reference/service-api/openapi.json post /cost/create



# Cost Purge
Source: https://docs.wandb.ai/weave/reference/service-api/costs/cost-purge

/weave/reference/service-api/openapi.json post /cost/purge



# Cost Query
Source: https://docs.wandb.ai/weave/reference/service-api/costs/cost-query

/weave/reference/service-api/openapi.json post /cost/query



# Evaluate Model
Source: https://docs.wandb.ai/weave/reference/service-api/evaluations/evaluate-model

/weave/reference/service-api/openapi.json post /evaluations/evaluate_model



# Evaluation Create
Source: https://docs.wandb.ai/weave/reference/service-api/evaluations/evaluation-create

/weave/reference/service-api/openapi.json post /v2/{entity}/{project}/evaluations
Create an evaluation object.



# Evaluation Delete
Source: https://docs.wandb.ai/weave/reference/service-api/evaluations/evaluation-delete

/weave/reference/service-api/openapi.json delete /v2/{entity}/{project}/evaluations/{object_id}
Delete an evaluation object.



# Evaluation List
Source: https://docs.wandb.ai/weave/reference/service-api/evaluations/evaluation-list

/weave/reference/service-api/openapi.json get /v2/{entity}/{project}/evaluations
List evaluation objects.



# Evaluation Read
Source: https://docs.wandb.ai/weave/reference/service-api/evaluations/evaluation-read

/weave/reference/service-api/openapi.json get /v2/{entity}/{project}/evaluations/{object_id}/versions/{digest}
Get an evaluation object.



# Evaluation Status
Source: https://docs.wandb.ai/weave/reference/service-api/evaluations/evaluation-status

/weave/reference/service-api/openapi.json post /evaluations/status



# Rescore Evaluation
Source: https://docs.wandb.ai/weave/reference/service-api/evaluations/rescore-evaluation

/weave/reference/service-api/openapi.json post /evaluations/rescore
Rescore an existing evaluation run with different scorer(s).

Applies the provided scorer(s) to the predictions from source_evaluation_run_id
and returns a new evaluation_run_id. Original prediction call IDs are preserved.



# Feedback Create
Source: https://docs.wandb.ai/weave/reference/service-api/feedback/feedback-create

/weave/reference/service-api/openapi.json post /feedback/create
Add feedback to a call or object.



# Feedback Create Batch
Source: https://docs.wandb.ai/weave/reference/service-api/feedback/feedback-create-batch

/weave/reference/service-api/openapi.json post /feedback/batch/create
Add multiple feedback items to calls or objects.



# Feedback Payload Schema
Source: https://docs.wandb.ai/weave/reference/service-api/feedback/feedback-payload-schema

/weave/reference/service-api/openapi.json post /feedback/payload_schema
Discover feedback payload schema (paths and types) from sample rows.



# Feedback Purge
Source: https://docs.wandb.ai/weave/reference/service-api/feedback/feedback-purge

/weave/reference/service-api/openapi.json post /feedback/purge
Permanently delete feedback.



# Feedback Query
Source: https://docs.wandb.ai/weave/reference/service-api/feedback/feedback-query

/weave/reference/service-api/openapi.json post /feedback/query
Query for feedback.



# Feedback Replace
Source: https://docs.wandb.ai/weave/reference/service-api/feedback/feedback-replace

/weave/reference/service-api/openapi.json post /feedback/replace



# Feedback Stats
Source: https://docs.wandb.ai/weave/reference/service-api/feedback/feedback-stats

/weave/reference/service-api/openapi.json post /feedback/stats
Return aggregated feedback statistics over time buckets.



# File Content
Source: https://docs.wandb.ai/weave/reference/service-api/files/file-content

/weave/reference/service-api/openapi.json post /file/content



# File Create
Source: https://docs.wandb.ai/weave/reference/service-api/files/file-create

/weave/reference/service-api/openapi.json post /file/create



# Files Stats
Source: https://docs.wandb.ai/weave/reference/service-api/files/files-stats

/weave/reference/service-api/openapi.json post /files/query_stats



# Inference Analysis Artificialanalysis Models
Source: https://docs.wandb.ai/weave/reference/service-api/inference/inference-analysis-artificialanalysis-models

/weave/reference/service-api/openapi.json get /inference/analysis/artificialanalysis/models
Returns a list of available models for Artificial Analysis.

This API is available without authentication.



# Inference Catalog Models
Source: https://docs.wandb.ai/weave/reference/service-api/inference/inference-catalog-models

/weave/reference/service-api/openapi.json get /inference/catalog/models
Returns a list of available Serverless Inference models.

This API is available without authentication.



# Inference Router Openrouter Models
Source: https://docs.wandb.ai/weave/reference/service-api/inference/inference-router-openrouter-models

/weave/reference/service-api/openapi.json get /inference/router/openrouter/models
Returns a list of models that are available to be used with OpenRouter.

This API is available without authentication.



# Nvidia Hardware
Source: https://docs.wandb.ai/weave/reference/service-api/inference/nvidia-hardware

/weave/reference/service-api/openapi.json get /inference/nvidia/v2/hardware
Returns available hardware and pricing for a given model.

Called by NVIDIA to show users their options and redirect them
based on what we support.  Only serverless options are returned.



# Aliases List
Source: https://docs.wandb.ai/weave/reference/service-api/objects/aliases-list

/weave/reference/service-api/openapi.json get /aliases
List all aliases in a project.



# Obj Add Tags
Source: https://docs.wandb.ai/weave/reference/service-api/objects/obj-add-tags

/weave/reference/service-api/openapi.json put /objs/{object_id}/versions/{digest}/tags
Add tags to an object version.



# Obj Create
Source: https://docs.wandb.ai/weave/reference/service-api/objects/obj-create

/weave/reference/service-api/openapi.json post /obj/create



# Obj Delete
Source: https://docs.wandb.ai/weave/reference/service-api/objects/obj-delete

/weave/reference/service-api/openapi.json post /obj/delete



# Obj Read
Source: https://docs.wandb.ai/weave/reference/service-api/objects/obj-read

/weave/reference/service-api/openapi.json post /obj/read



# Obj Remove Aliases
Source: https://docs.wandb.ai/weave/reference/service-api/objects/obj-remove-aliases

/weave/reference/service-api/openapi.json post /objs/{object_id}/aliases/remove
Remove aliases from an object.



# Obj Remove Tags
Source: https://docs.wandb.ai/weave/reference/service-api/objects/obj-remove-tags

/weave/reference/service-api/openapi.json post /objs/{object_id}/versions/{digest}/tags/remove
Remove tags from an object version.



# Obj Set Aliases
Source: https://docs.wandb.ai/weave/reference/service-api/objects/obj-set-aliases

/weave/reference/service-api/openapi.json put /objs/{object_id}/aliases
Set aliases for an object version.



# Objs Query
Source: https://docs.wandb.ai/weave/reference/service-api/objects/objs-query

/weave/reference/service-api/openapi.json post /objs/query



# Tags List
Source: https://docs.wandb.ai/weave/reference/service-api/objects/tags-list

/weave/reference/service-api/openapi.json get /tags
List all tags in a project.



# Export Trace
Source: https://docs.wandb.ai/weave/reference/service-api/opentelemetry/export-trace

/weave/reference/service-api/openapi.json post /otel/v1/traces



# Refs Read Batch
Source: https://docs.wandb.ai/weave/reference/service-api/refs/refs-read-batch

/weave/reference/service-api/openapi.json post /refs/read_batch



# Calls Score
Source: https://docs.wandb.ai/weave/reference/service-api/scores/calls-score

/weave/reference/service-api/openapi.json post /calls/score



# Score Create
Source: https://docs.wandb.ai/weave/reference/service-api/scores/score-create

/weave/reference/service-api/openapi.json post /v2/{entity}/{project}/scores
Create a score.



# Score List
Source: https://docs.wandb.ai/weave/reference/service-api/scores/score-list

/weave/reference/service-api/openapi.json get /v2/{entity}/{project}/scores
List scores.



# Get Caller Location
Source: https://docs.wandb.ai/weave/reference/service-api/service/get-caller-location

/weave/reference/service-api/openapi.json get /geolocate
Lookup the geographic location of a user based on their IP address.

This API exists for debugging purposes and may not be available in the future.



# Projects Info
Source: https://docs.wandb.ai/weave/reference/service-api/service/projects-info

/weave/reference/service-api/openapi.json post /service/projects_info



# Read Root
Source: https://docs.wandb.ai/weave/reference/service-api/service/read-root

/weave/reference/service-api/openapi.json get /health



# Read Version
Source: https://docs.wandb.ai/weave/reference/service-api/service/read-version

/weave/reference/service-api/openapi.json get /version



# Server Info
Source: https://docs.wandb.ai/weave/reference/service-api/service/server-info

/weave/reference/service-api/openapi.json get /server_info



# Table Create
Source: https://docs.wandb.ai/weave/reference/service-api/tables/table-create

/weave/reference/service-api/openapi.json post /table/create



# Table Create From Digests
Source: https://docs.wandb.ai/weave/reference/service-api/tables/table-create-from-digests

/weave/reference/service-api/openapi.json post /table/create_from_digests



# Table Query
Source: https://docs.wandb.ai/weave/reference/service-api/tables/table-query

/weave/reference/service-api/openapi.json post /table/query



# Table Query Stats
Source: https://docs.wandb.ai/weave/reference/service-api/tables/table-query-stats

/weave/reference/service-api/openapi.json post /table/query_stats



# Table Query Stats Batch
Source: https://docs.wandb.ai/weave/reference/service-api/tables/table-query-stats-batch

/weave/reference/service-api/openapi.json post /table/query_stats_batch



# Table Update
Source: https://docs.wandb.ai/weave/reference/service-api/tables/table-update

/weave/reference/service-api/openapi.json post /table/update



# Threads Query Stream
Source: https://docs.wandb.ai/weave/reference/service-api/threads/threads-query-stream

/weave/reference/service-api/openapi.json post /threads/stream_query



# Interface: GetCallsOptions
Source: https://docs.wandb.ai/weave/reference/typescript-sdk/interfaces/getcallsoptions

TypeScript SDK reference

[weave](../) / GetCallsOptions

## Table of contents

### Properties

* [columns](./getcallsoptions#columns)
* [expandColumns](./getcallsoptions#expandcolumns)
* [filter](./getcallsoptions#filter)
* [includeCosts](./getcallsoptions#includecosts)
* [includeFeedback](./getcallsoptions#includefeedback)
* [limit](./getcallsoptions#limit)
* [offset](./getcallsoptions#offset)
* [query](./getcallsoptions#query)
* [sortBy](./getcallsoptions#sortby)

## Properties

### columns

• `Optional` **columns**: `string`\[]

#### Defined in

[weaveClient.ts:84](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/weaveClient.ts#L84)

***

### expandColumns

• `Optional` **expandColumns**: `string`\[]

#### Defined in

[weaveClient.ts:85](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/weaveClient.ts#L85)

***

### filter

• `Optional` **filter**: [`CallsFilter`](./callsfilter)

#### Defined in

[weaveClient.ts:77](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/weaveClient.ts#L77)

***

### includeCosts

• `Optional` **includeCosts**: `boolean`

#### Defined in

[weaveClient.ts:79](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/weaveClient.ts#L79)

***

### includeFeedback

• `Optional` **includeFeedback**: `boolean`

#### Defined in

[weaveClient.ts:80](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/weaveClient.ts#L80)

***

### limit

• `Optional` **limit**: `number`

#### Defined in

[weaveClient.ts:81](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/weaveClient.ts#L81)

***

### offset

• `Optional` **offset**: `number`

#### Defined in

[weaveClient.ts:82](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/weaveClient.ts#L82)

***

### query

• `Optional` **query**: [`Query`](./query)

#### Defined in

[weaveClient.ts:78](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/weaveClient.ts#L78)

***

### sortBy

• `Optional` **sortBy**: [`SortBy`](./sortby)\[]

#### Defined in

[weaveClient.ts:83](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/weaveClient.ts#L83)


# Interface: LLMInit
Source: https://docs.wandb.ai/weave/reference/typescript-sdk/interfaces/llminit

TypeScript SDK reference

[weave](../) / LLMInit

## Table of contents

### Properties

* [model](./llminit#model)
* [providerName](./llminit#providername)

## Properties

### model

• **model**: `string`

#### Defined in

[genai/llm.ts:31](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/genai/llm.ts#L31)

***

### providerName

• `Optional` **providerName**: `string`

#### Defined in

[genai/llm.ts:32](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/genai/llm.ts#L32)


# Interface: Message
Source: https://docs.wandb.ai/weave/reference/typescript-sdk/interfaces/message

TypeScript SDK reference

[weave](../) / Message

## Table of contents

### Properties

* [content](./message#content)
* [parts](./message#parts)
* [role](./message#role)
* [toolCallId](./message#toolcallid)
* [toolName](./message#toolname)

## Properties

### content

• `Optional` **content**: `string`

#### Defined in

[genai/types.ts:11](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/genai/types.ts#L11)

***

### parts

• `Optional` **parts**: [`MessagePart`](../#messagepart)\[]

#### Defined in

[genai/types.ts:14](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/genai/types.ts#L14)

***

### role

• **role**: [`Role`](../#role)

#### Defined in

[genai/types.ts:10](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/genai/types.ts#L10)

***

### toolCallId

• `Optional` **toolCallId**: `string`

#### Defined in

[genai/types.ts:12](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/genai/types.ts#L12)

***

### toolName

• `Optional` **toolName**: `string`

#### Defined in

[genai/types.ts:13](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/genai/types.ts#L13)


# Interface: Query
Source: https://docs.wandb.ai/weave/reference/typescript-sdk/interfaces/query

TypeScript SDK reference

[weave](../) / Query

Query

## Table of contents

### Properties

* \[$expr](./query#$expr)

## Properties

### \$expr

• **\$expr**: `AndOperation` | `OrOperation` | `NotOperation` | `EqOperation` | `GtOperation` | `GteOperation` | `InOperation` | `ContainsOperation`

\$Expr

#### Defined in

[generated/traceServerApi.ts:817](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/generated/traceServerApi.ts#L817)


# Interface: Reasoning
Source: https://docs.wandb.ai/weave/reference/typescript-sdk/interfaces/reasoning

TypeScript SDK reference

[weave](../) / Reasoning

## Table of contents

### Properties

* [content](./reasoning#content)

## Properties

### content

• **content**: `string`

#### Defined in

[genai/types.ts:40](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/genai/types.ts#L40)


# Interface: SessionInit
Source: https://docs.wandb.ai/weave/reference/typescript-sdk/interfaces/sessioninit

TypeScript SDK reference

[weave](../) / SessionInit

## Table of contents

### Properties

* [agentName](./sessioninit#agentname)
* [model](./sessioninit#model)
* [sessionId](./sessioninit#sessionid)

## Properties

### agentName

• `Optional` **agentName**: `string`

#### Defined in

[genai/session.ts:7](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/genai/session.ts#L7)

***

### model

• `Optional` **model**: `string`

#### Defined in

[genai/session.ts:8](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/genai/session.ts#L8)

***

### sessionId

• `Optional` **sessionId**: `string`

Conversation ID propagated to every span under this session as
`gen_ai.conversation.id`. Auto-generated if omitted.

#### Defined in

[genai/session.ts:11](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/genai/session.ts#L11)


# Interface: SortBy
Source: https://docs.wandb.ai/weave/reference/typescript-sdk/interfaces/sortby

TypeScript SDK reference

[weave](../) / SortBy

SortBy

## Table of contents

### Properties

* [direction](./sortby#direction)
* [field](./sortby#field)

## Properties

### direction

• **direction**: `"asc"` | `"desc"`

Direction

#### Defined in

[generated/traceServerApi.ts:851](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/generated/traceServerApi.ts#L851)

***

### field

• **field**: `string`

Field

#### Defined in

[generated/traceServerApi.ts:849](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/generated/traceServerApi.ts#L849)


# Interface: SubAgentInit
Source: https://docs.wandb.ai/weave/reference/typescript-sdk/interfaces/subagentinit

TypeScript SDK reference

[weave](../) / SubAgentInit

## Table of contents

### Properties

* [model](./subagentinit#model)
* [name](./subagentinit#name)

## Properties

### model

• `Optional` **model**: `string`

#### Defined in

[genai/subagent.ts:15](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/genai/subagent.ts#L15)

***

### name

• **name**: `string`

#### Defined in

[genai/subagent.ts:14](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/genai/subagent.ts#L14)


# Interface: ToolInit
Source: https://docs.wandb.ai/weave/reference/typescript-sdk/interfaces/toolinit

TypeScript SDK reference

[weave](../) / ToolInit

## Table of contents

### Properties

* [args](./toolinit#args)
* [name](./toolinit#name)
* [toolCallId](./toolinit#toolcallid)

## Properties

### args

• `Optional` **args**: `string`

#### Defined in

[genai/tool.ts:17](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/genai/tool.ts#L17)

***

### name

• **name**: `string`

#### Defined in

[genai/tool.ts:16](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/genai/tool.ts#L16)

***

### toolCallId

• `Optional` **toolCallId**: `string`

#### Defined in

[genai/tool.ts:18](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/genai/tool.ts#L18)


# Interface: TurnInit
Source: https://docs.wandb.ai/weave/reference/typescript-sdk/interfaces/turninit

TypeScript SDK reference

[weave](../) / TurnInit

## Table of contents

### Properties

* [agentName](./turninit#agentname)
* [model](./turninit#model)

## Properties

### agentName

• `Optional` **agentName**: `string`

#### Defined in

[genai/turn.ts:27](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/genai/turn.ts#L27)

***

### model

• `Optional` **model**: `string`

#### Defined in

[genai/turn.ts:28](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/genai/turn.ts#L28)


# Interface: Usage
Source: https://docs.wandb.ai/weave/reference/typescript-sdk/interfaces/usage

TypeScript SDK reference

[weave](../) / Usage

## Table of contents

### Properties

* [cacheCreationInputTokens](./usage#cachecreationinputtokens)
* [cacheReadInputTokens](./usage#cachereadinputtokens)
* [inputTokens](./usage#inputtokens)
* [outputTokens](./usage#outputtokens)
* [reasoningTokens](./usage#reasoningtokens)

## Properties

### cacheCreationInputTokens

• `Optional` **cacheCreationInputTokens**: `number`

#### Defined in

[genai/types.ts:35](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/genai/types.ts#L35)

***

### cacheReadInputTokens

• `Optional` **cacheReadInputTokens**: `number`

#### Defined in

[genai/types.ts:36](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/genai/types.ts#L36)

***

### inputTokens

• `Optional` **inputTokens**: `number`

#### Defined in

[genai/types.ts:32](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/genai/types.ts#L32)

***

### outputTokens

• `Optional` **outputTokens**: `number`

#### Defined in

[genai/types.ts:33](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/genai/types.ts#L33)

***

### reasoningTokens

• `Optional` **reasoningTokens**: `number`

#### Defined in

[genai/types.ts:34](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/genai/types.ts#L34)


# Interface: WeaveAudio
Source: https://docs.wandb.ai/weave/reference/typescript-sdk/interfaces/weaveaudio

TypeScript SDK reference

[weave](../) / WeaveAudio

## Hierarchy

* `WeaveAudioInput`

  ↳ `WeaveAudio`

## Table of contents

### Properties

* [\_weaveType](./weaveaudio#_weavetype)
* [audioType](./weaveaudio#audiotype)
* [data](./weaveaudio#data)

## Properties

### \_weaveType

• **\_weaveType**: `"Audio"`

#### Defined in

[media.ts:48](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/media.ts#L48)

***

### audioType

• `Optional` **audioType**: `"wav"`

#### Inherited from

WeaveAudioInput.audioType

#### Defined in

[media.ts:44](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/media.ts#L44)

***

### data

• **data**: `Buffer`

#### Inherited from

WeaveAudioInput.data

#### Defined in

[media.ts:43](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/media.ts#L43)


# Interface: WeaveImage
Source: https://docs.wandb.ai/weave/reference/typescript-sdk/interfaces/weaveimage

TypeScript SDK reference

[weave](../) / WeaveImage

## Hierarchy

* `WeaveImageInput`

  ↳ `WeaveImage`

## Table of contents

### Properties

* [\_weaveType](./weaveimage#_weavetype)
* [data](./weaveimage#data)
* [imageType](./weaveimage#imagetype)

## Properties

### \_weaveType

• **\_weaveType**: `"Image"`

#### Defined in

[media.ts:14](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/media.ts#L14)

***

### data

• **data**: `Buffer`

#### Inherited from

WeaveImageInput.data

#### Defined in

[media.ts:9](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/media.ts#L9)

***

### imageType

• `Optional` **imageType**: `"png"`

#### Inherited from

WeaveImageInput.imageType

#### Defined in

[media.ts:10](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/media.ts#L10)


# MessagePart
Source: https://docs.wandb.ai/weave/reference/typescript-sdk/type-aliases/messagepart

TypeScript SDK reference

# MessagePart

Ƭ **MessagePart**: \{ `content`: `string` ; `type`: `"text"`  } | \{ `content`: `string` ; `type`: `"reasoning"`  } | \{ `arguments?`: `string` ; `toolCallId`: `string` ; `toolName`: `string` ; `type`: `"tool_call"`  } | \{ `result?`: `string` ; `toolCallId`: `string` ; `type`: `"tool_result"`  } | \{ `fileId`: `string` ; `mimeType?`: `string` ; `modality`: [`Modality`](./modality) ; `type`: `"file"`  } | \{ `content`: `string` ; `mimeType`: `string` ; `modality`: [`Modality`](./modality) ; `type`: `"blob"`  } | \{ `modality`: [`Modality`](./modality) ; `type`: `"uri"` ; `uri`: `string`  }

#### Defined in

[genai/types.ts:17](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/genai/types.ts#L17)

***


# Modality
Source: https://docs.wandb.ai/weave/reference/typescript-sdk/type-aliases/modality

TypeScript SDK reference

# Modality

Ƭ **Modality**: `"image"` | `"audio"` | `"video"` | `"document"`

#### Defined in

[genai/types.ts:7](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/genai/types.ts#L7)

***


# Op
Source: https://docs.wandb.ai/weave/reference/typescript-sdk/type-aliases/op

TypeScript SDK reference

# Op

Ƭ **Op**\<`T`>: \{ `__boundThis?`: [`WeaveObject`](../classes/weaveobject) ; `__isOp`: `true` ; `__name`: `string` ; `__parameterNames?`: `ParameterNamesOption` ; `__savedRef?`: `OpRef` | `Promise`\<`OpRef`> ; `__wrappedFunction`: `T` ; `invoke`: `CallMethod`\<`T`>  } & `T` & (...`args`: `Parameters`\<`T`>) => `ReturnType`\<`T`> extends `AsyncIterable`\<infer U> ? `AsyncIterable`\<`Awaited`\<`U`>> : `Promise`\<`Awaited`\<`ReturnType`\<`T`>>>

#### Type parameters

| Name | Type                                   |
| :--- | :------------------------------------- |
| `T`  | extends (...`args`: `any`\[]) => `any` |

#### Defined in

[opType.ts:7](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/opType.ts#L7)

***


# OpDecorator
Source: https://docs.wandb.ai/weave/reference/typescript-sdk/type-aliases/opdecorator

TypeScript SDK reference

# OpDecorator

Ƭ **OpDecorator**\<`T`>: (`value`: `T`, `context`: `ClassMethodDecoratorContext`) => `T` | `void` & (`target`: `Object`, `propertyKey`: `string` | `symbol`, `descriptor`: `TypedPropertyDescriptor`\<`T`>) => `TypedPropertyDescriptor`\<`T`> | `void`

Helper type for decorators
This represents a decorator function that can be used with both legacy and Stage 3 decorators.

For Stage 3 decorators:
target: The function being decorated (T)
context: MethodDecoratorContext

For legacy decorators:
target: The prototype (instance methods) or constructor (static methods)
propertyKey: The method name
descriptor: The property descriptor containing the method

#### Type parameters

| Name | Type                                   |
| :--- | :------------------------------------- |
| `T`  | extends (...`args`: `any`\[]) => `any` |

#### Defined in

[opType.ts:41](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/opType.ts#L41)

***


# Role
Source: https://docs.wandb.ai/weave/reference/typescript-sdk/type-aliases/role

TypeScript SDK reference

# Role

Ƭ **Role**: `"user"` | `"assistant"` | `"system"` | `"tool"`

Public data types for the Weave GenAI session SDK.

#### Defined in

[genai/types.ts:5](https://github.com/wandb/weave/blob/6538626556c93d6f31ae725fdefe4e6b8b71bc2c/sdks/node/src/genai/types.ts#L5)

## Functions


# API overview
Source: https://docs.wandb.ai/inference/api-reference

Explore the complete REST API reference for the Serverless Inference service, including endpoints and authentication.

This reference describes the Serverless Inference REST API, which lets you call foundation models programmatically from your own applications. Use it to integrate hosted inference into services, scripts, or notebooks without managing model infrastructure.

## Base URL

Access the Inference service at:

```plaintext theme={null}
https://api.inference.wandb.ai/v1
```

## Prerequisites

To call the Inference API, you need:

* A W\&B account with Inference credits.
* A valid W\&B API key.

If you belong to more than one team, or want to attribute your usage to a project, you'll also need team and project IDs. In code samples, these appear as `[YOUR-TEAM]/[YOUR-PROJECT]`. If you don't specify these, W\&B uses your default entity and the project name `inference`.

## Available methods

The Inference API provides OpenAI-compatible endpoints for interacting with foundation models. The following methods are available:

* **[Chat Completions](/inference/api-reference/chat-completions)**: Create chat completions using foundation models.
* **[List Models](/inference/api-reference/list-models)**: Get all available models and their IDs.

## Authentication

All API requests require authentication using your W\&B API key. Create an API key at [wandb.ai/settings](https://wandb.ai/settings).

Include your API key in the request headers:

* For the OpenAI SDK, set the `api_key` parameter.
* For direct API calls, use `Authorization: Bearer [YOUR-API-KEY]`.

## Error handling

For a complete list of error codes and how to resolve them, see [API errors](/support/inference).

## Next steps

After you have your API key, continue with one of the following:

* Try the [usage examples](/inference/examples) to see how the API works.
* Explore models in the [Inference UI](/inference/ui-guide).
* Check [usage limits](/inference/usage-limits) for your account.


# Chat completions
Source: https://docs.wandb.ai/inference/api-reference/chat-completions

Create chat completions using the OpenAI-compatible endpoint

Create a chat completion using the `/chat/completions` endpoint. This endpoint follows the OpenAI format for sending messages and receiving responses.

## Requirements

To create a chat completion, provide:

* The Inference service base URL: `https://api.inference.wandb.ai/v1`.
* Your W\&B API key: `[YOUR-API-KEY]`.
* Your W\&B team and project: `[YOUR-TEAM]/[YOUR-PROJECT]` (optional).
* A model ID from the [available models](/inference/models).

## Request examples

The following examples show how to send a chat completion request using Python and `curl`. Replace the placeholder values with your own API key, optional team and project, and a model ID.

<Tabs>
  <Tab title="Python">
    ```python theme={null}
    import openai

    client = openai.OpenAI(
        # The custom base URL points to Serverless Inference
        base_url='https://api.inference.wandb.ai/v1',

        # Create an API key at https://wandb.ai/settings
        # Consider setting it in the environment as OPENAI_API_KEY instead for safety
        api_key="[YOUR-API-KEY]",

        # Optional: Team and project for usage tracking
        project="[YOUR-TEAM]/[YOUR-PROJECT]",
    )

    # Replace [MODEL-ID] with any model ID from the available models list
    response = client.chat.completions.create(
        model="[MODEL-ID]",
        messages=[
            {"role": "system", "content": "[YOUR-SYSTEM-PROMPT]"},
            {"role": "user", "content": "[YOUR-PROMPT]"}
        ],
    )

    print(response.choices[0].message.content)
    ```
  </Tab>

  <Tab title="Bash">
    ```bash theme={null}
    curl https://api.inference.wandb.ai/v1/chat/completions \
      -H "Content-Type: application/json" \
      -H "Authorization: Bearer [YOUR-API-KEY]" \
      -H "OpenAI-Project: [YOUR-TEAM]/[YOUR-PROJECT]" \
      -d '{
        "model": "[MODEL-ID]",
        "messages": [
          { "role": "system", "content": "You are a helpful assistant." },
          { "role": "user", "content": "Tell me a joke." }
        ]
      }'
    ```
  </Tab>
</Tabs>

## Response format

A successful request returns a response in the following OpenAI-compatible format, including the generated assistant message and token usage details.

```json theme={null}
{
  "id": "chatcmpl-...",
  "object": "chat.completion",
  "created": 1234567890,
  "model": "meta-llama/Llama-3.1-8B-Instruct",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "Here's a joke for you..."
      },
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 25,
    "completion_tokens": 50,
    "total_tokens": 75
  }
}
```


# List models
Source: https://docs.wandb.ai/inference/api-reference/list-models

Retrieve a list of available Serverless Inference models and their IDs using the models endpoint.

This page shows how to retrieve all models available through the Serverless Inference API, along with their IDs. Use this endpoint to discover available models programmatically, select a model dynamically at runtime, or confirm which models your account can access.

## Request examples

The following examples show how to call the models endpoint from Python and from the command line.

<Tabs>
  <Tab title="Python">
    ```python theme={null}
    import openai

    client = openai.OpenAI(
        base_url="https://api.inference.wandb.ai/v1",
        api_key="[YOUR-API-KEY]",
        project="[YOUR-TEAM]/[YOUR-PROJECT]"  # Optional, for usage tracking
    )

    response = client.models.list()

    for model in response.data:
        print(model.id)
    ```
  </Tab>

  <Tab title="Bash">
    ```bash theme={null}
    curl https://api.inference.wandb.ai/v1/models \
      -H "Content-Type: application/json" \
      -H "Authorization: Bearer [YOUR-API-KEY]" \
      -H "OpenAI-Project: [YOUR-TEAM]/[YOUR-PROJECT]"
    ```
  </Tab>
</Tabs>

## Response format

The endpoint returns a list of model objects in OpenAI-compatible format, so you can parse the response using standard OpenAI client libraries. Each entry includes the model ID you pass to other Inference API endpoints.

```json theme={null}
{
  "object": "list",
  "data": [
    {
      "id": "deepseek-ai/DeepSeek-V3.1",
      "object": "model",
      "created": 0,
      "owned_by": "system",
      "root": "deepseek-ai/DeepSeek-V3.1"
    },
    {
      "id": "openai/gpt-oss-20b",
      "object": "model",
      "created": 0,
      "owned_by": "system",
      "root": "openai/gpt-oss-20b"
    }
    // ... more models
  ]
}
```


# Release Notes
Source: https://docs.wandb.ai/release-notes

Browse release notes for W&B Server and SDK updates, including new features, fixes, and breaking changes.

This section includes release notes for W\&B Server, the W\&B Models Python SDK, and the Weave Python SDK.

<Columns>
  <Card title="W&B Server" icon="rss" href="/release-notes/server-releases">
    * [Supported releases](/release-notes/server-releases)
    * [Archived releases](/release-notes/server-releases-archived)
  </Card>

  <Card title="W&B Models SDK" href="./release-notes/sdk-releases" icon="rss">
    * [All releases](/release-notes/sdk-releases)
  </Card>

  <Card title="Weave SDK" href="./release-notes/weave-sdk-releases" icon="rss">
    * [All releases](/release-notes/weave-sdk-releases)
  </Card>
</Columns>

<Tip>Learn about [W\&B releases policies and processes](./release-notes/release-policies).</Tip>


# Release policies and processes
Source: https://docs.wandb.ai/release-notes/release-policies

Understand the W&B Server release process including versioning, support windows, and upgrade policies.

This page gives details about W\&B Server releases and W\&B's release policies. This page relates to [W\&B Dedicated Cloud](/platform/hosting/hosting-options/dedicated-cloud/) and [Self-Managed](/platform/hosting/hosting-options/self-managed/) deployments. To learn more about an individual W\&B Server release, refer to [W\&B release notes](/release-notes/).

W\&B fully manages [W\&B Multi-tenant Cloud](/platform/hosting/hosting-options/multi_tenant_cloud) and the details in this page do not apply.

## Release support and end of life policy

W\&B supports a major W\&B Server release for 12 months from its initial release date.

* **Dedicated Cloud** instances are automatically updated to maintain support.
* Customers with **Self-Managed** instances are responsible for upgrading in time to maintain support. Avoid staying on an unsupported version.

  <Note>
    W\&B strongly recommends customers with **Self-Managed** instances to update their deployments with the latest release at minimum once per quarter to maintain support and receive the latest features, performance improvements, and fixes.
  </Note>

## Release types and frequencies

* **Major releases** are produced monthly, and may include new features, enhancements, performance improvements, medium and low severity bug fixes, and deprecations. An example of a major release is `0.68.0`.
* **Patch releases** within a major version are produced as needed, and include critical and high severity bug fixes. An example of a patch release is `0.67.1`.

## Release rollout

1. After testing and validation are complete, a release is first rolled out to all **Dedicated Cloud** instances to keep them fully updated.
2. After additional observation, the release is published, and **Self-Managed** deployments can upgrade to it on their own schedule, and are responsible for upgrading in time to comply with the [Release support and End of Life (EOL) policy](#release-support-and-end-of-life-policy). Learn more about [upgrading W\&B Server](/platform/hosting/server-upgrade-process/).

## Downtime during upgrades

* When a **Dedicated Cloud** instance is upgraded, downtime is generally not expected, but may occur in certain situations:
  * If a new feature or enhancement requires changes to the underlying infrastructure, such as compute, storage or network.
  * To roll out a critical infrastructure change such as a security fix.
  * If the instance's current version has reached its [End of Life (EOL)](/platform/hosting/server-upgrade-process/) and is upgraded by W\&B to maintain support.
* For **Self-Managed** deployments, the customer is responsible for implementing a rolling update process that meets their service level objectives (SLOs), such as by [running W\&B Server on Kubernetes](/platform/hosting/self-managed/on-premises-deployments/kubernetes-airgapped).

## Feature availability

After installing or upgrading, certain features may not be immediately available.

### Enterprise features

An Enterprise license includes support for important security capabilities and other enterprise-friendly functionality. Some advanced features require an Enterprise license.

* **Dedicated Cloud** includes an Enterprise license and no action is required.
* On **Self-Managed** deployments, features that require an Enterprise license are not available until it is set. To learn more or obtain an Enterprise license, refer to the [License](/platform/hosting/self-managed/requirements#license) section on the Self-Managed Requirements page.

### Private preview and opt-in features

Most features are available immediately after installing or upgrading W\&B Server. The W\&B team must enable certain features before you can use them in your instance.

<Warning>
  Any feature in a preview phase is subject to change. A preview feature is not guaranteed to become generally available.
</Warning>

* **Private preview**: W\&B invites design partners and early adopters to test these features and provide feedback. Private preview features are not recommended for production environments.

  The W\&B team must enable a private preview feature for your instance before you can use it. Public documentation is not available; instructions are provided directly. Interfaces and APIs may change, and the feature may not be fully implemented.
* **Public preview**: Contact W\&B to opt in to a public preview to try it out before it is generally available.

  The W\&B team must enable a public preview feature before you can use it in your instance. Documentation may not be complete, interfaces and APIs may change, and the feature may not be fully implemented.

To learn more about an individual W\&B Server release, including any limitations, refer to [W\&B Release notes](/release-notes/).


# W&B SDK releases
Source: https://docs.wandb.ai/release-notes/sdk-releases

View release notes for the W&B Python SDK including new features, bug fixes, and breaking changes by version.

This page includes release notes for the W\&B Python SDK (`wandb` package) used with W\&B Models. For the Weave Python SDK (`weave` package), see [Weave SDK releases](/release-notes/weave-sdk-releases). For W\&B Server, see [W\&B Server release notes](/release-notes/server-releases).

<Update label="0.27.2" description="June 6, 2026">
  ## Fixed

  * `Run.scan_history()` no longer returns duplicate or missing steps when fetching run history.
</Update>

<Update label="0.27.1" description="June 4, 2026">
  ## Added

  * The Automations API now supports artifact tag, collection tag, and artifact unlinking events.

  ## Changed

  * `wandb.Api` GraphQL operations now route through `wandb-core` instead of the legacy Python GraphQL client.
    * Failures from `wandb.Api` GraphQL operations now raise `WandbApiFailedError` instead of `requests` HTTP exceptions.
  * `Run.scan_history()` now reads from exported parquet history when available, which can significantly improve throughput for runs with large history. This capability was introduced under `beta_scan_history` in SDK `v0.23.1`.

  ## Removed

  * Removed the unsupported `wandb.apis.importers` API.
  * Removed stale OpenAI, Cohere, and LangChain LLM integrations, including legacy autologging and tracing APIs.
  * Removed the deprecated Keras `WandbCallback` and the legacy `wandb.integration.yolov8` callback package.

  ## Fixed

  * `wandb.sandbox` now rejects invalid argument values for placement, GPU, and egress.
</Update>

<Update label="0.27.0" description="May 13, 2026">
  ## Notable Changes

  This version drops support for Python 3.9 and Pydantic v1. Pydantic v2.6 or newer is now required.

  ## Added

  * The new package `wandb.sandbox` and the `wandb beta sandbox` CLI for using [Serverless Sandboxes](/sandboxes).
  * The new setting `stop_on_fatal_error` and environment variable `WANDB_STOP_ON_FATAL_ERROR` stop a run after a fatal error that prevents it from uploading metrics.
  * The new setting `finish_timeout` and corresponding environment variable `WANDB_FINISH_TIMEOUT` let you specify how many seconds to wait for data to upload at the end of a run. If a run times out, a warning message is printed. Use boolean setting `finish_timeout_raises` or corresponding environment variable `WANDB_FINISH_TIMEOUT_RAISES` to raise an error in addition to printing a warning message.
  * The new setting `capture_loggers` lets you specify Python loggers to capture into the run's **Logs** tab, using a mapping of logger name to minimum log level. Log records emitted by those loggers are published as console output to the run, similar to `stdout`/`stderr` capture.
  * The new method `run.write_logs()` writes text directly to the run's **Logs** tab instead of relying on automatic capture of `stdout`/`stderr`.

  ## Changed

  * System metrics for runs in Linux containers now report CPU and memory usage relative to the container's cgroup v2 limits instead of the host's totals.
  * Python 3.9 is no longer supported.
  * Pydantic 2.6 or above is now required. Pydantic v1 is no longer supported.
</Update>

<Update label="0.26.1" description="April 23, 2026">
  ## Added

  * `wandb.Api` methods that return artifacts, registries, automations, and related paginators accept an optional `start` argument so you can resume iteration from a saved cursor.
  * The `stop_fn` setting on `wandb.Settings` customizes how the SDK asks the Python process to stop when a run stop is requested (the default sends SIGINT). Pass the setting with `wandb.setup(settings=...)` or `wandb.init(settings=...)`. Because the value is a callable, it cannot be set with an environment variable.

  ## Changed

  * In `wandb beta leet`, WASD keys and arrow keys are interchangeable for moving focus within each pane (chart grids, list navigation). **Home**, **End**, **Page Up**, and **Page Down** keys work consistently. The media pane still uses arrow keys to scrub the X-axis and WASD keys to move between image tiles.

  ## Fixed

  * `wandb.init(id=run_id, reinit="create_new")` now raises an error if another run with the same `run_id` is still active in the same script.
  * `wandb.Api` no longer raises on some operations when offline mode is enabled with the `WANDB_MODE` environment variable or the `mode` setting.
</Update>

<Update label="0.26.0" description="April 13, 2026">
  ## Notable Changes

  This version drops compatibility with Server versions older than 0.63.0 for Dedicated Cloud and Self-Managed W\&B deployments.

  ## Added

  * `wandb beta core start|stop` commands run a detached `wandb-core` service and reuse it across multiple processes via the `WANDB_SERVICE` environment variable.
  * The `@wandb_log` decorator adds support for Kubeflow Pipelines v2 (`kfp>=2.0.0`).
  * Set the property `wandb.Settings(allow_media_symlink=true)` to link to media files to the run directory rather than copying them. This improves performance and saves disk space. A symbolic link is used if the filesystem supports it. If the filesystem does not support symbolic links, such as in Windows, a hard link is used instead. The SDK does not check whether the media file was modified locally before uploading it.
  * Use method `run.pin_config_keys(keys)` to programmatically pin specific config keys for display in the **References** section on the **Run Overview** page.
  * `Sweep.agent().Runs()` and `Sweep.agents().Runs()` methods return details about the sweep's runs running on a given agent or all agents.
  * On Google TPU infrastructure, the SDK attempts to collect new metrics such as `tensorcore_util` and latency distributions, in addition to `duty_cycle_pct` and `hbm_capacity_total`/`hbm_capacity_usage`. See [TPU monitoring library: Supported metrics](https://docs.cloud.google.com/tpu/docs/tpu-monitoring-library#list-all-supported-metric-names) in the Google Cloud documentation.
  * On Nvidia Hopper+ GPUs (H100 and newer), NVML GPU Performance Monitoring profiling metrics are collected, providing visibility into SM utilization, tensor/FP pipeline activity, DRAM bandwidth, and PCIe/NVLink throughput without running the DCGM daemon.
  * `wandb beta leet` TUI updates:
    * Support for displaying tags and notes in the run overview.
    * Support for filtering runs by metadata in multi-run workspace mode using the `f` keyboard shortcut.
    * Support for displaying `wandb.Image` data as ANSI thumbnails, with grid layout, X-axis scrubbing, full-screen mode, and keyboard or pointer navigation.
    * Cycle through the following display modes in an individual chart using the `y` keyboard shortcut:
      * Press once to turn on log-scale display for the Y-axis.
      * For system metrics like GPU utilization expressed as percentages, press again to turn off log-scale display and turn on bucketed heatmap mode.
      * Press again to go back to the default display mode.
    * New standalone system monitor mode: `wandb beta leet symon`.
    * New colorblind-friendly color schemes:

      * `dusk-shore`: sienna/blue/gray gradient, suitable for per-plot single-run mode.
      * `clear-signal`: sienna/blue/gray theme with the colors interleaved for maximum differentiation between adjacent series.

      Configure with `wandb beta leet config`.

  ## Changed

  * JSON serialization and deserialization now use `orjson` for improved performance.
  * Improved observability of system metrics in `wandb beta leet` TUI with multi-series overlays, inspection, and live/history zoom.
  * Reduced the likelihood of run base color collisions in `wandb beta leet`. Now, a workspace-scoped run color allocator preserves hashed palette affinity, resolving collisions with nearby HSL variants.

  ## Fixed

  * Fixed a bug where if no upstream tracking branch was set, the SDK would not create a `git diff` `.patch` file between the local branch and the nearest upstream branch. To opt in to this fix, pass property `wandb.Settings(disable_git_fork_point=False)`. See [Settings()](/models/ref/python/experiments/settings).
  * Fixed a bug where `update_automation()` could silently drop event filters, such as alias conditions on `OnAddArtifactAlias`, when a new event is provided.
  * Fixed a bug that could cause artifact client ID collisions in forked child processes by reseeding the fast ID generator after `fork()`.
  * Fixed a bug where the environment variable `WANDB__EXTRA_HTTP_HEADERS` was not applied to presigned object-store upload and download requests.
  * Fixed a deadlock in `artifact.download()` for artifacts with many large files.
  * Fixed a bug where `User.generate_api_key()` could fail to fetch the list of existing API key names.
</Update>

<Update label="0.25.1" description="March 10, 2026">
  ## Added

  * New run console logs pane in `wandb beta leet` command. Toggle with `l`.
  * New system metrics pane in multi-run workspace mode in `wandb beta leet` command. Toggle with `s`.
  * `wandb beta leet` command now supports system metrics filtering. Toggle with `\`.
  * `ArtifactType.collections()` now supports filtering and ordering of collections.
  * A warning is now logged if `run.log_artifact()` does not create a new version because the artifact content is identical to an existing version.
  * The new method `Project.collections()` fetches filtered and ordered artifact collections in a project.
  * The new `wandb purge-cache` CLI command cleans up cached files.

  ## Fixed

  * Fixed a rare deadlock caused when garbage collection triggers at an unlucky time and runs a `__del__` method that prints.
  * Fixed a bug where loading multiple runs with `api.Runs("...")`, then attempting to get a single run's user with code like the following could fail with an `AttributeError`:

    ```python theme={null}
    for run in runs:
      print(run.user)
    ```
</Update>

<Update label="0.25.0" description="February 12, 2026">
  ## Notable changes

  * Python 3.8 is no longer supported.

  ## Added

  * The `wandb beta leet` TUI now supports multi-run workspaces.
  * Configure `wandb beta leet`'s persisted configuration interactively with the `wandb beta leet config` command.
  * Access the project owner's user information with the `owner` property of `wandb.apis.public.Project`.

  ## Fixed

  * Fixed a bug where a sweep agent for a deleted sweep could run indefinitely with repeated 404 errors.
  * Fixed a bug where a `wandb-core` crash could produce extremely long and repetitive tracebacks in older Python versions.
  * Fixed a regression introduced in v0.24.0 where a TensorBoard sync could stop after syncing 1 MB of data.
</Update>

<Update label="0.24.2" description="February 4, 2026">
  ## Added

  * `wandb.Api()` now supports Federated Auth (JWT based authentication).

  ## Fixed

  * The presigned download URL is now refreshed if it expires during artifact file downloads.
</Update>

<Update label="0.24.1" description="January 29, 2026">
  ## Notable Changes

  * Runs created with `wandb==0.24.0` may fail to upload some data, which this release fixes. Missing data is stored in the run's `.wandb` file and can be reuploaded with `wandb sync`.

  ## Added

  * `download_history_exports` in `api.Run` class to download exported run history in parquet file format.

  ## Changed

  * When a settings file (such as `./wandb/settings` or `~/.config/wandb/settings`) contains an invalid setting, all settings files are ignored and an error is printed.

  ## Fixed

  * After `wandb login --host <invalid-url>`, using `wandb login --host <valid-url>` works as usual.
  * Fixed a bug where `wandb beta sync` can get stuck on `Syncing...` and fail to load credentials.
  * Fixed a bug that could occasionally lead to unuploaded data. Now, all data is stored in the run's `.wandb` file and can be reuploaded with `wandb sync`.

  These regressions were introduced in 0.24.0.
</Update>

<Update label="0.24.0" description="January 13, 2026">
  <Warning>
    This release has been removed from PyPI. Do not use this version. If you are on this version, upgrade to v0.24.1 or a newer version immediately. Due to a regression fixed in v0.24.1, runs created with `wandb==0.24.0` may fail to upload some data.
  </Warning>

  ## Notable Changes

  This version removes the legacy, deprecated `wandb.beta.workflows` module, including its `log_model()`/`use_model()`/`link_model()` functions. This is formally a breaking change.

  ## Added

  * `wandb agent` and `wandb.agent()` now accept a `forward_signals` flag (CLI: `--forward-signals/-f`) to relay SIGINT/SIGTERM and other catchable signals from the agent to its sweep child runs, enabling cleaner shutdowns when you interrupt an agent process.
  * `wandb beta sync` now supports a `--live` option for syncing a run while it's being logged.

  ## Removed

  * Removed the deprecated `wandb.beta.workflows` module, including its `log_model()`, `use_model()`, and `link_model()` functions, and whose modern successors are the `Run.log_artifact`, `Run.use_artifact`, and `Run.link_artifact` methods, respectively.

  ## Fixed

  * Fixed `Run.__exit__` type annotations to accept `None` values, which are passed when no exception is raised.
  * Fixed `Invalid Client ID digest` error when creating artifacts after calling `random.seed()`. Client IDs could collide when random state was seeded deterministically..
  * Fixed CLI error when listing empty artifacts.
  * Fixed regression for calling `api.run()` on a Sweeps run.
  * Fixed the "View run at" message printed at the end of a run which sometimes did not include a URL.
  * Runs queried from wandb.Api() now display a string representation in VSCode notebooks instead of a broken HTML window.
</Update>

<Update label="v0.23.1" description="December 2, 2025">
  ## Added

  * Regex support in metrics and run overview filters in `wandb beta leet` command.
  * Chart inspection in `wandb beta leet` command: right-click and drag to show (x, y) at the nearest data point; hold Alt for synchronized inspection across all visible charts.
  * The Automations API now supports creating and editing automations that trigger on run states.
  * The Automations API now supports basic zscore automation events.
  * `beta_history_scan` method to `Run` objects for faster history scanning performance with `wandb.Api`.

  ## Changed

  * `wandb.Api()` now raises a `UsageError` if `WANDB_IDENTITY_TOKEN_FILE` is set and an explicit API key is not provided.

  ## Deprecated

  * Anonymous mode is deprecated and will emit warnings. This deprecation includes the `anonymous` setting, the `WANDB_ANONYMOUS` environment variable, `wandb.init(anonymous=...)`, and commands `wandb login --anonymously` and `wandb.login(anonymous=...)`.

  ## Fixed

  * `wandb.Image()` no longer prints a deprecation warning.
  * `Registry.description` and `ArtifactCollection.description` no longer reject empty strings.
  * Instantiating `Artifact` objects is now significantly faster.
  * `wandb.Run.save()` now falls back to hardlinks and, if needed, copying (downgrading the 'live' file policy to 'now', if applicable) when symlinks are disabled or unavailable (e.g., cross‑volume or no Developer Mode on Windows).
  * Artifact collection aliases are now fetched lazily on accessing `ArtifactCollection.aliases` instead of on instantiating `ArtifactCollection`, improving performance of `Api.artifact_collections()`, `Api.registries().collections()`, etc.
  * Use explicitly provided API key in `wandb.init(settings=wandb.Settings(api_key="..."))` for login. Use the key from run when logging artifacts via `run.log_artifact()`.
  * W\&B LEET TUI correctly displays negative Y axis tick values and base/display units of certain system metrics.
  * Fixed a rare infinite loop in `console_capture.py`.
  * File upload/download now respects `WANDB_X_EXTRA_HTTP_HEADERS` except for [reference artifacts](/models/artifacts/track-external-files).
</Update>

<Update label="v0.23.0" description="November 11, 2025">
  ## Added

  * Experimental `wandb beta leet` command - Lightweight Experiment Exploration Tool - a terminal UI for viewing W\&B runs locally with real-time metrics visualization and system monitoring.
  * The registry API now supports programmatic management of user and team members of individual registries.
  * `Registry.id` has been added as a (read-only) property of `Registry` objects.

  ## Fixed

  * `Artifact.files()` now has a correct `len()` when filtering by the `names` parameter.
  * The numerator for file upload progress no longer occasionally exceeds the total file size.
  * `Artifact.link()` now logs unsaved artifacts instead of raising an error, consistent with the behavior of `Run.link_artifact()`.
  * Automatic code saving now works when running ipython notebooks in VSCode's Jupyter notebook extension.
  * Logging an artifact with infinite floats in `Artifact.metadata` now raises a `ValueError` early, instead of waiting on request retries to time out.
</Update>

<Update label="v0.22.3" description="October 28, 2025">
  ## Added

  * Settings `console_chunk_max_seconds` and `console_chunk_max_bytes` for size- and time-based multipart console logs file chunking.
  * Registry API query methods (`Api.registries()`, `Registry.{collections,versions}()`, `Api.registries().{collections,versions}()`) now accept a `per_page` keyword arg to override the default batch size for paginated results.

  ## Changed

  * API keys longer than 40 characters are now supported.

  ## Fixed

  * `run.config` now properly returns a dict when calling `artifact.logged_by()` in v0.22.1.
  * `wandb.Api(api_key=...)` now prioritizes the explicitly provided API key over thread-local cached credentials.
  * Fixed a rare deadlock in `console_capture.py`.

    If you dump thread tracebacks during the deadlock and see the `wandb-AsyncioManager-main` thread stuck on a line in `console_capture.py`: this is now fixed.
  * Fixed an issue where TensorBoard sync would sometimes stop working if the tfevents files were being written live.
  * `Artifact.manifest` delays downloading **and** generating the download URL for the artifact manifest until it's first used. If the manifest has not been locally modified, `Artifact.size` and `Artifact.digest` can return without waiting to fetch the full manifest.
  * Fixed uploading GCS folder references via `artifact.add_reference`.
  * The SDK now correctly infers notebooks paths in Jupyter sessions, using th server's root directory, so code saving works in subdirectories (e.g. `code/nested/<notebook>.ipynb`).

  View the [v0.22.3 changelog](https://github.com/wandb/wandb/releases/tag/v0.22.3) on GitHub.
</Update>

<Update label="v0.22.2" description="October 07, 2025">
  ## Fixed

  * Possibly fixed some cases where the `output.log` file was not being uploaded.
  * Fixed excessive data uploads when calling `run.save()` repeatedly on unchanged files.

  View the [v0.22.2 changelog](https://github.com/wandb/wandb/releases/tag/v0.22.2) on GitHub.
</Update>

<Update label="v0.22.1" description="September 29, 2025">
  ## Added

  * Optimize artifacts downloads re-verification with checksum caching.
  * Lazy loading support for `Api().runs()` to improve performance when listing runs. The new `lazy=True` parameter (default) loads only essential metadata initially, with automatic on-demand loading of heavy fields like config and summary when accessed.
  * Add `storage_region` option when creating artifacts. Users can use [CoreWeave AI Object Storage](https://docs.coreweave.com/docs/products/storage/object-storage) by specifying `wandb.Artifact(storage_region="coreweave-us")` when using wandb.ai for faster artifact upload/download on CoreWeave's infrastructure.

  ## Fixed

  * `Api.artifact_exists()` and `Api.artifact_collection_exists()` now raise on encountering timeout errors, rather than (potentially erroneously) returning `False`.

  View the [v0.22.1 changelog](https://github.com/wandb/wandb/releases/tag/v0.22.1) on GitHub.
</Update>

<Update label="v0.22.0" description="September 18, 2025">
  ## Notable Changes

  This version removes support of MacOS 10.

  ## Removed

  * Remove a build targeting MacOS 10.x due to multiple security and supply chain considerations.

  ## Fixed

  * Resuming a run with a different active run will now raise an error unless you call `run.finish()` first, or call `wandb.init()` with the parameter `reinit='create_new'`.
  * Fix `Api().runs()` for wandb server \<  0.51.0 (when `project.internalId` was added to gql API).
  * Sweeps: `command` run scripts that `import readline` whether directly or indirectly (e.g. `import torch` on Python 3.13) should no longer deadlock.

  View the [v0.22.0 changelog](https://github.com/wandb/wandb/releases/tag/v0.22.0) on GitHub.
</Update>

<Update label="v0.21.4" description="September 11, 2025">
  ## Added

  * Add DSPy integration: track evaluation metrics over time, log predictions and program signature evolution to W\&B Tables, and save DSPy programs as W\&B Artifacts (complete program or state as JSON/PKL).

  View the [v0.21.4 changelog](https://github.com/wandb/wandb/releases/tag/v0.21.4) on GitHub.
</Update>

<Update label="v0.21.3" description="August 30, 2025">
  ## Changed

  * Updated `click` dependency constraint from `>=7.1` to `>=8.0.1`.

  ## Fixed

  * The message "Changes to your wandb environment variables will be ignored" is no longer printed when nothing changed.

  View the [v0.21.3 changelog](https://github.com/wandb/wandb/releases/tag/v0.21.3) on GitHub.
</Update>

<Update label="v0.21.2" description="August 28, 2025">
  ## Notable Changes

  This version raises errors that would previously have been suppressed during calls to `Artifact.link()` or `Run.link_artifact()`. While this prevents undetected failures in those methods, it is also a breaking change.

  ## Added

  * New settings for `max_end_of_run_history_metrics` and `max_end_of_run_summary_metrics`.
  * New `wandb.integration.weave` module for automatically initializing Weave when a W\&B run is active and `weave` is imported.

  ## Changed

  * Errors encountered while linking an artifact are no longer suppressed/silenced, and `Artifact.link()` and `Run.link_artifact()` no longer return `None`.
  * The "Run history" and "Run summary" printed at the end of a run are now limited to 10 metrics each.

  ## Fixed

  * Dataclasses in a run's `config` no long raise `Object of type ... is not JSON serializable` when containing real classes as fields to the dataclass.
  * `Artifact.link()` and `Run.link_artifact()` should be faster on server versions 0.74.0+, requiring 4-5 fewer unnecessary blocking GraphQL requests.

  View the [v0.21.2 changelog](https://github.com/wandb/wandb/releases/tag/v0.21.2) on GitHub.
</Update>

<Update label="v0.21.1" description="August 07, 2025">
  ## Notable Changes

  The default ordering for `Api().runs(...)` and `Api().sweeps(...)` is now ascending order based on the runs `created_at` time.

  ## Added

  * Support `first` summary option in `define_metric`.
  * Add support for paginated sweeps.
  * `pattern` parameter to `Api().run().files` to only get files matching a given pattern from the W\&B backend.
  * Add optional `format` key to Launch input JSONSchema to specify a string with a secret format.

  ## Changed

  * `Sweep.name` property will now return user-edited display name if available (falling back to original name from sweep config, then sweep ID as before).
  * `Api().runs(...)` and `Api().sweeps(...)` now returns runs in ascending order according to the runs `created_at` time.
  * Artifact with large file (>2GB) uploads faster by using parallel hashing on system with more cores.
  * Remove the implementation of `__bool__` for the registry iterators to align with python lazy iterators.

  ## Deprecated

  * The `wandb.beta.workflows` module and its contents (including `log_model()`, `use_model()`, and `link_model()`) are deprecated and will be removed in a future release.

  ## Fixed

  * Correct the artifact url for organization registry artifacts to be independent of the artifact type.
  * Suffixes on sanitized `InternalArtifact` names have been shortened to 6 alphanumeric characters.
  * `wandb.Video` will not print a progress spinner while encoding video when `WANDB_SILENT`/`WANDB_QUIET` environment variables are set.
  * Fixed registries fetched using `api.registries()` from having an extra `wandb-registry-` prefix in the name and full\_name fields.
  * Fixed a crash that could happen when using `sync_tensorboard`.
  * `Api().run(...).upload_file` no longer throws an error when uploading a file in a different path relative to the provided root directory.
  * Calling `load()` function on a public API run object no longer throws `TypeError`.
  * When a Sweeps run function called by `wandb.agent()` API throws an exception, it will now appear on the logs page for the run. (This previously only happened for runs called by the `wandb agent` CLI command.).

  View the [v0.21.1 changelog](https://github.com/wandb/wandb/releases/tag/v0.21.1) on GitHub.
</Update>

<Update label="v0.21.0" description="July 01, 2025">
  ## Notable Changes

  This version removes the legacy implementation of the `service` process. This is a breaking change.

  ## Added

  * Setting `x_stats_track_process_tree` to track process-specific metrics such as the RSS, CPU%, and thread count in use for the entire process tree, starting from `x_stats_pid`. This can be expensive and is disabled by default.
  * Notes are now returned to the client when resuming a run.
  * Added support for creating custom Vega chart presets through the API. Users can now define and upload custom chart specifications that can be then reused across runs with wandb.plot\_table().

  ## Changed

  * Calling `Artifact.link()` no longer instantiates a throwaway placeholder run.
  * `wandb` now attempts to use Unix sockets for IPC instead of listening on localhost, making it work in environments with more restrictive permissions (such as Databricks).
  * `Api.artifact()` will now display a warning while fetching artifacts from migrated model registry collections.
  * The `.length` for objects queried from `wandb.Api` has been deprecated. Use `len(...)` instead.

  ## Removed

  * Removed the legacy python implementation of the `service` process. The `legacy-service` option of `wandb.require()` as well as the `x_require_legacy_service` and `x_disable_setproctitle` settings with the corresponding environment variables have been removed and will now raise an error if used.

  * Removed the private `wandb.Run._metadata` attribute. To override the auto-detected CPU and GPU counts as well as the GPU type, please use the new settings `x_stats_{cpu_count,cpu_logical_count,gpu_count,gpu_type}`.

  ## Fixed

  * Allow s3 style CoreWeave URIs for reference artifacts.
  * Fixed rare bug that made Ctrl+C ineffective after logging large amounts of data.
  * Respect `silent`, `quiet`, and `show_warnings` settings passed to a `Run` instance for warnings emitted by the service process.
  * `api.Runs` no longer makes an API call for each run loaded from W\&B.
  * Correctly parse the `x_extra_http_headers` setting from the env variable.
  * `.length` calls the W\&B backend to load the length of objects when no data has been loaded rather than returning `None`.

  View the [v0.21.0 changelog](https://github.com/wandb/wandb/releases/tag/v0.21.0) on GitHub.
</Update>

<Update label="v0.20.1" description="June 04, 2025">
  ## Fixed

  * `wandb.Image()` was broken in 0.20.0 when given NumPy arrays with values in the range \[0, 1], now fixed.

  * wandb.Table: Added new constructor param, `log_mode`, with options `"IMMUTABLE"` and `"MUTABLE"`. `IMMUTABLE` log mode (default) is existing behavior that only allows a table to be logged once. `MUTABLE` log mode allows the table to be logged again if it has been mutated. (@domphan-wandb in [https://github.com/wandb/wandb/pull/9758](https://github.com/wandb/wandb/pull/9758))

  * wandb.Table: Added a new `log_mode`, `"INCREMENTAL"`, which logs newly added table data incrementally. (@domphan-wandb in [https://github.com/wandb/wandb/pull/9810](https://github.com/wandb/wandb/pull/9810))

  View the [v0.20.1 changelog](https://github.com/wandb/wandb/releases/tag/v0.20.1) on GitHub.
</Update>

<Update label="v0.20.0" description="June 03, 2025">
  ## Notable Changes

  This version removes the ability to disable the `service` process. This is a breaking change.

  ## Added

  * Added `merge` parameter to `Artifact.add_dir` to allow overwrite of previously-added artifact files.
  * Support for pytorch.tensor for `masks` and `boxes` parameters when creating a `wandb.Image` object.
  * `sync_tensorboard` now supports syncing tfevents files stored in S3, GCS and Azure.
    * GCS paths use the format `gs://bucket/path/to/log/dir` and rely on application-default credentials, which can be configured using `gcloud auth application-default login`
    * S3 paths use the format `s3://bucket/path/to/log/dir` and rely on the default credentials set through `aws configure`
    * Azure paths use the format `az://account/container/path/to/log/dir` and the `az login` credentials, but also require the `AZURE_STORAGE_ACCOUNT` and `AZURE_STORAGE_KEY` environment variables to be set. Some other environment variables are supported as well, see [here](https://pkg.go.dev/gocloud.dev@v0.41.0/blob/azureblob#hdr-URLs).
  * Added support for initializing some Media objects with `pathlib.Path`.
  * New setting `x_skip_transaction_log` that allows to skip the transaction log. Note: Should be used with caution, as it removes the guarantees about recoverability.
  * `normalize` parameter to `wandb.Image` initialization to normalize pixel values for Images initialized with a numpy array or pytorch tensor.

  ## Changed

  * Various APIs now raise `TypeError` instead of `ValueError` or other generic errors when given an argument of the wrong type.
  * Various Artifacts and Automations APIs now raise `CommError` instead of `ValueError` upon encountering server errors, so as to surface the server error message.
  * `wandb.sdk.wandb_run.Run::save` method now requires the `glob_str` argument.

  ## Removed

  * Removed support for disabling the `service` process. The `x_disable_service`/`_disable_service` setting and the `WANDB_DISABLE_SERVICE`/`WANDB_X_DISABLE_SERVICE` environment variable have been deprecated and will now raise an error if used.
  * Removed ability to use `wandb.docker` after only importing `wandb`.
    * `wandb.docker` is not part of `wandb`'s public interface and is subject to breaking changes. Please do not use it.
  * Removed no-op `sync` argument from `wandb.Run::log` function.
  * Removed deprecated `wandb.sdk.wandb_run.Run.mode` property.
  * Removed deprecated `wandb.sdk.wandb_run.Run::join` method.

  ## Deprecated

  * The `start_method` setting is deprecated and has no effect; it is safely ignored.
  * The property `Artifact.use_as` and parameter `use_as` for `run.use_artifact()` are deprecated since these have not been in use for W\&B Launch.

  ## Fixed

  * Calling `wandb.teardown()` in a child of a process that called `wandb.setup()` no longer raises `WandbServiceNotOwnedError`.
    * This error could have manifested when using W\&B Sweeps
  * Offline runs with requested branching (fork or rewind) sync correctly.
  * Log exception as string when raising exception in Job wait\_until\_running method.
  * `wandb.Image` initialized with tensorflow data would be normalized differently than when initialized with a numpy array.
  * Using `wandb login` no longer prints a warning about `wandb.require("legacy-service")`.
  * Logging a `Table` (or other objects that create internal artifacts) no longer raises `ValueError` when logged from a run whose ID contains special characters.
  * `wandb.Api` initialized with the `base_url` now respects the provided url, rather than the last login url.

  View the [v0.20.0 changelog](https://github.com/wandb/wandb/releases/tag/v0.20.0) on GitHub.
</Update>

<Update label="v0.19.11" description="May 07, 2025">
  ## Added

  * Added creation, deletion, and updating of registries in the SDK.
  * `artifact.is_link` property to artifacts to determine if an artifact is a link artifact (such as in the Registry) or source artifact.
  * `artifact.linked_artifacts` to fetch all the linked artifacts to a source artifact and `artifact.source_artifact` to fetch the source artifact of a linked artifact.
  * `run.link_artifact()`, `artifact.link()`, and `run.link_model()` all return the linked artifact upon linking.
  * Multipart download for artifact file larger than 2GB, user can control it directly using `artifact.download(multipart=True)`.
  * `Project.id` property to get the project ID on a `wandb.public.Project`.
  * New public API for W\&B Automations.
    * New submodules and classes in `wandb.automations.*` to support programmatically managing W\&B Automations.
    * `Api.integrations()`, `Api.slack_integrations()`, `Api.webhook_integrations()` to fetch a team's existing Slack or webhook integrations.
    * `Api.create_automation()`, `Api.automation()`/`Api.automations()`, `Api.update_automation()`, `Api.delete_automation()` to create, fetch, edit, and delete Automations.
  * Create and edit automations triggered on `RUN_METRIC_CHANGE` events, i.e. on changes in run metric values (absolute or relative deltas).
  * Ability to collect profiling metrics for Nvidia GPUs using DCGM. To enable, set the `WANDB_ENABLE_DCGM_PROFILING` environment variable to `true`. Requires the `nvidia-dcgm` service to be running on the machine. Enabling this feature can lead to increased resource usage.

  ## Fixed

  * `run.log_code` correctly sets the run configs `code_path` value.
  * Correctly use `WANDB_CONFIG_DIR` for determining system settings file path.
  * Prevent invalid `Artifact` and `ArtifactCollection` names (which would make them unloggable), explicitly raising a `ValueError` when attempting to assign an invalid name.
  * Prevent pydantic `ConfigError` in Pydantic v1 environments from not calling `.model_rebuild()/.update_forward_refs()` on generated types with ForwardRef fields.
  * `wandb.init()` no longer raises `Permission denied` error when the wandb directory is not writable or readable.
  * Calling `file.delete()` on files queried via `api.Runs(...)` no longer raises `CommError`.
    * Bug introduced in 0.19.1

  View the [v0.19.11 changelog](https://github.com/wandb/wandb/releases/tag/v0.19.11) on GitHub.
</Update>

<Update label="v0.19.10" description="April 22, 2025">
  ## Added

  * The new `reinit="create_new"` setting causes `wandb.init()` to create a new run even if other runs are active, without finishing the other runs (in contrast to `reinit="finish_previous"`). This will eventually become the default.
  * Added `Artifact.history_step` to return the nearest run step at which history metrics were logged for the artifact's source run.
  * Added `data_is_not_path` flag to skip file checks when initializing `wandb.Html` with a sting that points to a file.

  ## Changed

  * `Artifact.download()` no longer raises an error when using `WANDB_MODE=offline` or when an offline run exists.

  ## Removed

  * Dropped the `-q` / `--quiet` argument to the `wandb` magic in IPython / Jupyter; use the `quiet` run setting instead.

  ## Deprecated

  * The following `wandb.Run` methods are deprecated in favor of properties and will be removed in a future release (@kptkin in [https://github.com/wandb/wandb/pull/8925](https://github.com/wandb/wandb/pull/8925)):.
    * `run.project_name()` is deprecated in favor of `run.project`
    * `run.get_url()` method is deprecated in favor of `run.url`
    * `run.get_project_url()` method is deprecated in favor of `run.project_url`
    * `run.get_sweep_url()` method is deprecated in favor of `run.sweep_url`

  ## Fixed

  * Fixed ValueError on Windows when running a W\&B script from a different drive.
  * Fix base\_url setting was not provided to wandb.login().
  * `wandb.Html()` no longer raises `IsADirectoryError` with a value that matched a directory on the users system.

  View the [v0.19.10 changelog](https://github.com/wandb/wandb/releases/tag/v0.19.10) on GitHub.
</Update>

<Update label="v0.19.9" description="April 01, 2025">
  ## Added

  * The `reinit` setting can be set to `"default"`.
  * Added support for building artifact file download urls using the new url scheme, with artifact collection membership context.

  ## Changed

  * Boolean values for the `reinit` setting are deprecated; use "return\_previous" and "finish\_previous" instead.
  * The "wandb" logger is configured with `propagate=False` at import time, whereas it previously happened when starting a run. This may change the messages observed by the root logger in some workflows.
  * Metaflow now requires `plum-dispatch` package.
  * Relaxed the `pydantic` version requirement to support both v1 and v2.
  * Existing `pydantic` types have been adapted to be compatible with Pydantic v1.
  * `wandb.init(dir=...)` now creates any nonexistent directories in `dir` if it has a parent directory that is writeable.
  * The server now supports fetching artifact files by providing additional collection information; updated the artifacts api to use the new endpoints instead.
  * Paginated methods (and underlying paginators) that accept a `per_page` argument now only accept `int` values. Default `per_page` values are set directly in method signatures, and explicitly passing `None` is no longer supported.

  ## Fixed

  * Calling `wandb.init()` in a notebook finishes previous runs as previously documented.
    * Bug introduced in 0.19.0
  * Fixed an error being thrown when logging `jpg`/`jpeg` images containing transparency data.
  * `wandb.init(resume_from=...)` now works without explicitly specifying the run's `id`.
  * Deleting files with the Public API works again.
    * Bug introduced in 0.19.1
  * Fixed media files not displaying in the UI when logging to a run with a custom storage bucket.

  View the [v0.19.9 changelog](https://github.com/wandb/wandb/releases/tag/v0.19.9) on GitHub.
</Update>

<Update label="v0.19.8" description="March 04, 2025">
  ## Fixed

  * Media file paths containing special characters (?, \*, ], \[ or \\) no longer cause file uploads to fail in `wandb-core`.

  ## Changed

  * The system monitor now samples metrics every 15 seconds by default, up from 10 seconds.

  View the [v0.19.8 changelog](https://github.com/wandb/wandb/releases/tag/v0.19.8) on GitHub.
</Update>

<Update label="v0.19.7" description="February 21, 2025">
  ## Added

  * Registry search api.

  ## Changed

  * Changed moviepy constraint to >=1.0.0.
  * `wandb.init()` displays more detailed information, in particular when it is stuck retrying HTTP errors.

  ## Removed

  * Removed the private `x_show_operation_stats` setting.

  ## Fixed

  * Fixed incorrect logging of an "wandb.Video requires moviepy \[...]" exception when using moviepy v2.
  * `wandb.setup()` correctly starts up the internal service process; this semantic was unintentionally broken in 0.19.2.
  * Fixed `TypeError: Object of type ... is not JSON serializable` when using numpy number types as values.

  View the [v0.19.7 changelog](https://github.com/wandb/wandb/releases/tag/v0.19.7) on GitHub.
</Update>

<Update label="v0.19.6" description="February 05, 2025">
  ## Added

  * Prometheus API support for Nvidia DCGM GPU metrics collection.

  ## Changed

  * Changed Nvidia GPU ECC counters from aggregated to volatile.

  ## Fixed

  * Fixed a performance issue causing slow instantiation of `wandb.Artifact`, which in turn slowed down fetching artifacts in various API methods.
  * Some errors from `wandb.Api` have better string representations.
  * Artifact.add\_reference, when used with file URIs for a directory and the name parameter, was incorrectly adding the value of `name` to the path of the file references.
  * Fixed a bug causing `Artifact.add_reference()` with `checksum=False` to log new versions of local reference artifacts without changes to the reference URI.

  View the [v0.19.6 changelog](https://github.com/wandb/wandb/releases/tag/v0.19.6) on GitHub.
</Update>

<Update label="v0.19.5" description="January 29, 2025">
  ## Added

  * Added `wandb login --base-url {host_url}` to login as an alias of `wandb login --host {host_url}`.

  ## Changed

  * Temporarily disabled collecting per-core CPU utilization stats.

  ## Fixed

  * Fixed a bug causing `offline` mode to make network requests when logging media artifacts. If you are using an older version of W\&B Server that does not support offline artifact uploads, use the setting `allow_offline_artifacts=False` to revert to older compatible behavior.
  * Expand sanitization rules for logged table artifact name to allow for hyphens and dots. This update brings the rules up-to-date with the current rules for artifact names. (Allowing letters, numbers, underscores, hyphens, and dots).
  * Correctly handle run rewind settings `fork_from` and `resume_from`.

  View the [v0.19.5 changelog](https://github.com/wandb/wandb/releases/tag/v0.19.5) on GitHub.
</Update>

<Update label="v0.19.4" description="January 16, 2025">
  ## Fixed

  * Fix incorrectly reported device counts and duty cycle measurements for TPUs with single devices per chip / multiple devices on the host and make TPU metrics sampling more robust.
  * Handle non-consecutive TPU device IDs in system monitor.

  View the [v0.19.4 changelog](https://github.com/wandb/wandb/releases/tag/v0.19.4) on GitHub.
</Update>

<Update label="v0.19.3" description="January 13, 2025">
  ## Fixed

  * Fix `wandb.Settings` update regression in `wandb.integration.metaflow`.

  View the [v0.19.3 changelog](https://github.com/wandb/wandb/releases/tag/v0.19.3) on GitHub.
</Update>

<Update label="v0.19.2" description="January 07, 2025">
  ## Added

  * Support JWT authentication in wandb-core.
  * Add support for logging nested custom charts.

  ## Changed

  * Calling `wandb.init(mode="disabled")` no longer disables all later runs by default. Use `wandb.setup(settings=wandb.Settings(mode="disabled"))` for this instead, or set `mode="disabled"` explicitly in each call to `wandb.init()`.

  ## Fixed

  * The stop button correctly interrupts runs whose main Python thread is running C code, sleeping, etc.
  * Remove unintentional print that occurs when inspecting `wandb.Api().runs()`.
  * Fix uploading large artifacts when using Azure Blob Storage.
  * The `wandb offline` command no longer adds an unsupported setting to `wandb.Settings`, resolving `ValidationError`.
  * Fix error when reinitializing a run, caused by accessing a removed attribute.
  * Fixed occasional deadlock when using `multiprocessing` to update a single run from multiple processes.
  * Prevent errors from bugs in older versions of `botocore &lt;  1.5.76`.
  * Fixed various checks against invalid `anonymous` settings value.

  ## Removed

  * The `wandb.wandb_sdk.wandb_setup._setup()` function's `reset` parameter has been removed. Note that this is a private function, even though there appear to be usages outside of the repo. Please `wandb.teardown()` instead of `_setup(reset=True)`.
  * In the private `wandb.wandb_sdk.wandb_setup` module, the `logger` and `_set_logger` symbols have been removed.

  ## Security

  * Bump `github.com/go-git/go-git` version to 5.13.0 to address CVE-2025-21613.
  * Bump `golang.org/x/net` version to 0.33.0 to address CVE-2024-45338.

  View the [v0.19.2 changelog](https://github.com/wandb/wandb/releases/tag/v0.19.2) on GitHub.
</Update>

<Update label="v0.19.1" description="December 13, 2024">
  ## Fixed

  * Fixed bug where setting WANDB\_\_SERVICE\_WAIT led to an exception during wandb.init().

  ## Changed

  * `run.finish()` displays more detailed information in the terminal and in Jupyter notebooks.
  * Improved error message for failing tensorboard.patch() calls to show the option to call tensorboard.unpatch() first.
  * Add projectId to deleteFiles mutation if the server supports it.

  ## Security

  * Bump `golang.org/x/crypto` version to 0.31.0 to address CVE-2024-45337.

  View the [v0.19.1 changelog](https://github.com/wandb/wandb/releases/tag/v0.19.1) on GitHub.
</Update>

<Update label="v0.19.0" description="December 05, 2024">
  ## Notable Changes

  This version drops Python 3.7 and removes the `wandb.Run.plot_table` method.
  This version adds pydantic>=2.6,\<33 as a dependency.

  ## Changed

  * Set default behavior to not create a W\&B Job.
  * Add pydantic>=2.6,\<33 as a dependency (@dmitryduev in [https://github.com/wandb/wandb/pull/8649](https://github.com/wandb/wandb/pull/8649) & estellazx.
    in [https://github.com/wandb/wandb/pull/8905](https://github.com/wandb/wandb/pull/8905))

  ## Removed

  * Remove `wandb.Run.plot_table` method. The functionality is still available and should be accessed using `wandb.plot_table`, which is now the recommended way to use this feature.
  * Drop support for Python 3.7.

  ## Fixed

  * Fix `ultralytics` reporting if there are no positive examples in a validation batch.
  * Debug printing for hyperband stopping algorithm printed one char per line.
  * Include the missing `log_params` argument when calling lightgbm's `wandb_callback` function.

  View the [v0.19.0 changelog](https://github.com/wandb/wandb/releases/tag/v0.19.0) on GitHub.
</Update>

<Update label="v0.18.7" description="November 13, 2024">
  ## Added

  * Added `create_and_run_agent` to `__all__` in `wandb/sdk/launch/__init__.py` to expose it as a public API.

  ## Changed

  * Tables logged in offline mode now have updated keys to artifact paths when syncing. To revert to old behavior, use setting `allow_offline_artifacts = False`.

  ## Deprecated

  * The `quiet` argument to `wandb.run.finish()` is deprecated, use `wandb.Settings(quiet=...)` to set this instead.

  ## Fixed

  * Fix `api.artifact()` to correctly pass the `enable_tracking` argument to the `Artifact._from_name()` method.

  View the [v0.18.7 changelog](https://github.com/wandb/wandb/releases/tag/v0.18.7) on GitHub.
</Update>

<Update label="v0.18.6" description="November 06, 2024">
  ## Added

  * Add a boolean `overwrite` param to `Artifact.add()`/`Artifact.add_file()` to allow overwrite of previously-added artifact files.

  ## Fixed

  * Add missing type hints of the `wandb.plot` module in the package stub.
  * Fix limiting azure reference artifact uploads to `max_objects`.
  * Fix downloading azure reference artifacts with `skip_cache=True`.
  * Fix multipart uploads for files with no content type defined in headers.
  * Fixed tensorboard failing to sync when logging batches of images.
  * Fixed behavior of `mode='x'`/`mode='w'` in `Artifact.new_file()` to conform to Python's built-in file modes.
  * Do not ignore parameter `distribution` when configuring sweep parameters from SDK.

  ## Changed

  * Added internal method, `api._artifact()`, to fetch artifacts so that usage events are not created if not called by an external user.
  * Changed default `mode` in `Artifact.new_file()` from `'w'` to `'x'` to accurately reflect existing default behavior.

  View the [v0.18.6 changelog](https://github.com/wandb/wandb/releases/tag/v0.18.6) on GitHub.
</Update>

<Update label="v0.18.5" description="October 17, 2024">
  ## Fixed

  * Import `Literal` from `typing_extensions` in Python 3.7; broken in 0.18.4.

  View the [v0.18.5 changelog](https://github.com/wandb/wandb/releases/tag/v0.18.5) on GitHub.
</Update>

<Update label="v0.18.4" description="October 17, 2024">
  ## Added

  * Track detailed metrics for Apple ARM systems including GPU, eCPU, and pCPU utilization, power consumption, and temperature, and memory/swap utilization.
  * Allow users to link Registry artifacts without inputting the organization entity name.
  * Added a warning message indicating that the `fps` argument will be ignored when creating a wandb.Video object from a file path string or a bytes object.
  * Update docstrings for `logged_artifacts` and `used_artifacts` methods in `Run` class.
  * The `_show_operation_stats` setting enables a preview of a better `run.finish()` UX.

  ## Fixed

  * Log power on AMD MI300X series GPUs.
  * Fixed typing issue of `wandb.Api`.
  * Ensure artifact objects are fully updated on `Artifact.save()`.

  ## Changed

  * Updated minimum version of `sentry-sdk` to 2.0.0 to address deprecation warnings.

  View the [v0.18.4 changelog](https://github.com/wandb/wandb/releases/tag/v0.18.4) on GitHub.
</Update>

<Update label="v0.18.3" description="October 01, 2024">
  ## Added

  * Add the ability to monitor the utilization metrics of Google's Cloud TPU devices.

  ## Fixed

  * Capture Nvidia GPU stats on Windows.
  * Fixed a regression introduced in v0.18.2 that affected capturing the names of Nvidia GPU devices.
  * `run.log_artifact()` no longer blocks other data uploads until the artifact upload finishes.
  * Fixed media dependency for rdkit updated from `rdkit-pypi` to `rdkit`.
  * Saving an artifact with many large files no longer exhausts OS threads.

  ## Changed

  * After `artifact = run.log_artifact()`, you must use `artifact.wait()` before operations that rely on the artifact having been uploaded. Previously, this wasn't necessary in some cases because `run.log_artifact()` blocked other operations on the run.

  View the [v0.18.3 changelog](https://github.com/wandb/wandb/releases/tag/v0.18.3) on GitHub.
</Update>

<Update label="v0.18.2" description="September 27, 2024">
  ## Added

  * Add `upsert_run_queue` method to `wandb.Api`.
  * Add `tags` parameter to `wandb.Api.artifacts()` to filter artifacts by tag.

  ## Fixed

  * Update the signature and docstring of `wandb.api.public.runs.Run.log_artifact()` to support artifact tags like `Run` instances returned by `wandb.init()`.
  * Add docstring for `wandb.watch()` to support auto-complete.
  * Fix glob matching in define metric to work with logged keys containing `/`.
  * Allow `a\.b` syntax in run.define\_metric() to refer to a dotted metric name.
    * NOTE: Not fixed if using `wandb.require("legacy-service")`
  * Fix Unknown image format error when uploading a gif through tensorboard.
  * Fix `OSError` from calling `Artifact.add_file()` with file paths on mounted filesystems.
  * Restored compatibility for macOS versions \<== 10.15 for wandb-core.

  View the [v0.18.2 changelog](https://github.com/wandb/wandb/releases/tag/v0.18.2) on GitHub.
</Update>

<Update label="v0.18.1" description="September 16, 2024">
  ## Fixed

  * Allow all users to read cache files when core is enabled.
  * Infinite scalars logged in TensorBoard are uploaded successfully rather than skipped.
  * Properly respect `WANDB_ERROR_REPORTING=false`. This fixes a regression introduced in 0.18.0.

  ## Changed

  * Remove sentry logging for sendLinkArtifact.
  * Default to capturing requirements.txt in Run.log\_code.

  View the [v0.18.1 changelog](https://github.com/wandb/wandb/releases/tag/v0.18.1) on GitHub.
</Update>

<Update label="v0.18.0" description="September 11, 2024">
  ## Notable Changes

  This version switches `wandb` to a new backend by enabling `wandb.require("core")` by default. This should not be a breaking change, but the new backend may have unexpected differences in behavior for legacy functionality and rare edge cases.

  ## Added

  * Add support for artifact tags, via `Artifact.tags` and `Run.log_artifact()`.

  ## Fixed

  * Detect the notebook name in VS Code's built-in jupyter server.

  ## Changed

  * The new "core" backend, previously activated using wandb.require("core"), is now used by default. To revert to the legacy behavior, add `wandb.require("legacy-service")` at the beginning of your script. Note: In a future minor release, the option to disable this new behavior will be removed.

  View the [v0.18.0 changelog](https://github.com/wandb/wandb/releases/tag/v0.18.0) on GitHub.
</Update>

<Update label="v0.17.9" description="September 05, 2024">
  ## Changed

  * Changed the default system metrics sampling interval to 10 seconds without averaging, while allowing custom intervals via `wandb.init(settings=wandb.Settings(_stats_sampling_interval=...))`.

  ## Deprecated

  * `define_metric(summary='best', goal=...)` is deprecated and soon will be removed, use `define_metric(summary='min')` or `define_metric(summary='min')` instead.

  View the [v0.17.9 changelog](https://github.com/wandb/wandb/releases/tag/v0.17.9) on GitHub.
</Update>

<Update label="v0.17.8" description="August 28, 2024">
  ## Added

  * Capture SM (Streaming Multiprocessor), memory, and graphics clock speed (MHz), (un)corrected error counts, fan speed (%), and encoder utilization for Nvidia GPU devices when using core.
  * Allow iterating over `wandb.Config` like a dictionary.
  * PR curves, images and histograms are supported when using TensorBoard with core enabled.
  * Added `wandb.require("legacy-service")` as the opt-out analog of `wandb.require("core")`.

  ## Fixed

  * Avoid leaving behind wandb-core process if user hits Ctrl+C twice.
  * Fix deprecation warnings arising from NumPy >= 2.1 by removing `newshape` argument from `numpy.reshape`.
  * Skip uploading/downloading GCS reference artifact manifest entries corresponding to folders.

  ## Deprecated

  * Ability to disable the service process (`WANDB__DISABLE_SERVICE`) is deprecated and will be removed in the next minor release.

  View the [v0.17.8 changelog](https://github.com/wandb/wandb/releases/tag/v0.17.8) on GitHub.
</Update>

<Update label="v0.17.7" description="August 15, 2024">
  ## Fixed

  * Ensure Nvidia GPU metrics are captured if `libnvidia-ml.so` is unavailable when using core.
  * Allow `define_metric("x", step_metric="x")` when using core.
  * Correctly upload empty files when using core.
  * Fix occasional "send on closed channel" panic when finishing a run using core.

  View the [v0.17.7 changelog](https://github.com/wandb/wandb/releases/tag/v0.17.7) on GitHub.
</Update>

<Update label="v0.17.6" description="August 08, 2024">
  ## Added

  * Specify job input schemas when calling manage\_config\_file or manage\_wandb\_config to create a nicer UI when launching the job.
  * Use the filesystem rather than protobuf messages to transport manifests with more than 100k entries to the core process.
  * Adds the `box3d` constructor for `Box3D`.

  ## Changed

  * `run.define_metric()` raises an error when given extraneous arguments.
  * In disabled mode, use the `wandb.sdk.wandb_run.Run` class instead of `wandb.sdk.wandb_run.RunDisabled`, which has been removed.
  * When `WANDB_MODE = offline` calling `artifact.download()` now throws an error instead of stalling.

  ## Fixed

  * Correctly handle directory stubs when logging external artifact in azure storage account with Hierarchical Namespace enabled.
  * Docstring in `api.runs()` regarding default sort order, missed in [https://github.com/wandb/wandb/pull/7675](https://github.com/wandb/wandb/pull/7675).

  View the [v0.17.6 changelog](https://github.com/wandb/wandb/releases/tag/v0.17.6) on GitHub.
</Update>

<Update label="v0.17.5" description="July 19, 2024">
  ## Added

  * When using wandb-core, support multipart uploads to S3.

  ## Changed

  * `run.finish()` may raise an exception in cases where previously it would `os._exit()`.
  * `run.link_artifact()` can now surface server errors.

  ## Fixed

  * Handle `path_prefix`es that don't correspond to directory names when downloading artifacts.
  * Fix creating or updating an artifact with the `incremental=True` flag.
  * Use filled resource\_arg macros when submitting W\&B Launch jobs to AmazonSageMaker.

  View the [v0.17.5 changelog](https://github.com/wandb/wandb/releases/tag/v0.17.5) on GitHub.
</Update>

<Update label="v0.17.4" description="July 03, 2024">
  ## Added

  * Support queue template variables in launch sweep scheduler jobs.

  ## Fixed

  * Use `sys.exit()` instead of `os._exit()` if an internal subprocess exits with a non-zero code.
  * Fix an occasional race condition when using `core` that could affect run logs.
  * Fix OSError on `Artifact.download(skip_cache=True)` when encountering different filesystems.

  View the [v0.17.4 changelog](https://github.com/wandb/wandb/releases/tag/v0.17.4) on GitHub.
</Update>

<Update label="v0.17.3" description="June 24, 2024">
  ## Fixed

  * Correctly name the netrc file on Windows as `_netrc`.
  * With core enabled, nested `tqdm` bars show up correctly in the Logs tab.
  * Fix W\&B Launch registry ECR regex separating tag on forward slash and period.

  View the [v0.17.3 changelog](https://github.com/wandb/wandb/releases/tag/v0.17.3) on GitHub.
</Update>

<Update label="v0.17.2" description="June 17, 2024">
  ## Added

  * Add prior runs when creating a sweep from the CLI.

  ## Fixed

  * Fix issues with `numpy>=2` support.
  * Fix "UnicodeDecodeError: 'charmap'" when opening HTML files on Windows by specifying UTF-8 encoding.
  * Ensure `Artifact.delete()` on linked artifacts only removes link, not source artifact.
  * Sweep runs no longer appear to be resumed when they are not by @TimH98 [https://github.com/wandb/wandb/pull/7684](https://github.com/wandb/wandb/pull/7684).

  ## Changed

  * Upgrade github.com/vektah/gqlparser/v2 from 2.5.11 to 2.5.16.

  View the [v0.17.2 changelog](https://github.com/wandb/wandb/releases/tag/v0.17.2) on GitHub.
</Update>

<Update label="v0.17.1" description="June 07, 2024">
  ## Added

  * Added `api.runs().histories()` to fetch history metrics for runs that meet specified conditions.
  * Display warning when Kubernetes pod fails to schedule.
  * Added `ArtifactCollection.save()` to allow persisting changes.
  * Added the ability to overwrite history of previous runs at an arbitrary step and continue logging from that step.
  * Added new Workspace API for programatically editing W\&B Workspaces.
  * Added `Artifact.unlink()` to allow programmatic unlinking of artifacts.
  * Added basic TensorBoard support when running with `wandb.require("core")`.
    * The TensorBoard tab in W\&B will work.
    * Charts show up in W\&B, possibly better than when running without core.
    * Not all types of data are supported yet. Unsupported data is not shown in charts.

  ## Fixed

  * Fix `define_metric` behavior for multiple metrics in `shared` mode.
  * Correctly pass in project name to internal api from run while calling run.use\_artifact().
  * Correctly upload console output log files when resuming runs enabled with `console_multipart` setting by @kptkin in [https://github.com/wandb/wandb/pull/7694](https://github.com/wandb/wandb/pull/7694) and @dmitryduev in [https://github.com/wandb/wandb/pull/7697](https://github.com/wandb/wandb/pull/7697).
  * Interpret non-octal strings with leading zeros as strings and not integers in sweep configs by @KyleGoyette [https://github.com/wandb/wandb/pull/7649](https://github.com/wandb/wandb/pull/7649).
  * Support Azure repo URI format in Launch @KyleGoyette [https://github.com/wandb/wandb/pull/7664](https://github.com/wandb/wandb/pull/7664).
  * Fix path parsing for artifacts with aliases containing forward slashes.
  * Add missing docstrings for any public methods in `Api` class.
  * Correctly add latest alias to jobs built by the job builder @KyleGoyette [https://github.com/wandb/wandb/pull/7727](https://github.com/wandb/wandb/pull/7727).

  ## Changed

  * Option to change naming scheme for console output logs from `output.log` to `logs/YYYYMMDD_HHmmss.ms_output.log`.
  * Require `unsafe=True` in `use_model` calls that could potentially load and deserialize unsafe pickle files by @anandwandb [https://github.com/wandb/wandb/pull/7663](https://github.com/wandb/wandb/pull/7663).
  * Update order in api.runs() to ascending to prevent duplicate responses by @thanos-wandb [https://github.com/wandb/wandb/pull/7675](https://github.com/wandb/wandb/pull/7675).
  * Eliminate signed URL timeout errors during artifact file uploads in core.

  ## Deprecated

  * Deprecated `ArtifactCollection.change_type()` in favor of `ArtifactCollection.save()`.

  View the [v0.17.1 changelog](https://github.com/wandb/wandb/releases/tag/v0.17.1) on GitHub.
</Update>

<Update label="v0.17.0" description="May 07, 2024">
  ## Notable Changes

  Renamed `wandb.plots` to `wandb.plot`, renamed all integrations from `wandb.<name>` to `wandb.integration.<name>`, and removed the `[async]` extra.

  This version packages the `wandb-core` binary, formerly installed by the `wandb-core` Python package on PyPI. The `wandb-core` package is now unused and can be uninstalled.

  ## Added

  * The `wandb` package now includes the `wandb-core` binary.
    * `wandb-core` is a new and improved backend for the W\&B SDK that focuses on performance, versatility, and robustness.
    * Currently, it is opt-in. To start using the new backend, add `wandb.require("core")` to your script after importing `wandb`.
  * `wandb-core` now supports Artifact file caching by @moredatarequired in [https://github.com/wandb/wandb/pull/7364](https://github.com/wandb/wandb/pull/7364) and [https://github.com/wandb/wandb/pull/7366](https://github.com/wandb/wandb/pull/7366).
  * Added artifact\_exists() and artifact\_collection\_exists() methods to Api to check if an artifact or collection exists.
  * `wandb launch -u <git-uri | local-path> ` creates and launches a job from the given source code.

  ## Fixed

  * Prevent crash on `run.summary` for finished runs.
  * Correctly report file upload errors when using wandb-core.
  * Implemented a stricter check for AMD GPU availability.
  * Fixed `run.save()` on Windows.
  * Show a warning instead of failing when using registries other than ECR and GAR with the Kaniko builder.
  * Fixed `wandb.init()` type signature including `None`.

  ## Changed

  * When using `wandb-core` need to specify a required flag (`wandb.require("core")`) to enable it, before it was picked up automatically.
  * Use ETags instead of MD5 hashes for GCS reference artifacts.

  ## Removed

  * Removed the deprecated `wandb.plots.*` functions and top-level third-party integrations `wandb.[catboost,fastai,keras,lightgbm,sacred,xgboost]`. Please use `wandb.plot` instead of `wandb.plots` and `wandb.integration.[catboost,fastai,keras,lightgbm,sacred,xgboost]` instead of `wandb.[catboost,fastai,keras,lightgbm,sacred,xgboost]`.
  * Removed the `[async]` extra and the `_async_upload_concurrency_limit` setting.
  * Removed undocumented settings: `_except_exit` and `problem`.

  View the [v0.17.0 changelog](https://github.com/wandb/wandb/releases/tag/v0.17.0) on GitHub.
</Update>

<Update label="v0.16.6" description="April 03, 2024">
  ## Added

  * Added support for overriding kaniko builder settings in the agent config.
  * Added link to the project workspace of a run in the footer.
  * Added support for overriding stopped run grace period in the agent config.
  * Added setting (`_disable_update_check`) to disable version checks during init.
  * `WandbLogger.sync` in the OpenAI Fine-Tuning integration gets a new `log_datasets` boolean argument to turn off automatic logging of datasets to Artifacts.
  * Reduced default status print frequency of launch agent. Added verbosity controls to allow for increased status print frequency and printing debug information to stdout.

  ## Changed

  * Limit policy option on artifact cli's put() to choices, \["mutable", "immutable"].
  * Updated artifact public api methods to handle nullable Project field on the ArtifactSequence/ArtifactCollection type, based on gorilla server changes.

  ## Fixed

  * Fixed `run.save()` not working with files inside `run.dir`, introduced in previous release.
  * Fixed rare panic during large artifact uploads.
  * Fixed wandb.login() causing runs not to be associated with launch queue.
  * Fixed job artifact download failing silently and causing run crash when using W\&B Launch by @KyleGoyette [https://github.com/wandb/wandb/pull/7285](https://github.com/wandb/wandb/pull/7285).
  * Fix handling of saving training files to Artifacts in the OpenAI Fine-Tuning integration.

  View the [v0.16.6 changelog](https://github.com/wandb/wandb/releases/tag/v0.16.6) on GitHub.
</Update>

<Update label="v0.16.5" description="March 25, 2024">
  ## Added

  * Added feature to move staging files to cache (instead of copying) for mutable artifact file uploads when caching is enabled.
  * Added support to skip caching files to the local filesystem while uploading files to artifacts.
  * Added support to skip staging artifact files during upload by selecting a storage policy.
  * Preliminary support for forking a run using `wandb.init(fork_from=...)`.
  * `run.save()` accepts `pathlib.Path` values;.

  ## Changed

  * When printing the run link point to the workspace explicitly.

  ## Fixed

  * In case of transient server issues when creating the wandb API key kubernetes secret, we'll retry up to 5 times.

  ## Removed

  * When printing run's information in the terminal remove links to jobs.

  View the [v0.16.5 changelog](https://github.com/wandb/wandb/releases/tag/v0.16.5) on GitHub.
</Update>

<Update label="v0.16.4" description="March 05, 2024">
  ## Added

  * Added ability to change artifact collection types.
  * Add support for installing deps from pyproject.toml.
  * Support kaniko build with user-provided pvc and docker config.
  * Added ability to import runs between W\&B instances.

  ## Changed

  * Wandb-core rate-limits requests to the backend and respects RateLimit-\* headers.
    by @timoffex in [https://github.com/wandb/wandb/pull/7065](https://github.com/wandb/wandb/pull/7065)

  ## Fixed

  * Fix passing of template variables in the sweeps-on-launch scheduler.
  * Link job artifact to a run to be specified as input.
  * Fix sagemaker entrypoint to use given entrypoint.
  * Parse upload headers correctly.
  * Properly propagate server errors.
  * Make file upload faster by using parallelism.
  * Don't send git data if it's not populated.
  * Fix console logging resumption, avoid overwrite.
  * Remove hostname validation when using --host on wandb login.
  * Don't discard past visualizations when resuming a run.
  * Avoid retrying on conflict status code.
  * Fix visualization config merging for resumed runs in wandb-core.
  * Replace usage of standard library's json with `segmentio`'s.
  * Remove stderr as writer for the logs.
  * Disable negative steps from initialization.
  * Fix report loading in pydantic26.
  * Revert "make upload request async to support progress reporting (#6497)".
  * Fix entrypoint specification when using a Dockerfile.wandb.
  * Fix stream releasing probe handle too early.
  * Always attempt to pull latest image for local container.

  View the [v0.16.4 changelog](https://github.com/wandb/wandb/releases/tag/v0.16.4) on GitHub.
</Update>

<Update label="v0.16.3" description="February 06, 2024">
  ## Enhancements

  * Generate data type info in core.
  * Add support for Launch.
  * Added option to control number of grouped sampled runs in reports.
  * Add shared mode to enable multiple independent writers to the same run.
  * Reduce artifact download latency via optional cache copy + threads.
  * Add partial file downloads, via directory prefix.
  * Update the Diffusers Integration.
  * Update Ultralytics Integration.
  * Add Pytorch Lightning Fabric Logger.
  * Update go packages.

  ## Fixes

  * Remove project and runner fields from agent config.
  * Recognize deleted k8s jobs as failed.
  * Warn of extra fields in environment block instead of erroring.
  * Entity override bug where ENVVAR is prioritized over kwargs.
  * Local container runner doesn't ignore override args.
  * Merge-update config with sweep/launch config.
  * Fix retry logic in wandb-core and system\_tests conftest.
  * Use RW locks in system monitor's assets management.
  * Set build context to entrypoint dir if it contains Dockerfile.wandb.
  * Security(launch): Mount wandb api key in launch job pods from a k8s secret.
  * Wandb job create should not look for requirements.txt if Dockerfile.wandb is next to entrypoint.
  * Fix \_parse\_path when only id is passed to wandb.Api().run().
  * Update video.py: Fix fps bug.
  * Clean up temp folders.
  * Fix long artifact paths on Windows.
  * Update Report API to work with pydantic2.6.
  * Fetch all commits to enable checking out by sha.
  * Dont swallow exceptions in pyagent.
  * Artifact file upload progress in nexus.
  * Exercise caution in system monitor when rocm-smi is installed on a system with no amd gpus.
  * Typo in cli.py.
  * Remove deadsnakes from accelerator build step.

  View the [v0.16.3 changelog](https://github.com/wandb/wandb/releases/tag/v0.16.3) on GitHub.
</Update>

<Update label="v0.16.2" description="January 09, 2024">
  ## Enhancements

  * Refactor store logic and add store writer.
  * Add AMD GPU monitoring.
  * Add console log file upload.
  * Add registry uri field to builders.
  * Add `wandb beta sync` feature to upload runs to W\&B.
  * CLI supports allow-listed queue parameters.
  * Add support for requirements and patch.diff.
  * Capture SLURM-related env vars in metadata.
  * \--priority flag on `wandb launch` command to specify priority when enqueuing jobs.
  * Add verify feature to wandb login.
  * Sweeps on Launch honors selected job priority for sweep runs.
  * Commence operation SDKrab.
  * Make upload request async to support progress reporting.
  * Add TensorBoard log dir watcher.
  * Upload wandb-summary.json and config.yaml files.

  ## Fixes

  * Update error message and remove extra.
  * Clean up issues with file sending.
  * Add jitter to retry clients' backoff strategy.
  * Only skip file download if digest matches.
  * Fix resume and add tests.
  * Capture errors from the creation of k8s job from yaml.
  * Get default entity before checking template vars.
  * Remove run creation for artifact downloads if not using core.
  * Retrieve ETag for ObjectVersion instead of Object for versioned buckets.
  * Revert #6759 and read object version etag in place.
  * Put back the upload file count.
  * Add send cancel request to sender.
  * Check errors in memory monitoring.
  * Add job\_type flag in CLI to allow override of job\_type.
  * Handle ultralytics utils import refactor.
  * Download artifacts with wandb-core without an active run.
  * Fix error logging matplotlib scatter plot if the minimum version of plotly library is not met.
  * Allow overriding the default .netrc location with NETRC env var.

  View the [v0.16.2 changelog](https://github.com/wandb/wandb/releases/tag/v0.16.2) on GitHub.
</Update>

<Update label="v0.16.1" description="December 05, 2023">
  ## Enhancements

  * Remove recursive download.
  * Add debounce summary in handler.
  * Fix bug in ultralytics import and version pinning.
  * Support template variables when queueing launch runs.
  * Add --skip-console option to offline sync cli command.
  * Add basic graphql versioning mechanism.
  * Add Apple M\* GPU stats monitoring.
  * Add helper to load wandb.Config from env vars.
  * Port OpenAI WandbLogger for openai-python v1.0.
  * Fix version check for openAI WandbLogger.
  * Diffusers autologger.
  * Adding parameter to image to specify file type jpg, png, bmp, gif.

  ## Fixes

  * Make offline sync work properly.
  * Fix run existence check to not depend on files being uploaded in the run.
  * Gcp storage uri verifaction failed due to improper async wrapping.
  * Updating summary with nested dicts now doesn't throw an error.
  * Add prioritization mode to RunQueue create.
  * Create symlink to the server logs in the runs folder.
  * Single value problem in wandb/wandb\_torch.py::log\_tensor\_stats.
  * Tmin vs tmax in wandb/wandb\_torch.py::log\_tensor\_stats.
  * Minor fix up for non server cases.
  * Make old settings more robust.
  * Handle tags when resuming a run.

  View the [v0.16.1 changelog](https://github.com/wandb/wandb/releases/tag/v0.16.1) on GitHub.
</Update>

<Update label="v0.16.0" description="November 07, 2023">
  ## Enhancements

  * Add nested config support.
  * Finish artifact saver.
  * Add sync file counts in the footer.
  * Implement directory watcher and related functionality.
  * Run agent on an event loop.
  * Add sampled history.
  * Prototype for models api.
  * Register sweep scheduler virtual agents async.
  * Make file uploads work with local.
  * Allow users to log custom chart tables in a different section.
  * Generalize uploader to filemanager to allow downloads.
  * Drop python 3.6 support.
  * Delete staging files in Nexus.
  * Set up artifact downloads to use sdk nexus core.
  * Add file upload progress and make completion callback a list of callbacks.
  * Add wandb.ai/run-id label to jobs.

  ## Fixes

  * Ensures that complete run.config is captured in SageMaker.
  * Improve error handling for gitlib for FileNotFoundErrors.
  * Increase max message size, handle errors.
  * Agent better balances multiple queues.
  * Remove versioning enabled check in GCS reference handler.
  * Add google-cloud-aiplatform to launch shard.
  * Add use\_artifact as passthrough messages.
  * Adjust vertex environment variables.
  * Update artifacts cache file permissions for NamedTemporaryFile.
  * Fix a number of issues.
  * Saving mlp figure to buffer and reading with PIL should specify format.
  * Send content-length, check response status code in uploader.
  * Fix step logic when resuming runs with no metrics logged.
  * Hook\_handle being set to list instead of dict on unhook.
  * Verify gcp credentials before creating vertex job.
  * Add load option to docker buildx.
  * Fix perf regressions in artifact downloads and fix file download location.
  * Add warning when `log_code` can't locate any files.
  * Adjust ipython hooks for v8.17.

  View the [v0.16.0 changelog](https://github.com/wandb/wandb/releases/tag/v0.16.0) on GitHub.
</Update>

<Update label="v0.15.12" description="October 03, 2023">
  ## Enhancements

  * Implement config debouncing mechanism.
  * Fix channel swapping on ultrlytics classification task.
  * Implement nexus alpha cpp interface.
  * Expose system metrics in the run object (PoC).
  * Pin ultralytics version support to `v8.0.186`.

  ## Fixes

  * Get logs from failed k8s pods.
  * Allow adding s3 bucket as reference artifact.
  * Fix race condition in agent thread clean up.
  * Don't assume run and its i/o artifacts are in the same project.
  * Fix wandb.Api().run(run\_name).log\_artifact(artifact).
  * ValueError with None value in sweep.
  * Fix typo in s3 handler.
  * Fix the argument order for new argument target\_fraction.
  * Fix potential race in config debouncer.
  * Fix graphql type mapping.
  * Fix concurrency limit in uploader.

  View the [v0.15.12 changelog](https://github.com/wandb/wandb/releases/tag/v0.15.12) on GitHub.
</Update>

<Update label="v0.15.11" description="September 21, 2023">
  ## Enhancements

  * Add support for code saving in script mode.
  * Add support for `policy=end` in `wandb.save()`.
  * Add system info to metadata.
  * Add nvidia gpu system info to metadata.
  * Delete run queues with public api.
  * Introduce custom proxy support for wandb http(s) traffic.

  ## Fixes

  * Fix logger when logging filestream exception.
  * Use watch api to monitor launched CRDs.
  * Forbid enqueuing docker images without target project.
  * Add missing Twitter import for API users.
  * Get S3 versionIDs from directory references.
  * Make watch streams recover from connection reset.
  * Use json.loads(..., strict=False) to ignore invalid utf-8 and control characters in api.Run.load.
  * Correctly identify colab as a jupyter-like env in settings.
  * Improve memory management for summary updates.
  * Add environment variable to configure batch size for download urls.
  * Fail rqis if no run is created.

  ## Cleanup

  * Update README.md.

  View the [v0.15.11 changelog](https://github.com/wandb/wandb/releases/tag/v0.15.11) on GitHub.
</Update>

<Update label="v0.15.10" description="September 06, 2023">
  ## Enhancements

  * Add async support to `Autologger` API and enable it for Openai.
  * Add official support for python 3.11 and drop support for python 3.6.
  * Implement network logging and file pusher timeout for debugging.
  * Set ttl(time to live) for artifact versions.
  * Add support for define metric.
  * Include agent version when creating launch agent.
  * Runless git jobs can use requirements.txt in parent directories.
  * Retrieve the parent collection from an Artifact.
  * Improve file uploads.
  * Add run id option to artifact put method to log artifacts to existing runs.
  * Add metadata property to Run object.
  * Support setting a custom Dockerfile in launch overrides.
  * Add Nvidia GPU asset to system monitor.
  * Enable deleting artifact collections from SDK.
  * Add dockerfile CLI param & use Dockerfile.wandb by default if present.
  * Extend cache cleanup to allow specifying a target fraction.
  * Add an eval-able repr to ArtifactManifestEntry.
  * Enable docker-based wheel building for nexus.
  * Add Nvidia GPU asset to system monitor.
  * Clear the cache on add to prevent overflow.
  * Capture disk i/o utilization in system metrics.
  * Add disk io counters to monitor metrics.
  * Make paths for disk usage monitoring configurable.
  * Use `WANDB_SWEEP_ID` to include a run in an existing sweep.
  * Handle LinkArtifact calls made to Nexus Core.
  * Fix retry logic for http clients and allow user customization.
  * Support user defined headers in the gql client transport.
  * Enable set types in wandb.Config.
  * Visualize images with bbox overlays for `ultralytics`.
  * Add exponential decay sampling utility for line\_plot.
  * Always print the traceback inside of the `wandb.init()` context manager.
  * Add setting to disable automatic machine info capture.

  ## Fixes

  * Extend try in agent loop to cover all job handling.
  * Guard against undefined filestream timeout.
  * Error if code artifact underlying job has been deleted.
  * Use a unique name for the artifact created by `verify`.
  * Use resume=allow when auto requeuing.
  * Correct entrypoint path from disabled git repo subir.
  * Override individual job resource\_args.
  * Fix import issue to support python 3.6.
  * Fix override entrypoint when using sweeps on launch without a scheduler job.
  * Fix resume reference when nil.
  * Further speed up import time.
  * Fix sample kubernetes agent manifest secret mount.
  * Rm unused import.
  * Watch to get kubernetes run statuses.
  * Prohibit saving artifacts to a different project than their base artifact.
  * Require existing artifacts to save to their source entity/project.
  * Adjust system monitor start and stop functionality.
  * Remove suspect characters when directory creation fails.
  * Default log\_code exclusion behavior now correctly handles `wandb` in the root path prefix.
  * Disallow project queue creation.
  * Catch all sweep set state errors.
  * Create\_job now works from jupyter notebook.
  * Fix race condition for defer and update control.
  * Improved handling and logging of tensor types.
  * Launch cli command should exit with non-zero status if underlying launched run exits with non-zero status.
  * Fix correctness for offline mode.
  * Reports api - fix media\_keys json path.
  * Allow uint8 images to be logged as wandb.Image().
  * Fall back to /tmp/username/.config/wandb in old settings.
  * Use UpsertBucketRetryPolicy in all gql.UpsertBucket calls.
  * Update report id validation and encoding.
  * Add support for propagating messages from the internal process.

  ## Cleanup

  * Fix to new ruff rule E721 additions.

  View the [v0.15.10 changelog](https://github.com/wandb/wandb/releases/tag/v0.15.10) on GitHub.
</Update>

<Update label="v0.15.9" description="August 28, 2023">
  ## Enhancements

  * Launch sweep schedulers to team queues from UI.
  * Make vertex launcher more customizable.
  * Default to noop builder if docker not installed.

  ## Fixes

  * Use built in entrypoint and args commands for sagemaker.
  * Copy parent source project info to new draft artifact.
  * Avoid error at end of run with bigints.
  * Manually created image jobs can rerun correctly.

  View the [v0.15.9 changelog](https://github.com/wandb/wandb/releases/tag/v0.15.9) on GitHub.
</Update>

<Update label="v0.15.8" description="August 01, 2023">
  ## Enhancements

  * Use mutation createRunFiles to get uploadUrls.
  * Add create\_run\_queue to public API.
  * Add hidden option to use orjson instead of json.
  * Improve error message when building with noop builder.
  * Create launch agent includes agent config if present.
  * Check if job ingredients exist before making job.
  * Gracefully handle Kubernetes 404 error.

  ## Fixes

  * Only creating new project if it doesn't already exist.
  * Support namespace in metadata key of resource args.
  * Use "" instead of None for project kwarg when no project given.
  * Add + to torch cpu regex + tests.
  * Implement timeout for file\_stream and add debug logs.
  * Fix collection filtering when getting aliases.
  * Replace `dir_watcher` settings with SettingsStatic.
  * Set correct base for incremental artifacts.
  * Drop https from azure registries to ensure compatibility with \$ macro.
  * Handle None description correctly.
  * Don't create k8s secret if it already exists.
  * Drop S3 bucket versioning check.
  * Speed up import time and fix `pkg_resources` DeprecationWarning.

  View the [v0.15.8 changelog](https://github.com/wandb/wandb/releases/tag/v0.15.8) on GitHub.
</Update>

<Update label="v0.15.7" description="July 25, 2023">
  ## Fixes

  * Images not syncing until the end run (revert #5777).

  View the [v0.15.7 changelog](https://github.com/wandb/wandb/releases/tag/v0.15.7) on GitHub.
</Update>

<Update label="v0.15.6" description="July 24, 2023">
  ## Enhancements

  * Add job link to wandb footer.
  * Re-implement job requeueing, fixed cancel behavior.
  * Manually create jobs from cli.
  * Allow users to specify job name via the `job_name` setting.
  * Add an simplified trace API to log prompt traces.
  * Support `.keras` model format with `WandbModelCheckpoint` and TF 2.13.0 compatible.
  * Initial support for migrating W\&B runs and reports between instances.

  ## Fixes

  * Make LightGBM callback compatible with 4.0.0.
  * Use default settings for project retrieval if available.

  View the [v0.15.6 changelog](https://github.com/wandb/wandb/releases/tag/v0.15.6) on GitHub.
</Update>

<Update label="v0.15.5" description="July 05, 2023">
  ## Enhancements

  * Improve handling of docker image job names and tags.
  * Support kaniko builds on AKS.
  * Allow kaniko builds to run in other namespaces.
  * Support access key for Azure references.
  * Add information to failed run queue items, support warnings for run queue items.
  * Allow direct configuration of registry uri for all registries.
  * Enhance download URL fetching process with batch and retry logic.
  * Add flag to skip missing S3 references in `Artifact.download`.
  * Implement job requeueing when pod disconnects.
  * Add setting to disable setproctitle.

  ## Fixes

  * Handle uri schemes in LogicalPath.
  * Update object storage to include reference and prevent id reuse.
  * Update click package version requirements.
  * Improve lazy import to be thread-safe.
  * Change typo in kaniko image name.
  * Correct date parsing in SageMaker configuration.
  * Make docker build non interactive to prevent region based questions.
  * Update "cuda" base image path to "accelerator" base image path.
  * Replace artifact name with placeholder to skip validation.
  * Prevent jobs with large outputs from hanging on local-container.
  * Ensure resume does not push sensitive info.
  * Fix handling of references when downloading.
  * Correct launch sweep author to personal username.
  * Change artifact methods and attributes to private.

  View the [v0.15.5 changelog](https://github.com/wandb/wandb/releases/tag/v0.15.5) on GitHub.
</Update>

<Update label="v0.15.4" description="June 06, 2023">
  ## Enhancements

  * Set job source in settings.
  * Launch sweeps controlled by wandb run.
  * Add autolog for Cohere python SDK.
  * Support launching custom k8s objects.
  * Conserve memory when hashing files.
  * Add new\_draft method to modify and log saved artifacts as new version.
  * Don't install frozen reqs if there is a reqs file.
  * Don't remove temp files from artifacts cache by default.
  * Add source\_entity and update sequenceName handling.
  * Add 'remove' to Artifacts API.
  * Optuna scheduler for sweeps on launch.
  * Support notebook job creation.
  * Enable launch macros for all runners.
  * Add autologging for supported huggingface pipelines.
  * Add usage metrics and table logging to OpenAI autologger.
  * Add support for monitoring AMD GPU system metrics.
  * Capture absolute GPU memory allocation.

  ## Fixes

  * Ensure wandb can be used in AWS lambda.
  * Permit `LogicalPath` to strip trailing slashes.
  * Exercise caution when creating \~/.config/wandb/settings file.
  * Update custom chart query handling and add alternate constructor for table-based charts.
  * Add s3 multipart uploading for artifact files.
  * Handle incompatible artifact name strings.
  * Docker runner always pull for image sourced jobs.
  * Improve error handling for package installation.
  * Custom k8s objects respect command/args overrides.
  * Remove entity, project from valid properties and adjust name handling.
  * Use env var for launch agent base url.
  * Write to the cache defensively (catch OSError).
  * Handle exception in finish\_thread\_id and fail run queue items.
  * Add pull secrets for pre made images when registry is specified.
  * Read kaniko pod sa name from env var.
  * Misc gcp fixes.
  * Support local environment and registry declaration.
  * Support ssh git urls and submodules in agent.
  * Update git repo handling for failure cases and rename to gitlib.
  * Unify offline and online mode during init and fix multiprocess attach.
  * Prevent errors by checking for `wandb.run` in Gym integration.
  * Fix wandb tfevent sync issue.

  View the [v0.15.4 changelog](https://github.com/wandb/wandb/releases/tag/v0.15.4) on GitHub.
</Update>

<Update label="v0.15.3" description="May 17, 2023">
  ## Fixes

  * Allow SDK to work if SA token can't be read.
  * Clean up the k8s token discovery logic in util.py::image\_id\_from\_k8s.
  * Update `WandbTracer` to work with new langchain version.
  * Update summary for changed keys only.

  View the [v0.15.3 changelog](https://github.com/wandb/wandb/releases/tag/v0.15.3) on GitHub.
</Update>

<Update label="v0.15.2" description="May 05, 2023">
  ## Fixes

  * Update WandbTracer for new langchain release by @parambharat @tssweeney in [https://github.com/wandb/wandb/pull/5467](https://github.com/wandb/wandb/pull/5467).
  * Fix error message in langchain wandb\_tracer version check.

  View the [v0.15.2 changelog](https://github.com/wandb/wandb/releases/tag/v0.15.2) on GitHub.
</Update>

<Update label="v0.15.1" description="May 02, 2023">
  ## Enhancements

  * Implement new Kubernetes runner config schema.
  * Allow platform override for docker builder.
  * Get full name of artifact for easier artifact retrieval.
  * Make default root for artifacts download configurable.
  * Add Azure storage handler in SDK.
  * Add method to convert wandb.Table to pandas.DataFrame.
  * Sweeps on launch command args passed as params.

  ## Fixes

  * Don't assume keys in args and config refer to the same thing.
  * Make ElasticContainerRegistry environment handle "ImageNotFoundException" gracefully.
  * Disable kaniko builder retry.
  * Refine error message for auth error.
  * Kubernetes runner does not respect override args.
  * Allow attr-dicts as sweeps configs.
  * Checksum the read-only staging copy instead of the original file.
  * Skip getting run info if run completes successfully or is from a different entity.
  * Default to project "uncategorized" instead of "None" when fetching artifacts.
  * Add enabled check to gym VideoRecorder.
  * Fix handling of default project and entity.
  * Update import\_hook.py with latest changes in the wrapt repository.
  * Fix support for local urls in k8s launch agent.
  * Improve notebook environment detection and testing.
  * Implement recursive isinstance check utility for the Settings object.
  * Correctly parse edge cases in OpenMetrics filter definitions in System Monitor.
  * Update debug logs to include SDK's version.
  * Filter AWS Trainium metrics by local rank if executed with torchrun.
  * Inform users about WandbTracer incompatibility with LangChain > 0.0.153.

  View the [v0.15.1 changelog](https://github.com/wandb/wandb/releases/tag/v0.15.1) on GitHub.
</Update>

<Update label="v0.15.0" description="April 19, 2023">
  ## Enhancements

  * Add support for LangChain media type.
  * Add autolog for OpenAI's python library by @dmitryduev @parambharat @kptkin @raubitsj in [https://github.com/wandb/wandb/pull/5362](https://github.com/wandb/wandb/pull/5362).

  ## Fixes

  * Add function signature wrapper to the patched openai methods.
  * Adjust OpenAI autolog public API to improve user experience by @dmitryduev @kptkin @raubitsj in [https://github.com/wandb/wandb/pull/5381](https://github.com/wandb/wandb/pull/5381).

  View the [v0.15.0 changelog](https://github.com/wandb/wandb/releases/tag/v0.15.0) on GitHub.
</Update>

<Update label="v0.14.2" description="April 07, 2023">
  ## Fixes

  * Fix `wandb sync` regression.

  View the [v0.14.2 changelog](https://github.com/wandb/wandb/releases/tag/v0.14.2) on GitHub.
</Update>

<Update label="v0.14.1" description="April 05, 2023">
  ## Enhancements

  * Improve run.log\_artifact() with default type and path references.
  * Add opt-in support for async artifact upload.
  * Update summary for changed keys only.
  * Use a persistent session object for GraphQL requests.
  * Allow setting of extra headers for the gql client.
  * Allow filtering metrics based on OpenMetrics endpoints.

  ## Fixes

  * More informative message when failing to create staging artifact directory.
  * Set default value for Kubernetes backoffLimit to 0.
  * Remove default sorting when dumping config into a yaml file.
  * Fix encoding for html types on windows.
  * Clean up auto resume state when initializing a new run.
  * Harden `wandb.init()` error handling for backend errors.
  * Fix system monitor shutdown logic.
  * Allow users to specify pinned versions in requirements.txt.
  * Make `wandb.log()` handle empty string values properly.
  * Raise exception when accessing methods and attributes of a finished run.

  ## Cleanup

  * Use 'secrets' module instead of custom random token generator.
  * Move \_manifest\_json\_from\_proto to sender.py.

  View the [v0.14.1 changelog](https://github.com/wandb/wandb/releases/tag/v0.14.1) on GitHub.
</Update>

<Update label="v0.14.0" description="March 14, 2023">
  ## Enhancements

  * Support cuda base image for launch runs.
  * Warn users of which packages failed to install during build process.
  * Add support for importing runs from MLFlow.
  * Mark queued runs that fail to launch as `FAILED`.

  ## Fixes

  * Temporarily remove local api key validation.
  * Launch agent gracefully removes thread when it has an exception.
  * Give clear error message when cannot connect to Docker daemon.
  * Launch support for EKS instance roles.
  * Cleaner error messages when launch encounters docker errors and graceful fail.
  * Hash docker images based on job version and dockerfile contents.
  * Security(launch): warn when agent is started polling on a team queue.
  * Add telemetry when syncing tfevents files.
  * Fix regression preventing run stopping from working.
  * Instruct user how to handle missing kubernetes import when using kubernetes runner or kaniko builder.
  * Hide unsupported launch CLI options.
  * Make launch image builder install Pytorch properly with dependencies on different hardware.

  View the [v0.14.0 changelog](https://github.com/wandb/wandb/releases/tag/v0.14.0) on GitHub.
</Update>

<Update label="v0.13.11" description="March 07, 2023">
  ## Enhancements

  * Improve launch agent logging.
  * Sweep run\_cap now works for launch sweeps.
  * Launch sweep jobs from image\_uri.
  * Add `num_workers` param to scheduler section in `launch_config`.
  * Raise ArtifactNotLoggedError instead of ValueError.
  * Launch agent uses thread pool to run jobs.
  * Make runners and builders use Environment & Registry classes.
  * Add OpenMetrics support for System Metrics.
  * Add ability to filter system metrics consumed from OpenMetrics endpoints.
  * Add support for gymnasium env monitoring, in addition to gym.
  * Add `max_scheduler` key to launch agent config.
  * Add an integration with `ultralytics` library for YOLOv8.

  ## Fixes

  * Clean up IPython's widget deprecation warning.
  * Add special Exceptions for the manager logic, when trying to connect to a gone service.
  * Fix issue where global config directory had to be writable to use Api.
  * Make error message during run initialization more actionable and fix uncaught exception.
  * Add deepcopy dunder method to the Run class.
  * Remove default to project always in sweep.
  * Error out when trying to create a launch sweep without a job specified.
  * Mkdir\_exists\_ok now (again) checks permission on existence.
  * Only log the received job when launching something sourced from a job.
  * Fix issue where queued runs sourced from images would vanish in URI.
  * Add write permissions to copied artifacts.
  * Improve `queue` argument parsing in `sweep` cli command.
  * When in disable mode don't spin up service.
  * Fix support for docker images with user specified entrypoint in local container.
  * API - ArtifactFiles no longer errors when accessing an item.
  * Verify job exists before starting the sweeps scheduler.
  * Handle system metrics requiring extra setup and teardown steps.
  * Fix a typo in `CONTRIBUTING.md`.
  * Correctly detect notebook name and fix code saving in Colab.
  * Allow up to max\_artifacts (fix off by 1 error).
  * Exercise extra caution when starting asset monitoring threads.
  * Fix bug where boto3 dependency crashes on import when downl….
  * Verify `num_workers` cli arg is valid and default to 8 if not.
  * Fix the file reference added to the verification artifact.
  * Special handling for sweeps scheduler in agent.
  * Only re-download or overwrite files when there are changes.
  * Avoid introspection in offline mode.
  * Topological ordering of `wandb.Settings`.
  * Avoid lazy loading for tensorboard patching.

  ## Cleanup

  * Fix bugbear B028 add stacklevel.
  * Move launch errors closer to the code.
  * Move mailbox error closer to the code.
  * Add unsupported error type.
  * Add support for the ruff linter.
  * Cosmetic changes for readability.
  * Introduce environment and registry abstract classes.
  * Fix unused union type in launch agent.
  * Remove the artifact from the manifest.
  * Enable typechecking for interface.artifacts and add type hints / casts.
  * Type-annotate `wandb_setup.py`.
  * Remove unused #noqa directives.
  * Disable sentry tracking when testing.

  View the [v0.13.11 changelog](https://github.com/wandb/wandb/releases/tag/v0.13.11) on GitHub.
</Update>

<Update label="v0.13.10" description="February 07, 2023">
  ## Enhancements

  * Reuse session for file upload requests.
  * Expose aliases list endpoint for artifact collections.
  * Include the username of the run's author in the environment variables.
  * Add support for local-container resource args.
  * Add the ability to append to a run with `wandb sync --append`.
  * Add an escape hatch (`disable_job_creation`) to disable automatic job creation.

  ## Fixes

  * Remove underscores from generated job name in kubernetes runner.
  * Sweep command args can once again be int type.
  * Ensure prepared artifacts have the `latest` alias.
  * Catch FileNotFoundError and PermissionError during cache.cleanup().
  * Fix order of python executable resolves.
  * Fix console handling when forking and setting stdout==stderr.
  * Fix issue where job artifacts are being logged without latest alias.
  * Ensure job names do not exceed maximum allowable for artifacts.

  ## Dev

  * Ensure manifest version is verified.
  * Add tests for custom SSL certs and disabling SSL.
  * Fix nightly docker builds.
  * Dont create universal py2/py3 package.
  * Fix flake8-bugbear B028 and ignore B017.
  * Fix gcloud sdk version requested in nightly tests.
  * Remove unused parameters in `StorageHandler.load_{path,file,reference}`.
  * Split unit tests to system tests and proper unit tests.
  * Address fixture server move from port 9010 to 9015 in local-testcontainer.
  * Add aliases to ac query response.
  * Run regression suite nightly.
  * Fix broken lightning test.
  * Enable type checking for wandb\_init.py.
  * Deprecate defaulting to default queue in launch-agent command.
  * Add unit test for kubernetes runner with annotations.
  * Fix train\_gpu\_ddp test.
  * Fix docker testimage to pull amd64 version.
  * Fix codeowners after test restructure.
  * Fix md5 test failures on Windows.
  * Split out relay server so it can be shared with yea-wandb.
  * Fix a flake8 complaint in a test.
  * Fix several import tests.
  * Don't use symlinks for SSL test assets, because Windows.
  * Add unit tests for filesync.Stats.
  * Add async retry logic.
  * Strengthen tests for ArtifactSaver, StepUpload.
  * Agent logs full stack trace when catching exception.
  * Swallow warning printed by neuron-ls.
  * Pin pip and tox in development environments.

  ## Cleanup

  * Strengthen StepUpload tests; make exception-handling more thorough in upload/commit.
  * Refactor Artifact query to fetch entity and project.
  * Replace more communicate calls with deliver.
  * Internally use Future to communicate success/failure of commit, not threading.Event.
  * Use stdlib ThreadPoolExecutor in StepUpload instead of managing our own.

  View the [v0.13.10 changelog](https://github.com/wandb/wandb/releases/tag/v0.13.10) on GitHub.
</Update>

<Update label="v0.13.9" description="January 11, 2023">
  ## Fixes

  * Exercise extra caution when checking if AWS Trainium is available in the system.
  * Restore 'util.generate\_id' for legacy / user code.
  * Replace `release` with `abandon` when releasing mailbox handle during init.

  View the [v0.13.9 changelog](https://github.com/wandb/wandb/releases/tag/v0.13.9) on GitHub.
</Update>

<Update label="v0.13.8" description="January 10, 2023">
  ## Enhancements

  * Keep uncommitted uploads in separate staging area.
  * Improve file descriptor management.
  * Default to using model-registry project for agent and launch\_add.
  * Add `exist_ok=False` to `file.download()`.
  * Auto create job artifacts from runs with required ingredients.
  * Add generalized response injection pattern for tests.
  * Replace multiprocessing.Queue's with queue.Queue's.
  * Use transaction log to cap memory usage.
  * Support system metrics for AWS Trainium.

  ## Fixes

  * Correct the type hint for wandb.run.
  * Resume collecting system metrics on object restart.
  * Fix env handling and node\_selector handling.
  * Fix Job.call() using the wrong keyword (queue vs queue\_name) when calling launch\_add.
  * Sweeps schedulers handles multi word parameters.
  * Allow spaces in requirements file, remove duplicate wandb bootstrap file.
  * Correctly handle url-encoded local file references.
  * Get digest directly instead of from the manifests' manifest.
  * Artifact.version should be the version index from the associated collection.
  * Remove duplicate generate\_id functions, replace shortuuid with secrets.
  * Fix type check for jax.Array introduced in jax==0.4.1.
  * Fix hang after failed wandb.init() (add cancel).
  * Allow users to provide path to custom executables.
  * Fix TypeError when trying to slice a Paginator object.
  * Add `AttributeError` to the list of handled exceptions when saving a keras model.
  * Remove args from jobs.

  ## Dev

  * Update t2\_fix\_error\_cond\_feature\_importances to install scikit-learn.
  * Update base Docker images for nightly testing.
  * Change sklearn to scikit-learn in functional sacred test.
  * Add error check for `--build` when resource=local-process.
  * Update scheduler and agent resource handling to allow DRC override.
  * Require sdk-team review for adding or removing high-level….
  * Remove requirement to make target project match queue.
  * Enhance nightly cloud testing process.
  * Update pull request template.
  * Return updated runSpec after pushToRunQueue query.
  * Fix for run spec handling in sdk.
  * Remove test dependency on old fastparquet package.
  * Fix dtype np.float (does not exist), set to python float.
  * Correct 'exclude' to 'ignore-paths' in .pylintrc.
  * Use pytest tmp\_path so we can inspect failures.
  * Reset build command after building.
  * Rerun flaking tests in CI with pytest-rerunfailures.
  * Remove dead code from filesync logic.
  * Remove unused fields from a filesync message.
  * Refactor retry logic to use globals instead of dependency-injecting them.
  * Add unit tests for filesync.StepUpload.
  * Add tests for Api.upload\_file\_retry.
  * Remove fallback resource when not specified for a queue.
  * Improve storage handler test coverage.
  * Fix import tests.
  * Make MetricsMonitor less verbose on errors.
  * Address fixture server move from port 9003 to 9010 in local-testcontainer.
  * Vendor promise==2.3.0 to unequivocally rm six dependency.
  * Allow setting artifact cache dir in wandb.init(...).
  * Temporary lower network buffer for testing.
  * Add telemetry if the user running in pex environment.
  * Add more flow control telemetry.
  * Add settings and debug for service startup issues (wait\_for\_ports).
  * Fix AWS Trainium test.
  * Fix status checker thread issue when user process exits without finish().
  * Add telemetry for service disabled usage.

  ## Cleanup

  * Use the same syntax whenever raising exceptions.
  * Combine \_safe\_mkdirs with mkdir\_exist\_ok.
  * Use a pytest fixture for the artifact cache.
  * Use ArtifactEntry directly instead of subclassing.
  * Consolidate hash utilities into lib.hashutil.
  * Format public file with proper formatting.
  * Install tox into proper env in dev env setup tool.
  * Clean up the init and run logic.

  View the [v0.13.8 changelog](https://github.com/wandb/wandb/releases/tag/v0.13.8) on GitHub.
</Update>

<Update label="v0.13.7" description="December 14, 2022">
  ## Fixes

  * Revert `Circular reference detected` change to resolve `Object of type Tensor is not JSON serializable`.

  View the [v0.13.7 changelog](https://github.com/wandb/wandb/releases/tag/v0.13.7) on GitHub.
</Update>

<Update label="v0.13.6" description="December 06, 2022">
  ## Enhancements

  * Add `Sweep.expected_run_count` to public Api.
  * Support volume mounts and security contexts in kubernetes runner.
  * Add a new `--build` flag for building and then pushing the image to a queue.
  * Add ability to log learning rate using WandbMetricsLogger.
  * Improve Report API in preparation for GA.

  ## Fixes

  * Add filter for `artifact_version` to only retrieve committed artifacts.
  * Deflake `wandb verify`.
  * Fix the type of the override args passed through to a LaunchProject from a Job.
  * Remove extra colon from log prefix.
  * Add support for service running in a pex based environment.
  * Fix probing static IPU info.
  * Change `artifactSequence` to `artifactCollection` in public GQL requests.
  * Fix TF compatibility issues with `WandbModelCheckpoint`.
  * Make Keras WandbCallback compatible with TF version >= 2.11.0.
  * Update gym integration to match last version.
  * Harden internal thread management in SystemMetrics.

  ## Dev

  * Improved git fetch time by specifying a `refspec` and `depth=1`.
  * Fix linguist rule to ignore grpc generated files.
  * New shard for launch tests.
  * Upgrade Node 12 based GitHub Actions.
  * Skip flaky `artifact_metadata_save` test.
  * Replace sleeps with flush when waiting on a file to write.
  * Use `tmp_path` fixture instead of writing local files during tests.
  * Fix broken queue test.
  * `skip` instead of `xfail` for test `test_artifact_metadata_save`.
  * Add many tests for InternalApi.upload\_file.
  * Add artifact Sequence fallback for older servers.
  * Make protobuf version requirements more granular.

  ## Cleanup

  * When committing artifacts, don't retry 409 Conflict errors.
  * Add programmatic alias addition/removal from SDK on artifacts.
  * Remove `wandb.sklearn.plot_decision_boundaries` that contains dead logic.
  * Adds an option to force pull the latest version of a test dev-container image.
  * Noop builder.
  * Remove unused attribute.
  * Update `mypy` to 0.991.
  * Add more robust uri parsing.
  * Turn on linting for internal\_api.py.
  * Remove dependency on six by modifying vendored libs.

  View the [v0.13.6 changelog](https://github.com/wandb/wandb/releases/tag/v0.13.6) on GitHub.
</Update>

<Update label="v0.13.5" description="November 03, 2022">
  ## Enhancements

  * Add an option to upload image references.

  ## Fixes

  * Generate more readable image names.
  * Use hash(`etag`+`url`) instead of just `etag`, as key, in artifacts cache.
  * Wait for artifact to commit before telling the user it's ready when using `wandb artifact put`.
  * Prefix vendor watchdog library.
  * Fix `Circular reference detected` error, when updating metadata with numpy array longer than 32 elements.
  * Add a random string to run\_id on SageMaker not to break DDP mode.

  ## Dev

  * Make sure we dont shutdown test cluster before grabbing results.
  * Add standalone artifact test to nightly cpu suite.
  * Rename default branch to `main`.
  * Update mypy extension for protobuf type checking.
  * Update codeql-analysis.yml branch name.
  * Move functional import tests to nightly and expand python version coverage.
  * Add Slack notification for failed nightly import tests.
  * Fix broken CLI tests that attempt uploading non-existent artifacts.

  ## Cleanup

  * Job creation through use\_artifact instead of log\_artifact.
  * Add a GH action to automate parts of the release process.
  * 3D Point Clouds now viewable in UI in all situations.
  * Git URLs were failing if fsmonitor is enabled.
  * Ignore new proto generated file directories.
  * Fix a bug preventing Run Queue deletion in the SDK.
  * Add support for `pushToRunQueueByName` mutation.
  * Refactor system metrics monitoring and probing.
  * Fix gitattribute for protobuf generated files.

  View the [v0.13.5 changelog](https://github.com/wandb/wandb/releases/tag/v0.13.5) on GitHub.
</Update>

<Update label="v0.13.4" description="October 05, 2022">
  ## Enhancements

  * Show entity and project in k8s job names.
  * Add environment variable sweep command macro.
  * Add `from_*` constructors and scene camera and bounding box confidence scores to `Object3D` data type.
  * Add simple progress indicator for artifact downloads.
  * Add `WandbMetricsLogger` callback - a `Keras` dedicated metrics logger callback.
  * Add `WandbModelCheckpoint` callback - a `Keras` model checkpointing callback.
  * Add `WandbEvalCallback` callback - a `Keras` callback for logging model predictions as W\&B tables.

  ## Fixes

  * Cast agent's config max\_jobs attribute to integer.
  * Correct the displayed path to the `debug-cli.log` (debug log).
  * Catch retry-able request timeout when uploading artifacts to AWS.
  * Improve user feedback for long running calls: summary, finish.
  * Fix RuntimeError when using `keras.WandbCallback` with `tf.MirroredStrategy`.

  ## Dev

  * Add code analysis/scanning with `codeql`.
  * Validate PR titles to ensure compliance with Conventional Commits guidelines.
  * Harden launch by pining the build versions of `kaniko` and `launch-agent-dev`.
  * Add telemetry for the `mmengine` package.
  * Add the `build` type to our conventional commits setup.
  * Add `tensorflow_datasets` requirement to `imports12` shard.
  * Fix sb3 test by pinning upstream requirement.
  * Make the SDK compatible with protobuf v4.
  * Fix flake8 output coloring.
  * Fix artifact reference test asset directory.

  ## Cleanup

  * Fix type hint for `filters` argument in `public_api.runs`.
  * Improve type annotations around artifact-file-creation.
  * Improve type annotations and VSCode config for public API.
  * Make type annotations more easily navigable in VSCode.
  * Introduce str NewTypes and use them for various Artifact fields.
  * Add type annotations to get better IDE hints for boto3 usage.

  View the [v0.13.4 changelog](https://github.com/wandb/wandb/releases/tag/v0.13.4) on GitHub.
</Update>

<Update label="v0.13.3" description="September 08, 2022">
  ## Enhancement

  * Adds `raytune` examples / tests.
  * Refactors `pytest` unit tests to run against real `wandb server`.
  * Adds Launch `kubernetes` support of taints and tolerations.
  * Adds Sweeps on Launch on Kubernetes.
  * Adds parallelism to functional testing.
  * Upgrades `mypy` to version `0.971`.
  * Adds Mailbox async internal process communication.
  * Implements searching launch job in sweep config.
  * Improves performance when sending large messages.
  * Vendors the latest `nvidia-ml-py-11.515.48`.
  * Improves performance by increase recv size on service socket.
  * Adds isort support with black profile.
  * Implements pushing test-results to CircleCI for nightly tests.
  * Adds debug mode for `pytest` unit tests.
  * Adds support for arguments in Launch Jobs.
  * Adds FetchRunQueueItemById query.
  * Adds telemetry for keras-cv.
  * Adds sentry session tracking.
  * Adds the ability to log artifact while linking to registered model.

  ### Cleanup

  * Breaks gradient and parameters hooks.
  * Adds explicit error message for double uri/docker-image.
  * Tests that the wandb\_init fixture args are in sync with wandb.init().
  * Upgrades the GKE cluster used for nightly tests to `n1-standard-8`.
  * Moves service teardown to the end of tests.
  * Reduce the `pytest` job parallelism from 10 to 6.
  * Removes service user doc.
  * Move `_timestamp` logic to the internal process.
  * Adds Launch `gitversion` error message.
  * Updates KFP machine VM image in CircleCI.
  * Upgrades sweeps to latest version.
  * Implements Sweep scheduler cleanup and better tests.
  * Adds a requirement for the sdk-team to approve API changes.
  * Adds additional time for artifact commit.
  * Implements tox configuration with dynamic resolution.
  * Removes `buildx` version pin for nightly builds.
  * Moves Launch run configs from entrypoint into params.
  * Removes Slack orb usage from Win job on CircleCI.
  * Adds heartbeat parsing for Launch run args using legacy agent.
  * Add better error handling when tearing down service.
  * Cleans up Launch job creation pipeline.
  * Adds detail to error message when uploading an artifact with the wrong type.
  * Adds optional timeout parameter to artifacts wait().
  * Sanitizes numpy generics in keys.
  * Removes reassignment of run function in public api.
  * Makes pulling sweeps optional when using public api to query for runs.
  * Updates ref docs for `wandb.init()` to give more info on special characters.

  ## Bug Fix

  * Fixes Sweeps on Launch Jobs requirement.
  * Fixes Artifact metadata JSON-encoding to accept more types.
  * Adjusts `root_dir` setting processing logic.
  * Prevents run.log() from mutating passed in arguments.
  * Fixes `05-batch5.py` test.
  * Allows users to control the `run_id` through the launch spec.
  * Fixes accidental overwrite in `config.yml`.
  * Ensures propagating overridden `base_url` when initializing public API.
  * Fixes Sweeps on Launch CLI launch config, relpath.
  * Fixes broken Launch apikey error message.
  * Marks flakey sweeps test xfail.
  * Fixes Launch `gitversion` error message.
  * Fixes `yea-wandb` dev release -> release.
  * Cleans up outstanding issues after the client->wandb rename.
  * Fixes test precision recall.
  * Fixes functional sklearn test.
  * Fixes hang caused by keyboard interrupt on windows.
  * Fixes default test container tag.
  * Fixes summary handling in conftest.py.
  * Fixes some small typos in cli output.
  * Fixes issue triggered by colab update by using default file and catching exceptions.
  * Fixes mailbox locking issue.
  * Fixes variable inclusion in log string.
  * Corrects `wandb.Artifacts.artifact.version` attribute.
  * Fixes piping of docker args by Launch Agent.
  * Fixes RecursionError when printing public API User object without email fetched.
  * Fixes deserialization of numeric column names.

  View the [v0.13.3 changelog](https://github.com/wandb/wandb/releases/tag/v0.13.3) on GitHub.
</Update>

<Update label="v0.13.2" description="August 22, 2022">
  ## Bug Fix

  * Fix issue triggered by colab update by using default file and catching exceptions.

  View the [v0.13.2 changelog](https://github.com/wandb/wandb/releases/tag/v0.13.2) on GitHub.
</Update>

<Update label="v0.13.1" description="August 05, 2022">
  ## Bug Fix

  * Prevents run.log() from mutating passed in arguments.

  View the [v0.13.1 changelog](https://github.com/wandb/wandb/releases/tag/v0.13.1) on GitHub.
</Update>

<Update label="v0.13.0" description="August 04, 2022">
  ## Enhancement

  * Turns service on by default.
  * Adds support logic for handling server provided messages.
  * Allows runs to produce jobs on finish.
  * Adds Job, QueuedRun and job handling in launch.
  * Supports in launch agent of instance roles in ec2 and eks.
  * Adds default behavior to the Keras Callback: always save model checkpoints as artifacts.
  * Sanitizes the artifact name in the KerasCallback for model artifact saving.
  * Improves console logging by moving emulator to the service process.
  * Fixes data corruption issue when logging large sizes of data.
  * Adds the state to the Sweep repr in the Public API.
  * Adds an option to specify different root dir for git using settings or environment variables.
  * Adds an option to pass `remote url` and `commit hash` as arguments to settings or as environment variables.
  * Improves time resolution for tracked metrics and for system metrics.
  * Defaults to project name from the sweep config when project is not specified in the `wandb.sweep()` call.
  * Adds support to use namespace set user by the launch agent.
  * Adds telemetry to track when a run might be overwritten.
  * Adds a tool to export `wandb`'s history into `sqlite`.
  * Replaces some `Mapping[str, ...]` types with `NamedTuples`.
  * Adds import hook for run telemetry.
  * Implements profiling support for IPUs.

  ## Bug Fix

  * Fixes sweep agent with service.
  * Fixes an empty type equals invalid type and how artifact dictionaries are handled.
  * Fixes `wandb.Config` object to support default values when getting an attribute.
  * Removes default config from jobs.
  * Fixes an issue where patch is `None`.
  * Fixes requirements.txt parsing in nightly SDK installation checks.
  * Fixes 409 Conflict handling when GraphQL requests timeout.
  * Fixes service teardown handling if user process has been terminated.
  * Adds `storage_path` and fixed `artifact.files`.
  * Fixes performance issue syncing runs with a large number of media files.

  ### Cleanup

  * Adds an escape hatch logic to disable service.
  * Annotates `wandb/docker` and reverts change in the docker fixture.
  * Fixes GFLOPS to GFLOPs in the Keras `WandbCallback`.
  * Adds type-annotate for `file_stream.py`.
  * Renames repository from `client` to `wandb`.
  * Updates documentation: adding `--report_to wandb` for HuggingFace Trainer.
  * Makes aliases optional in link\_artifact.
  * Renames `wandb local` to `wandb server`.
  * Updates README badges.

  View the [v0.13.0 changelog](https://github.com/wandb/wandb/releases/tag/v0.13.0) on GitHub.
</Update>

<Update label="v0.12.21" description="July 05, 2022">
  ## Enhancement

  * Fixes config not showing up until the run finish.
  * Adds new types to the TypeRegistry to handling artifact objects in jobs and run configs.
  * Adds new query to the internal api getting the state of the run.
  * Replaces unsafe yaml loaders with yaml.safe\_load.
  * Improves testing tooling by allowing to specify shards in manual testing.
  * Fixes ROC and PR curves in the sklearn integration by stratifying sampling.
  * Fixes input box in notebooks exceeding cell space.
  * Allows string to be passed as alias to link\_model.
  * Adds Support for FLOPS Calculation in `keras`'s `WandbCallback`.
  * Extends python report editing.

  ## Bug Fix

  * Fixes stats logger so it can find all the correct GPUs in child processes.
  * Fixes regression in s3 reference upload for folders.
  * Fixes artifact commit logic to handle collision in the backend.
  * Checks for `None` response in the retry logic (safety check).
  * Adds sweeps on top of launch (currently in MVP).
  * Renames functional tests dir and files.

  ### Cleanup

  * Fixes conditions order of `_to_dict` helper.
  * Fixes changelog broken link to PR 3709.
  * Fixes public api query (QueuedJob Api ).
  * Renames local runners to local-container and local-process.
  * Adds type annotations to files in the wandb/filesync directory.
  * Re-organizes all the testing directories to have common root dir.
  * Fixes testing configuration and add bigger machine on `CircleCi`.
  * Fixes typo in the `wandb-service-user` readme file.
  * Fixes broken artifact test for regression.
  * Removes unused files (relating to `py27`) and empty `submodules` declaration.
  * Adds extra for model reg dependency on cloudpickle.
  * Replaces deprecated threading aliases.
  * Updates the `sdk` readme to the renamed (local -> server) commands.

  View the [v0.12.21 changelog](https://github.com/wandb/wandb/releases/tag/v0.12.21) on GitHub.
</Update>

<Update label="v0.12.20" description="June 29, 2022">
  ## Bug Fix

  * Retry `commit_artifact` on conflict-error.

  View the [v0.12.20 changelog](https://github.com/wandb/wandb/releases/tag/v0.12.20) on GitHub.
</Update>

<Update label="v0.12.19" description="June 22, 2022">
  ## Bug Fix

  * Fix regression in s3 reference upload for folders.

  View the [v0.12.19 changelog](https://github.com/wandb/wandb/releases/tag/v0.12.19) on GitHub.
</Update>

<Update label="v0.12.18" description="June 09, 2022">
  ## Enhancement

  * Launch: BareRunner based on LocalRunner.
  * Add ability to specify api key to public api.
  * Add support in artifacts for files with unicode on windows.
  * Added telemetry for new packages.
  * Improve API key management.
  * Add information about `wandb server` during login.

  ## Bug Fix

  * Natively support timestamps in Python Table Types.
  * Add support for magic with service.
  * Add unit tests for DirWatcher and supporting classes.
  * Improve `DirWatcher.update_policy` O(1) instead of O(num files uploaded).
  * Add argument to control what to log in SB3 callback.
  * Improve parameter naming in sb3 integration.
  * Adjust the requirements for the dev environment setup on an M1 Mac.
  * Launch: Fix NVIDIA base image Linux keys.
  * Fix launch run queue handling from config file.
  * Fix issue where tfevents were not always consumed.
  * \[Snyk] Fix for 8 vulnerabilities.
  * Fix s3 storage handler to upload folders when key names collide.
  * Correctly load timestamps from tables in artifacts.
  * Require `protobuf&lt;44`.
  * Make Containers created through launch re-runnable as container jobs.
  * Fix tensorboard integration skipping steps at finish().
  * Rename `wandb local` to `wandb server`.
  * Fix busted docker inspect command.
  * Add dedicated sentry wandb.
  * Image Type should gracefully handle older type params.

  ### Cleanup

  * Inline FileEventHandler.synced into the only method where it's used.
  * Use passed size argument to make `PolicyLive.min_wait_for_size` a classmethod.
  * Make FileEventHandler an ABC, remove some "default" method impls which were only used once.
  * Remove unused field from DirWatcher.
  * Make sweeps an extra instead of vendoring.
  * Add nightly CI testing.
  * Improve keras and data type Reference Docs.
  * Update `pytorch` version requirements in dev environments.
  * Clean up CircleCI config.
  * Add `py310` testing in CI.
  * Ditch `dateutil` from the requirements.
  * Add deprecated string to `Table.add_row`.

  View the [v0.12.18 changelog](https://github.com/wandb/wandb/releases/tag/v0.12.18) on GitHub.
</Update>

<Update label="v0.12.17" description="May 26, 2022">
  ## Bug Fix

  * Update requirements to fix incompatibility with protobuf >= 4.

  View the [v0.12.17 changelog](https://github.com/wandb/wandb/releases/tag/v0.12.17) on GitHub.
</Update>

<Update label="v0.12.16" description="May 03, 2022">
  ## Enhancement

  * Improve W\&B footer by aligning summary/history in notebook env.
  * Enable experimental history step logging in artifacts.
  * Add `args_no_boolean_flags` macro to sweep configuration.
  * Add logging support for `jax.bfloat.bfloat16`.
  * Raise exception when Table size exceeds limit.
  * Add kaniko k8s builder for wandb launch.
  * Add wandb.init() timeout setting.
  * Do not assume executable for given entrypoints with wandb launch.
  * Jupyter environments no longer collect command arguments.
  * Add support for TensorFlow/Keras SavedModel format.

  ## Bug Fix

  * Support version IDs in artifact refs, fix s3/gcs references in Windows.
  * Fix support for multiple finish for single run using wandb-service.
  * Fix duplicate backtrace when using wandb-service.
  * Fix wrong entity displayed in login message.
  * Fix hang when `wandb.init()` is interrupted mid setup using wandb-service.
  * Fix handling keyboard interrupt to avoid hangs with wandb-service enabled.
  * Fix console logging with very long print out when using wandb-service.
  * Fix broken artifact string in launch init config.

  ### Cleanup

  * Fix typo in wandb.log() docstring.
  * Cleanup custom chart code and add type annotations to plot functions.
  * Improve `wandb.init(settings=)` to handle `Settings` object similarly to `dict` parameter.
  * Add documentation note about api.viewer in api.user() and api.users().
  * Be explicit about us being py3+ only in setup.py.
  * Add type annotations to DirWatcher.
  * Improve wandb.log() docstring to use the correct argument name.

  View the [v0.12.16 changelog](https://github.com/wandb/wandb/releases/tag/v0.12.16) on GitHub.
</Update>

<Update label="v0.12.15" description="April 21, 2022">
  ## Enhancement

  * Optimize wandb.Image logging when linked to an artifact.

  View the [v0.12.15 changelog](https://github.com/wandb/wandb/releases/tag/v0.12.15) on GitHub.
</Update>

<Update label="v0.12.14" description="April 08, 2022">
  ## Bug Fix

  * Fix regression: disable saving history step in artifacts.

  View the [v0.12.14 changelog](https://github.com/wandb/wandb/releases/tag/v0.12.14) on GitHub.
</Update>

<Update label="v0.12.13" description="April 07, 2022">
  ## Bug Fix

  * Revert strictened api\_key validation.

  View the [v0.12.13 changelog](https://github.com/wandb/wandb/releases/tag/v0.12.13) on GitHub.
</Update>

<Update label="v0.12.12" description="April 05, 2022">
  ## Enhancement

  * Allow run objects to be passed to other processes when using wandb-service.
  * Add create user to public api.
  * Support logging from multiple processes with wandb-service.
  * Add gpus flag for local launch runner with cuda.
  * Improve Launch deployable agent.
  * Add Launch kubernetes integration.
  * KFP: Add wandb visualization helper.
  * KFP: Link back to Kubeflow UI.
  * Add boolean flag arg macro.

  ## Bug Fix

  * Improve host / WANDB\_BASE\_URL validation.
  * Fix/insecure tempfile.
  * Fix excess warning span if requested WANDB\_DIR/root\_dir is not writable.
  * Fix line\_series to plot array of strings.
  * Properly handle command line args with service.
  * Improve api\_key validation.
  * Fix multiple performance issues caused by not using defaultdict.
  * Enable inf max jobs on launch agent.
  * Fix colab command to work with launch.
  * Fix typo in Config docstring.
  * Make code saving not a policy, keep previous custom logic.
  * Fix logging sequence images with service.
  * Add username to debug-cli log file to prevent conflicts of multiple users.
  * Fix python sweep agent for users of wandb service / pytorch-lightning.
  * Remove unnecessary launch reqs checks.
  * Workaround for MoviePy's Unclosed Writer.
  * Improve handling of Run objects when service is not enabled.

  View the [v0.12.12 changelog](https://github.com/wandb/wandb/releases/tag/v0.12.12) on GitHub.
</Update>

<Update label="v0.12.11" description="March 01, 2022">
  ## Enhancement

  * Add captions to Molecules.
  * Add CatBoost Integration.
  * Launch: AWS Sagemaker integration.
  * Launch: Remove repo2docker and add gpu support.
  * Adds Timestamp inference from Python for Weave.
  * Launch GCP vertex integration.
  * Use Artifacts when put into run config. Accept a string to represent an artifact in the run config.
  * Improve xgboost `wandb_callback` (#2929).
  * Add initial kubeflow pipeline support.

  ## Bug Fix

  * Fix logging of images with special characters in the key.
  * Fix azure blob upload retry logic.
  * Fix program field for scripts run as a python module.
  * Fix issue where `sync_tensorboard` could die on large histograms.
  * Fix wandb service performance issue during run shutdown.
  * Fix vendoring of gql and graphql.
  * Flush log data without finish with service.
  * Fix wandb service hang when the service crashes.
  * Fix issue logging images with "/" on Windows.
  * Add image filenames to images/separated media.
  * Add setproctitle to requirements.txt.
  * Fix issue where sagemaker run ids break run queues.
  * Fix encoding exception when using %%capture magic.

  View the [v0.12.11 changelog](https://github.com/wandb/wandb/releases/tag/v0.12.11) on GitHub.
</Update>

<Update label="v0.12.10" description="February 01, 2022">
  ## Enhancement

  * Improve validation when creating Tables with invalid columns from dataframes.
  * Enable digest deduplication for `use_artifact()` calls.
  * Initial prototype of azure blob upload support.

  ## Bug Fix

  * Fix wandb launch using python dev versions.
  * Fix loading table saved with mixed types.
  * Fix ResourceWarning when calling wandb.log().
  * Fix missing cursor in ProjectArtifactCollections.
  * Fix windows table logging classes issue.
  * Gracefully handle string labels in wandb.sklearn.plot.classifier.calibration\_curve.
  * Do not display login warning when calling wandb.sweep().

  ### Cleanup

  * Drop python2 backport deps (enum34, subprocess32, configparser).
  * Settings refactor.

  View the [v0.12.10 changelog](https://github.com/wandb/wandb/releases/tag/v0.12.10) on GitHub.
</Update>

<Update label="v0.12.9" description="December 16, 2021">
  ## Bug Fix

  * Fix regression in `upload_file()` exception handler.

  View the [v0.12.9 changelog](https://github.com/wandb/wandb/releases/tag/v0.12.9) on GitHub.
</Update>

<Update label="v0.12.8" description="December 16, 2021">
  ## Enhancement

  * Update contributing guide and dev env setup tool.
  * Improve `wandb_callback` for LightGBM (#2945).

  ## Bug Fix

  * Reduce GPU memory usage when generating histogram of model weights.
  * Support mixed classes in bounding box and image mask annotation layers.
  * Add max-jobs and launch async args.
  * Support lists of Summary objects encoded as strings to wandb.tensorboard.log.
  * Fix handling of 0 dim np arrays.
  * Fix handling of empty default config file.
  * Add service backend using sockets (support fork).
  * Send git port along with url when sending git repo.
  * Add support raw ip addresses for launch.
  * Tables no longer serialize and hide 1d NDArrays.
  * Fix artifact file uploads to S3 stores.
  * Send uploaded file list on file stream heartbeats.
  * Add support for keras experimental layers.
  * Fix `from wandb import magic` to not require tensorflow.
  * Fix launch permission error.

  View the [v0.12.8 changelog](https://github.com/wandb/wandb/releases/tag/v0.12.8) on GitHub.
</Update>

<Update label="v0.12.7" description="November 18, 2021">
  ## Bug Fix

  * Fix issue where console log streaming was causing excessive network traffic.
  * Metaflow: Make optional dependencies actually optional.
  * Fix docstrings for wandb.watch() and ValidationDataLogger.
  * Prevent launch agent from sending runs to a different project or entity.
  * Fix logging pr\_curves through tensorboard.
  * Prevent TPU monitoring from reporting invalid metrics when not available.
  * Make import order dependencies for WandbCallback more robust.
  * Fix a bug in feature importance plotting to handle matrices of different shapes.
  * Fix base url handling to allow trailing /.
  * Prevent wandb.agent() from sending too many heartbeats impacting rate limits.
  * Redact sensitive information from debug logs.

  ## Enhancement

  * Add wandb.Molecule support for rdkit supported formats.
  * Add module-level docstrings for reference doc modules.
  * Store launch metadata in file.
  * Add Project.sweeps() public API call to view all sweeps in a project.
  * Ensures API key prompt remains captive when user enters nothing.
  * Refactors wandb.sklearn into submodules.
  * Support code artifacts in wandb launch.
  * Improve launch agent (async, stop, heartbeat updates).
  * Improve usage and error messages for anonymous mode.
  * Add example on how to find runs with wandb.Api().runs(...) matching a regex.

  View the [v0.12.7 changelog](https://github.com/wandb/wandb/releases/tag/v0.12.7) on GitHub.
</Update>

<Update label="v0.12.6" description="October 27, 2021">
  ## Bug Fix

  * Fix sklearn `plot_calibration_curve()` issue breaking the provided model.
  * Fix CondaEnvExportError by redirecting stderr.
  * Fix `use_artifact()` when specifying an artifact from a different project.

  ## Enhancement

  * Add metric names to pr curve charts in tensorboard.

  View the [v0.12.6 changelog](https://github.com/wandb/wandb/releases/tag/v0.12.6) on GitHub.
</Update>

<Update label="v0.12.5" description="October 19, 2021">
  ## Bug Fix

  * Report errors for invalid characters in logged media keys on windows.
  * Handle errors when probing for TPUs in unsupported environments.
  * Fixed bug where `%%wandb` followed by wandb.init() does not display run links.
  * Fixed api.runs() to correctly return all runs for the current entity/project.

  ## Enhancement

  * Add `wandb.require(experiment="service")` to improve multiprocessing support.
  * Add support for swappable artifacts in launch context.
  * Add `wandb.login(timeout=)` support for jupyter environments.
  * Add ability to disable git ref saving with `WANDB_DISABLE_GIT`.
  * Support newer versions of pytest-mock and PyYAML.
  * Add ability to delete artifacts with aliases: `artifact.delete(delete_aliases=True)`.
  * Add `unwatch()` method to the Run object.

  View the [v0.12.5 changelog](https://github.com/wandb/wandb/releases/tag/v0.12.5) on GitHub.
</Update>

<Update label="v0.12.4" description="October 05, 2021">
  ## Bug Fix

  * Fix regression introduced in 0.12.2 causing network access when `WANDB_MODE=offline`.

  View the [v0.12.4 changelog](https://github.com/wandb/wandb/releases/tag/v0.12.4) on GitHub.
</Update>

<Update label="v0.12.3" description="September 30, 2021">
  ## Bug Fix

  * Fixes the grid search stopping condition in the local controller.

  ## Enhancement

  * New jupyter magic for displaying runs, sweeps, and projects `%wandb path/to/run -h 1024`.
  * We no longer display run iframe by default in jupyter, add `%%wandb` to a cell to display a run.
  * Makes api key prompting retry indefinitely on malformed input.
  * Invite users to teams via the api `api.team("team_name").invite("username_or_email")`.
  * Remove users from a team via the api `api.team("team_name").members[0].delete()`.
  * Create service accounts via the api `api.team("team_name").create_service_account("Description")`.
  * Manage api keys via the api `api.user("username_or_email").generate_api_key()`.
  * Add pytorch profiling trace support with `wandb.profiler.torch_trace_handler()`.

  View the [v0.12.3 changelog](https://github.com/wandb/wandb/releases/tag/v0.12.3) on GitHub.
</Update>

<Update label="v0.12.2" description="September 15, 2021">
  ## Bug Fix

  * Fix tensorboard\_sync to handle ephemeral Sagemaker tfevents files.
  * Fix Reports query from the public api (broken pagination and report path).
  * Fix `wandb.login()` when relogin is specified (only force login once).

  ## Enhancement

  * Clean up footer output of summary and history metrics.
  * Clean up error message from `wandb sweep --update`.
  * Add warning for `wandb local` users to update their docker.
  * Add optional argument log\_learning\_curve to wandb.sklearn.plot\_classifier().
  * Restore frozen pip package versions when using `wandb launch`.
  * Add support for jupyter notebooks in launch.
  * Add `wandb.login()` timeout option.

  View the [v0.12.2 changelog](https://github.com/wandb/wandb/releases/tag/v0.12.2) on GitHub.
</Update>

<Update label="v0.12.1" description="August 26, 2021">
  ## Bug Fix

  * Fix tensorflow/keras 2.6 not logging validation examples.
  * Fix metrics logged through tensorboard not supporting time on x-axis.
  * Fix `WANDB_IGNORE_GLOBS` environment variable handling.
  * Fix handling when sys.stdout is configured to a custom logger.
  * Fix sklearn feature importance plots not matching feature names properly.
  * Fix an issue where colab urls were not being captured.
  * Save program commandline if run executable was outside cwd.

  ## Enhancement

  * Add Prodigy integration to upload annotated datasets to W\&B Tables.
  * Add initial Metaflow support.
  * Add experimental wandb launch support.
  * Add warnings that public API requests are timing out and allow override.
  * Improve error handling in local controller sweeps engine.

  View the [v0.12.1 changelog](https://github.com/wandb/wandb/releases/tag/v0.12.1) on GitHub.
</Update>

<Update label="v0.12.0" description="August 10, 2021">
  ### No Longer Supported

  * Remove Python 3.5 support.

  ## Bug Fix

  * Fix issue that could cause artifact uploads to fail if artifact files are being modified.
  * Fix issue where `wandb.restore()` wouldn't work with runs from a sweep.

  ## Enhancement

  * Improve run execution time calculation.

  View the [v0.12.0 changelog](https://github.com/wandb/wandb/releases/tag/v0.12.0) on GitHub.
</Update>

<Update label="v0.11.2" description="August 02, 2021">
  ## Bug Fix

  * Restore vendored graphql-core library because of network regression.

  View the [v0.11.2 changelog](https://github.com/wandb/wandb/releases/tag/v0.11.2) on GitHub.
</Update>

<Update label="v0.11.1" description="July 29, 2021">
  ## Deprecated

  * Python 3.5 will not be supported as of `wandb==0.12.0`.

  ## Bug Fix

  * Reduce Memory Footprint of Images In Tables.
  * Added a dependency on graphql-core>=2.3.0.
  * Removed urllib3 pin to avoid conflicts, if you see urllib3 related errors run `pip install --upgrade urllib3`.
  * Improved Public API HTTP error messages.
  * Set run.dir to the generated directory name in disabled mode.

  ## Enhancement

  * Adds support for native Jax array logging.
  * Tables now support Molecule data type.
  * Improve Stable-Baselines3 API by auto log model's name and always upload models at the end of training.
  * Implements the sweep local controller using wandb/sweeps.

  View the [v0.11.1 changelog](https://github.com/wandb/wandb/releases/tag/v0.11.1) on GitHub.
</Update>

<Update label="v0.11.0" description="July 15, 2021">
  ### No Longer Supported

  * Remove Python 2.7 support.

  ## Bug Fix

  * Fix issue where `wandb.watch()` broke model saving in pytorch.
  * Fix issue where uniform sweep parameters were parsed as int\_uniform.
  * Fix issue where file\_stream thread was killed on 4xx errors.

  ## Enhancement

  * Improve performance of artifact logging by making it non-blocking.
  * Add wandb integration for Stable-Baselines3.
  * Improve keras callback validation logging inference logic.
  * Expose sweep state via the public API.
  * Improve performance of sweep run fetches via the API.

  View the [v0.11.0 changelog](https://github.com/wandb/wandb/releases/tag/v0.11.0) on GitHub.
</Update>

<Update label="v0.10.33" description="June 28, 2021">
  ## Bug Fix

  * Fix issue where wandb restore 404ed if the run did not have a diff.patch file.
  * Fix issue where wandb.log() raised an Exception after trying to log a pandas dataframe.
  * Fix issue where runs could be marked finished before files were finished uploading.

  ## Enhancement

  * Disable reloading of run metadata (such as command) in resumed runs.
  * Allow logging of pandas dataframes by automatically converting them to W\&B tables.
  * Fix up `log_code()` exclude fn to handle .wandb dir.
  * Improve handling of PyTorch model topology.
  * Increase config debounce interval to 30s to reduce load on WB/backend.
  * Improve reliability of CLI in generating sweeps with names, programs, and settings.

  View the [v0.10.33 changelog](https://github.com/wandb/wandb/releases/tag/v0.10.33) on GitHub.
</Update>

<Update label="v0.10.32" description="June 10, 2021">
  ## Bug Fix

  * Make `log_artifact()` more resilient to network errors.
  * Removed Duplicate Artifact Dependencies.
  * Workaround urlib3 issue on windows.
  * Fix regression where ipython was hanging.
  * Allow logging of numpy high precision floating point values.
  * Reduce liklyhood of collisions for file backed media or artifact objects.
  * Fix wandb.watch() regression when logging pytorch graphs.

  ## Enhancement

  * Add support for logging joined and partitioned table.
  * Handle schema validation warnings for sweep configs.
  * Improve wandb sync to handle errors.
  * Add ability to label scripts and repositories who use wandb.

  View the [v0.10.32 changelog](https://github.com/wandb/wandb/releases/tag/v0.10.32) on GitHub.
</Update>

<Update label="v0.10.31" description="May 27, 2021">
  ## Bug Fix

  * Wandb.login() did not properly persist the host parameter.
  * Fix issue where step information was not synced properly when syncing tensorboard directories.
  * Fix some unicode issues with python2.7.
  * Fixed bug in `plot_calibration_curve` for ComplementNB.
  * Fall back to not using SendFile on some linux systems.
  * Fix console issues where lines were truncated.
  * Fix console issues where console logging could block.

  ## Enhancement

  * Add support for preemptible sweeps.
  * Add command line for sweep control.
  * Add support to load artifact collection properties.

  View the [v0.10.31 changelog](https://github.com/wandb/wandb/releases/tag/v0.10.31) on GitHub.
</Update>

<Update label="v0.10.30" description="May 07, 2021">
  ## Bug Fix

  * Found and fixed the remaining issues causing runs to be marked crashed during outages.
  * Improved performance for users of `define_metric`, pytorch-lightning, and aggressive config saving.
  * Fix issue when trying to log a cuda tensor to config or summary.
  * Remove dependency on torch `backward_hooks` to compute graph.
  * Fix an issue preventing the ability to resume runs on sagemaker.
  * Fix issues preventing pdb from working reliably with wandb.
  * Fix deprecation warning in vendored library (user submission).
  * Fix logging behavior where the library was accidently outputting logs to the console.
  * Fix disabled mode to not create wandb dir and log files.
  * Renamed types to prep for Tables launch.

  ## Enhancement

  * Allow renaming groups with public api.

  View the [v0.10.30 changelog](https://github.com/wandb/wandb/releases/tag/v0.10.30) on GitHub.
</Update>

<Update label="v0.10.29" description="May 03, 2021">
  ## Bug Fix

  * Fix more network handling issues causing runs to be marked crashed (wandb sync to recover).
  * Improve logging and exception handling to improve reporting and logging of crashed processes.

  View the [v0.10.29 changelog](https://github.com/wandb/wandb/releases/tag/v0.10.29) on GitHub.
</Update>

<Update label="v0.10.28" description="April 28, 2021">
  ## Bug Fix

  * Fix network handling issue causing runs to be marked crashed (wandb sync to recover).
  * Use `register_full_backward_hook` to support models with Dict outputs.
  * Allow periods in table columns.
  * Fix artifact cache collisions when using forked processes.
  * Fix issue where custom charts do not display properly with pytorch-lightning.

  ## Enhancement

  * Add experimental incremental artifact support.
  * Improve warnings when logging is being rate limited.

  View the [v0.10.28 changelog](https://github.com/wandb/wandb/releases/tag/v0.10.28) on GitHub.
</Update>

<Update label="v0.10.27" description="April 19, 2021">
  ## Bug Fix

  * Fix tensorboard\_sync condition where metrics at end of short run are dropped.
  * Fix `wandb sync` when tensorboard files are detected.
  * Fix api key prompt in databricks notebook.

  ## Enhancement

  * Integrate DSViz into Keras WandbCallback.
  * Add support for conda dependencies (user submit).

  View the [v0.10.27 changelog](https://github.com/wandb/wandb/releases/tag/v0.10.27) on GitHub.
</Update>

<Update label="v0.10.26" description="April 13, 2021">
  ## Bug Fix

  * Fix network handling issue where syncing stopped (use wandb sync to recover).
  * Fix auth problem when using sagemaker and hugginface integrations together.
  * Fix handling of NaN values in tables with non floats.
  * Lazy load API object to prevent unnecessary file access on module load.

  ## Enhancement

  * Improve error messages when using public api history accessors.

  View the [v0.10.26 changelog](https://github.com/wandb/wandb/releases/tag/v0.10.26) on GitHub.
</Update>

<Update label="v0.10.25" description="April 05, 2021">
  ## Bug Fix

  * Fix possible artifact cache race when using parallel artifact reads.
  * Fix artifact reference when `checksum=False`.

  ## Enhancement

  * Release `run.define_metric()` to simplify custom x-axis and more.
  * Add column operators `add_column`, `get_column`, `get_index` to `wandb.Table()`.

  View the [v0.10.25 changelog](https://github.com/wandb/wandb/releases/tag/v0.10.25) on GitHub.
</Update>

<Update label="v0.10.24" description="March 30, 2021">
  ## Bug Fix

  * Significant fixes to stdout/stderr console logging.
  * Prevent excessive network when saving files with policy=`live`.
  * Fix errors when trying to send large updates (most common with `wandb sync`).

  ## Enhancement

  * Automatically generate `run_table` artifact for logged tables.
  * Add bracket notation to artifacts.
  * Improve URL validation when specifying server url to `wandb login`.

  View the [v0.10.24 changelog](https://github.com/wandb/wandb/releases/tag/v0.10.24) on GitHub.
</Update>

<Update label="v0.10.23" description="March 22, 2021">
  ## Bug Fix

  * Fix logged artifacts to be accessible after wait().
  * Fix spell.run integration.
  * Performance fix syncing console logs with carriage returns.
  * Fix confusion matrix with class names and unlabeled data.

  ## Enhancement

  * Add the ability to save artifacts without creating a run.
  * Add Foreign Table References to wandb.Table.
  * Allow the same runtime object to be logged to multiple artifacts.
  * Add experimental `run._define_metric()` support.
  * Warn and ignore unsupported multiprocess `wandb.log()` calls.

  View the [v0.10.23 changelog](https://github.com/wandb/wandb/releases/tag/v0.10.23) on GitHub.
</Update>

<Update label="v0.10.22" description="March 09, 2021">
  ## Bug Fix

  * Fix system metric logging rate in 0.10.x.
  * Fix Audio external reference issue.
  * Fix short runs with tensorboard\_sync.
  * Ignore `wandb.init(id=)` when running a sweep.
  * Sanitize artifact metadata if needed.

  ## Enhancement

  * Allow syncing of tfevents with `wandb sync --sync-tensorboard`.

  View the [v0.10.22 changelog](https://github.com/wandb/wandb/releases/tag/v0.10.22) on GitHub.
</Update>

<Update label="v0.10.21" description="March 02, 2021">
  ## Bug Fix

  * Fix artifact.get() regression since 0.10.18.
  * Allow 0 byte artifacts.
  * Fix codesaving and program name reporting.

  ## Enhancement

  * Added support for glb files for `wandb.Object3D()`.
  * Added support for external references for `wandb.Audio()`.
  * Custom chart support tensorboard `pr_curves` plugin.
  * Support saving entire code directory in an artifact.

  View the [v0.10.21 changelog](https://github.com/wandb/wandb/releases/tag/v0.10.21) on GitHub.
</Update>

<Update label="v0.10.20" description="February 22, 2021">
  ## Bug Fix

  * Wandb.login() now respects disabled mode.
  * Handle exception when trying to log TPUs in colab.

  ## Enhancement

  * Add `WANDB_START_METHOD=thread` to support non-multiprocessing.
  * Add `group` and `job_type` to Run object in the export API.
  * Improve artifact docstrings.

  View the [v0.10.20 changelog](https://github.com/wandb/wandb/releases/tag/v0.10.20) on GitHub.
</Update>

<Update label="v0.10.19" description="February 14, 2021">
  ## Bug Fix

  * Fix artifact manifest files incorrectly named with patch suffix.

  View the [v0.10.19 changelog](https://github.com/wandb/wandb/releases/tag/v0.10.19) on GitHub.
</Update>

<Update label="v0.10.18" description="February 08, 2021">
  ## Enhancement

  * Add run delete and file delete to the public API.
  * Align steps between `tensorboard_sync` and wandb.log() history.
  * Add `WANDB_START_METHOD` to allow POSIX systems to use fork.
  * Support mixed types in wandb.Table() with `allow_mixed_types`.

  ## Bug Fix

  * Fix potential leaked file due to log not being closed properly.
  * Improve `wandb verify` to better handle network issues and report errors.
  * Made file downloads more deterministic with respect to filesystem caches.

  View the [v0.10.18 changelog](https://github.com/wandb/wandb/releases/tag/v0.10.18) on GitHub.
</Update>

<Update label="v0.10.17" description="February 01, 2021">
  ## Bug Fix

  * Fix regression seen with python 3.5.
  * Silence vendored watchdog warnings on mac.

  View the [v0.10.17 changelog](https://github.com/wandb/wandb/releases/tag/v0.10.17) on GitHub.
</Update>

<Update label="v0.10.16" description="February 01, 2021">
  ## Enhancement

  * Artifacts now support parallel writers for large distributed workflows.
  * Artifacts support distributed tables for dataset visualization.
  * Improvements to PR templates.
  * Added more type annotations.
  * Vendored watchdog 0.9.0 removing it as a dependency.
  * New documentation generator.
  * Public api now has `file.direct_url` to avoid redirects for signed urls.

  ## Bug Fix

  * Allow `config-defaults.yaml` to be overwritten when running sweeps.
  * General bug fixes and improvements to `wandb verify`.
  * Disabled widgets in Spyder IDE.
  * Fixed WANDB\_SILENT in Spyder IDE.
  * Reference file:// artifacts respect the `name` attribute.

  View the [v0.10.16 changelog](https://github.com/wandb/wandb/releases/tag/v0.10.16) on GitHub.
</Update>

<Update label="v0.10.15" description="January 24, 2021">
  ## Enhancement

  * Add `wandb verify` to troubleshoot local installs.

  ## Bug Fix

  * Fix tensorboard\_sync issue writing to s3.
  * Prevent git secrets from being stored.
  * Disable verbose console messages when using moviepy.
  * Fix artifacts with checkpoints to be more robust when overwriting files.
  * Fix artifacts recycled id issue.

  View the [v0.10.15 changelog](https://github.com/wandb/wandb/releases/tag/v0.10.15) on GitHub.
</Update>

<Update label="v0.10.14" description="January 15, 2021">
  ## Enhancement

  * Add wandb.Audio support to Artifacts.

  ## Bug Fix

  * Fix wandb config regressions introduced in 0.10.13.
  * Rollback changes supporting media with slashes in keys.

  View the [v0.10.14 changelog](https://github.com/wandb/wandb/releases/tag/v0.10.14) on GitHub.
</Update>

<Update label="v0.10.13" description="January 11, 2021">
  ## Enhancement

  * Add support for Mac M1 GPU monitoring.
  * Add support for TPU monitoring.
  * Add setting to disable sagemaker integration.

  ## Bug Fix

  * Fix tensorboard\_sync with tensorboardX and tf1.
  * Fix issues logging images with slashes.
  * Fix custom charts issues.
  * Improve error messages using `wandb pull`.
  * Improve error messages with `wandb.Table()`.
  * Make sure silent mode is silent.
  * Fix `wandb online` to renable logging.
  * Multiple artifact fixes.

  View the [v0.10.13 changelog](https://github.com/wandb/wandb/releases/tag/v0.10.13) on GitHub.
</Update>

<Update label="v0.10.12" description="December 03, 2020">
  ## Enhancement

  * Add Artifact.used\_by and Artifact.logged\_by.
  * Validate type consistency when logging Artifacts.
  * Enhance JoinedTable to not require downloaded assets.
  * Add ability to recursively download dependent artifacts.
  * Enable gradient logging with keras and tf2+.
  * Validate pytorch models are passed to wandb.watch().
  * Improved docstrings for public methods / objects.
  * Warn when image sequences are logged with different sizes.

  ## Bug Fix

  * Fix incorrectly generated filenames in summary.
  * Fix anonymous mode to include the api key in URLs.
  * Fix pickle issue with disabled mode.
  * Fix artifact from\_id query.
  * Fix handling of Tables with different image paths.

  View the [v0.10.12 changelog](https://github.com/wandb/wandb/releases/tag/v0.10.12) on GitHub.
</Update>

<Update label="v0.10.11" description="November 18, 2020">
  ## Enhancement

  * Disable wandb logging with `wandb disabled` or `wandb.init(mode="disabled")`.
  * Support cloning an artifact when logging wandb.Image().

  ## Bug Fix

  * Multiple media artifact improvements and internal refactor.
  * Improve handling of artifact errors.
  * Fix issue where notebook name was ignored.
  * Extend silent mode for jupyter logging.
  * Fix issue where vendored libraries interfered with python path.
  * Fix various exceptions (divide by zero, int conversion, TypeError).

  View the [v0.10.11 changelog](https://github.com/wandb/wandb/releases/tag/v0.10.11) on GitHub.
</Update>

<Update label="v0.10.10" description="November 09, 2020">
  ## Enhancement

  * Added confusion matrix plot.
  * Better jupyter messages with wandb.init()/reinit/finish.

  ## Bug Fix

  * Fix for fastai 2.1.5 (removed log\_args).
  * Fixed media logging when directories are changed.

  View the [v0.10.10 changelog](https://github.com/wandb/wandb/releases/tag/v0.10.10) on GitHub.
</Update>

<Update label="v0.10.9" description="November 04, 2020">
  ## Enhancement

  * Added artifact media logging (alpha).
  * Add scriptable alerts.
  * Add url attribute for sweep public api.
  * Update docstrings for wandb sdk functions.

  ## Bug Fix

  * Fix cases where offline mode was making network connections.
  * Fix issues with python sweeps and run stopping.
  * Fix logging issue where we could accidently display an api key.
  * Fix wandb login issues with malformed hosts.
  * Allow wandb.restore() to be called without wandb.init().
  * Fix resuming (reusing run\_id) with empty summary.
  * Fix artitifact download issue.
  * Add missing wandb.unwatch() function.
  * Avoid creating spurious wandb directories.
  * Fix collections import issue when using an old version of six.

  View the [v0.10.9 changelog](https://github.com/wandb/wandb/releases/tag/v0.10.9) on GitHub.
</Update>

<Update label="v0.10.8" description="October 22, 2020">
  ## Enhancement

  * Allow callables to be serialized.

  ## Bug Fix

  * Fix compatibility issue with python 3.9.
  * Fix `wandb sync` failure introduced in 0.10.6.
  * Improve python agent handling of failing runs.
  * Fix rare condition where resuming runs does not work.
  * Improve symlink handling when called in thread context.
  * Fix issues when changing directories before calling wandb.init().

  View the [v0.10.8 changelog](https://github.com/wandb/wandb/releases/tag/v0.10.8) on GitHub.
</Update>

<Update label="v0.10.7" description="October 15, 2020">
  ## Bug Fix

  * Fix issue when checking for updated releases on pypi.

  View the [v0.10.7 changelog](https://github.com/wandb/wandb/releases/tag/v0.10.7) on GitHub.
</Update>

<Update label="v0.10.6" description="October 15, 2020">
  ## Bug Fix

  * Make sure code saving is enabled in jupyter environments after login.
  * Sweep agents have extended timeout for large sweep configs.
  * Support WANDB\_SILENT environment variable.
  * Warn about missing python package when logging images.
  * Fix wandb.restore() to apply diff patch.
  * Improve artifact error messages.
  * Fix loading of config-defaults.yaml and specified list of yaml config files.

  View the [v0.10.6 changelog](https://github.com/wandb/wandb/releases/tag/v0.10.6) on GitHub.
</Update>

<Update label="v0.10.5" description="October 07, 2020">
  ## Enhancement

  * Add new custom plots: `wandb.plot.*`.
  * Add new python based sweep agent: `wandb.agent()`.

  ## Bug Fix

  * Console log fixes (tqdm on windows, fix close exceptions).
  * Add more attributes to the Run object (group, job\_type, urls).
  * Fix sagemaker login issues.
  * Fix issue where plots were not uploaded until the end of run.

  View the [v0.10.5 changelog](https://github.com/wandb/wandb/releases/tag/v0.10.5) on GitHub.
</Update>

<Update label="v0.10.4" description="September 29, 2020">
  ## Bug Fix

  * Fix an issue where wandb.init(allow\_val\_change=) throws exception.

  View the [v0.10.4 changelog](https://github.com/wandb/wandb/releases/tag/v0.10.4) on GitHub.
</Update>

<Update label="v0.10.3" description="September 29, 2020">
  ## Enhancement

  * Added warning when trying to sync pre 0.10.0 run dirs.
  * Improved jupyter support for wandb run syncing information.

  ## Bug Fix

  * Fix artifact download issues.
  * Fix multiple issues with tensorboard\_sync.
  * Fix multiple issues with juypter/python sweeps.
  * Fix issue where login was timing out.
  * Fix issue where config was overwritten when resuming runs.
  * Ported sacred observer to 0.10.x release.
  * Fix predicted bounding boxes overwritten by ground truth boxes.
  * Add missing save\_code parameter to wandb.init().

  View the [v0.10.3 changelog](https://github.com/wandb/wandb/releases/tag/v0.10.3) on GitHub.
</Update>

<Update label="v0.10.2" description="September 20, 2020">
  ## Enhancement

  * Added upload\_file to API.
  * Wandb.finish() can be called without matching wandb.init().

  ## Bug Fix

  * Fix issue where files were being logged to wrong parallel runs.
  * Fix missing properties/methods -- as\_dict(), sweep\_id.
  * Fix wandb.summary.update() not updating all keys.
  * Code saving was not properly enabled based on UI settings.
  * Tensorboard now logging images before end of program.
  * Fix resume issues dealing with config and summary metrics.

  View the [v0.10.2 changelog](https://github.com/wandb/wandb/releases/tag/v0.10.2) on GitHub.
</Update>

<Update label="v0.10.1" description="September 16, 2020">
  ## Enhancement

  * Added sync\_tensorboard ability to handle S3 and GCS files.
  * Added ability to specify host with login.
  * Improved artifact API to allow modifying attributes.

  ## Bug Fix

  * Fix codesaving to respect the server settings.
  * Fix issue running wandb.init() on restricted networks.
  * Fix issue where we were ignoring settings changes.
  * Fix artifact download issues.

  View the [v0.10.1 changelog](https://github.com/wandb/wandb/releases/tag/v0.10.1) on GitHub.
</Update>

<Update label="v0.10.0" description="September 11, 2020">
  ## Enhancement

  * Added history sparklines at end of run.
  * Artifact improvements and API for linking.
  * Improved offline support and syncing.
  * Basic noop mode support to simplify testing.
  * Improved windows/pycharm support.
  * Run object has more modifiable properties.
  * Public API supports attaching artifacts to historic runs.

  ## Bug Fix

  * Many bugs fixed due to simplifying logic.

  View the [v0.10.0 changelog](https://github.com/wandb/wandb/releases/tag/v0.10.0) on GitHub.
</Update>

<Update label="v0.9.7" description="September 08, 2020">
  ## Enhancement

  * New sacred observer available at wandb.sacred.WandbObserver.
  * Improved artifact reference tracking for HTTP urls.

  ## Bug Fix

  * Print meaningful error message when runs are queried with `summary` instead of `summary_metrics`.

  View the [v0.9.7 changelog](https://github.com/wandb/wandb/releases/tag/v0.9.7) on GitHub.
</Update>

<Update label="v0.9.6" description="August 28, 2020">
  ## Enhancement

  * Sub paths of artifacts now expose an optional root directory argument to download().
  * Artifact.new\_file accepts an optional mode argument.
  * Removed legacy fastai docs as we're now packaged with fastai v2!

  ## Bug Fix

  * Fix yaml parsing error handling logic.
  * Bad spelling in torch docstring, thanks @mkkb473.

  View the [v0.9.6 changelog](https://github.com/wandb/wandb/releases/tag/v0.9.6) on GitHub.
</Update>

<Update label="v0.9.5" description="August 17, 2020">
  ## Enhancement

  * Remove unused y\_probas in sklearn plots, thanks @dreamflasher.
  * New deletion apis for artifacts.

  ## Bug Fix

  * Fix `wandb restore` when not logged in.
  * Fix artifact download paths on Windows.
  * Retry 408 errors on upload.
  * Fix mask numeric types, thanks @numpee.
  * Fix artifact reference naming mixup.

  View the [v0.9.5 changelog](https://github.com/wandb/wandb/releases/tag/v0.9.5) on GitHub.
</Update>

<Update label="v0.9.4" description="July 24, 2020">
  ## Enhancement

  * Default pytorch histogram logging frequency from 100 -> 1000 steps.

  ## Bug Fix

  * Fix multiple prompts for login when using the command line.
  * Fix "no method rename\_file" error.
  * Fixed edgecase histogram calculation in PyTorch.
  * Fix error in jupyter when saving session history.
  * Correctly return artifact metadata in public api.
  * Fix matplotlib / plotly rendering error.

  View the [v0.9.4 changelog](https://github.com/wandb/wandb/releases/tag/v0.9.4) on GitHub.
</Update>

<Update label="v0.9.3" description="July 10, 2020">
  ## Enhancement

  * New artifact cli commands!

  ```shell theme={null}
  wandb artifact put path_file_or_ref
  wandb artifact get artifact:version
  wandb artifact ls project_name
  ```

  * New artifact api commands!

  ```python theme={null}
  wandb.log_artifact()
  wandb.use_artifact()
  wandb.Api().artifact_versions()
  wandb.Api().run.used_artifacts()
  wandb.Api().run.logged_artifacts()
  wandb.Api().Artifact().file()
  ```

  * Improved syncing of large wandb-history.jsonl files for wandb sync.
  * New Artifact.verify method to ensure the integrity of local artifacts.
  * Better testing harness for api commands.
  * Run directory now store local time instead of utc time in the name, thanks @aiyolo!
  * Improvements to our doc strings across the board.
  * Wandb.Table now supports a `dataframe` argument for logging dataframes as tables!

  ## Bug Fix

  * Artifacts work in python2.
  * Artifacts default download locations work in Windows.
  * GCS references now properly cache / download, thanks @yoks!
  * Fix encoding of numpy arrays to JSON.
  * Fix string comparison error message.

  View the [v0.9.3 changelog](https://github.com/wandb/wandb/releases/tag/v0.9.3) on GitHub.
</Update>

<Update label="v0.9.2" description="June 29, 2020">
  ## Enhancement

  * Major overhaul of artifact caching.
  * Configurable cache directory for artifacts.
  * Configurable download directory for artifacts.
  * New Artifact.verify method to ensure the integrity of local artifacts.
  * Use\_artifact no longer requires `type`.
  * Deleted artifacts can now be recommitted.
  * Lidar scenes now support vectors.

  ## Bug Fix

  * Fix issue with artifact downloads returning errors.
  * Segmentation masks now handle non-unint8 data.
  * Fixed path parsing logic in `api.runs()`.

  View the [v0.9.2 changelog](https://github.com/wandb/wandb/releases/tag/v0.9.2) on GitHub.
</Update>

<Update label="v0.9.1" description="June 09, 2020">
  ## Bug Fix

  * Fix issue where files were always logged to latest run in a project.
  * Fix issue where url was not display url on first call to wandb.init().

  View the [v0.9.1 changelog](https://github.com/wandb/wandb/releases/tag/v0.9.1) on GitHub.
</Update>

<Update label="v0.9.0" description="June 05, 2020">
  ## Bug Fix

  * Handle multiple inits in Jupyter.
  * Handle ValueError's when capturing signals, thanks @jsbroks.
  * Wandb agent handles rate limiting properly.

  ## Enhancement

  * Wandb.Artifact is now generally available!
  * Feature\_importances now supports CatBoost, thanks @neomatrix369.

  View the [v0.9.0 changelog](https://github.com/wandb/wandb/releases/tag/v0.9.0) on GitHub.
</Update>

<Update label="v0.8.36" description="May 11, 2020">
  ## Bug Fix

  * Catch all exceptions when saving Jupyter sessions.
  * Validation\_data automatically set in TF >= 2.2.
  * *implements*\* hooks now implemented in keras callback for TF >= 2.2.

  ## Enhancement

  * Raw source code saving now disabled by default.
  * We now support global settings on boot to enable code saving on the server.
  * New `code_save=True` argument to wandb.init() to enable code saving manually.

  View the [v0.8.36 changelog](https://github.com/wandb/wandb/releases/tag/v0.8.36) on GitHub.
</Update>

<Update label="v0.8.35" description="May 01, 2020">
  ## Bug Fix

  * Ensure cells don't hang on completion.
  * Fixed jupyter integration in PyCharm shells.
  * Made session history saving handle None metadata in outputs.

  View the [v0.8.35 changelog](https://github.com/wandb/wandb/releases/tag/v0.8.35) on GitHub.
</Update>

<Update label="v0.8.34" description="April 28, 2020">
  ## Enhancement

  * Save session history in jupyter notebooks.
  * Kaggle internet enable notification.
  * Extend wandb.plots.feature\_importances to work with more model types, thanks @neomatrix369!

  ## Bug Fix

  * Code saving for jupyter notebooks restored.
  * Fixed thread errors in jupyter.
  * Ensure final history rows aren't dropped in jupyter.

  View the [v0.8.34 changelog](https://github.com/wandb/wandb/releases/tag/v0.8.34) on GitHub.
</Update>

<Update label="v0.8.33" description="April 24, 2020">
  ## Enhancement

  * Add default class labels for semantic segmentation.
  * Enhance bounding box API to be similar to semantic segmentation API.

  ## Bug Fix

  * Increase media table rows to improve ROC/PR curve logging.
  * Fix issue where pre binned histograms were not being handled properly.
  * Handle nan values in pytorch histograms.
  * Fix handling of binary image masks.

  View the [v0.8.33 changelog](https://github.com/wandb/wandb/releases/tag/v0.8.33) on GitHub.
</Update>

<Update label="v0.8.32" description="April 14, 2020">
  ## Enhancement

  * Improve semantic segmentation image mask logging.

  View the [v0.8.32 changelog](https://github.com/wandb/wandb/releases/tag/v0.8.32) on GitHub.
</Update>

<Update label="v0.8.31" description="March 19, 2020">
  ## Enhancement

  * Close all open files to avoice ResourceWarnings, thanks @CrafterKolyan!

  ## Bug Fix

  * Parse "tensor" protobufs, fixing issues with tensorboard syncing in 2.1.

  View the [v0.8.31 changelog](https://github.com/wandb/wandb/releases/tag/v0.8.31) on GitHub.
</Update>

<Update label="v0.8.30" description="March 19, 2020">
  ## Enhancement

  * Add ROC, precision\_recall, HeatMap, explainText, POS, and NER to wandb.plots.
  * Add wandb.Molecule() logging.
  * Capture kaggle runs for metrics.
  * Add ability to watch from run object.

  ## Bug Fix

  * Avoid accidently picking up global debugging logs.

  View the [v0.8.30 changelog](https://github.com/wandb/wandb/releases/tag/v0.8.30) on GitHub.
</Update>

<Update label="v0.8.29" description="March 05, 2020">
  ## Enhancement

  * Improve bounding box annotations.
  * Log active GPU system metrics.
  * Only writing wandb/settings file if wandb init is called.
  * Improvements to wandb local command.

  ## Bug Fix

  * Fix GPU logging on some devices without power metrics.
  * Fix sweep config command handling.
  * Fix tensorflow string logging.

  View the [v0.8.29 changelog](https://github.com/wandb/wandb/releases/tag/v0.8.29) on GitHub.
</Update>

<Update label="v0.8.28" description="February 21, 2020">
  ## Enhancement

  * Added code saving of main python module.
  * Added ability to specify metadata for bounding boxes and segmentation masks.

  ## Bug Fix

  * Fix situations where uncommitted data from wandb.log() is not persisted.

  View the [v0.8.28 changelog](https://github.com/wandb/wandb/releases/tag/v0.8.28) on GitHub.
</Update>

<Update label="v0.8.27" description="February 11, 2020">
  ## Bug Fix

  * Fix dependency conflict with new versions of six package.

  View the [v0.8.27 changelog](https://github.com/wandb/wandb/releases/tag/v0.8.27) on GitHub.
</Update>

<Update label="v0.8.26" description="February 10, 2020">
  ## Enhancement

  * Add best metric and epoch to run summary with Keras callback.
  * Added wandb.run.config\_static for environments required pickled config.

  ## Bug Fix

  * Fixed regression causing failures with wandb.watch() and DataParallel.
  * Improved compatibility with python 3.8.
  * Fix model logging under windows.

  View the [v0.8.26 changelog](https://github.com/wandb/wandb/releases/tag/v0.8.26) on GitHub.
</Update>

<Update label="v0.8.25" description="February 04, 2020">
  ## Bug Fix

  * Fix exception when using wandb.watch() in a notebook.
  * Improve support for sparse tensor gradient logging on GPUs.

  View the [v0.8.25 changelog](https://github.com/wandb/wandb/releases/tag/v0.8.25) on GitHub.
</Update>

<Update label="v0.8.24" description="February 03, 2020">
  ## Bug Fix

  * Relax version dependency for PyYAML for users with old environments.

  View the [v0.8.24 changelog](https://github.com/wandb/wandb/releases/tag/v0.8.24) on GitHub.
</Update>

<Update label="v0.8.23" description="February 03, 2020">
  ## Enhancement

  * Added scikit-learn support.
  * Added ability to specify/exclude specific keys when building wandb.config.

  ## Bug Fix

  * Fix wandb.watch() on sparse tensors.
  * Fix incompatibilty with ray 0.8.1.
  * Fix missing pyyaml requirement.
  * Fix "W\&B process failed to launch" problems.
  * Improved ability to log large model graphs and plots.

  View the [v0.8.23 changelog](https://github.com/wandb/wandb/releases/tag/v0.8.23) on GitHub.
</Update>

<Update label="v0.8.22" description="January 24, 2020">
  ## Enhancement

  * Added ability to configure agent commandline from sweep config.

  ## Bug Fix

  * Fix fast.ai prediction logging.
  * Fix logging of eager tensorflow tensors.
  * Fix jupyter issues with logging notebook name and wandb.watch().

  View the [v0.8.22 changelog](https://github.com/wandb/wandb/releases/tag/v0.8.22) on GitHub.
</Update>

<Update label="v0.8.21" description="January 15, 2020">
  ## Enhancement

  * Ignore wandb.init() specified project and entity when running a sweep.

  ## Bug Fix

  * Fix agent "flapping" detection.
  * Fix local controller not starting when sweep is pending.

  View the [v0.8.21 changelog](https://github.com/wandb/wandb/releases/tag/v0.8.21) on GitHub.
</Update>

<Update label="v0.8.20" description="January 10, 2020">
  ## Enhancement

  * Added support for LightGBM.
  * Added local board support (Experimental).
  * Added ability to modify sweep configuration.
  * Added GPU power logging to system metrics.

  ## Bug Fix

  * Prevent sweep agent from failing continuously when misconfigured.

  View the [v0.8.20 changelog](https://github.com/wandb/wandb/releases/tag/v0.8.20) on GitHub.
</Update>

<Update label="v0.8.19" description="December 18, 2019">
  ## Enhancement

  * Added beta support for ray/tune hyperopt search strategy.
  * Added ability to specify max runs per agent.
  * Improve experience starting a sweep without a project already created.

  ## Bug Fix

  * Fix repeated wandb.Api().Run(id).scan\_history() calls get updated data.
  * Fix early\_terminate/hyperband in notebook/python environments.

  View the [v0.8.19 changelog](https://github.com/wandb/wandb/releases/tag/v0.8.19) on GitHub.
</Update>

<Update label="v0.8.18" description="December 04, 2019">
  ## Enhancement

  * Added min\_step and max\_step to run.scan\_history for grabbing sub-sections of metrics.
  * Wandb.init(reinit=True) now automatically calls wandb.join() to better support multiple runs per process.

  ## Bug Fix

  * Wandb.init(sync\_tensorboard=True) works again for TensorFlow 2.0.

  View the [v0.8.18 changelog](https://github.com/wandb/wandb/releases/tag/v0.8.18) on GitHub.
</Update>

<Update label="v0.8.17" description="December 02, 2019">
  ## Enhancement

  * Handle tags being passed in as a string.

  ## Bug Fix

  * Pin graphql-core \<  3.0.0 to fix install errors.
  * TQDM progress bars update logs properly.
  * Oversized summary or history logs are now dropped which prevents retry hanging.

  View the [v0.8.17 changelog](https://github.com/wandb/wandb/releases/tag/v0.8.17) on GitHub.
</Update>

<Update label="v0.8.16" description="November 21, 2019">
  ## Bug Fix

  * Fix regression syncing some versions of Tensorboard since 0.8.13.
  * Fix network error in Jupyter.

  View the [v0.8.16 changelog](https://github.com/wandb/wandb/releases/tag/v0.8.16) on GitHub.
</Update>

<Update label="v0.8.15" description="November 05, 2019">
  ## Bug Fix

  * Fix calling wandb.init() with sync\_tensorboard multiple times in Jupyter.
  * Fix RuntimeError race when using threads and calling wandb.log().
  * Don't initialize Sentry when error reporting is disabled.

  ## Enhancement

  * Added best\_run() to wandb.sweep() public Api objects.
  * Remove internal tracking keys from wandb.config objects in the public Api.

  View the [v0.8.15 changelog](https://github.com/wandb/wandb/releases/tag/v0.8.15) on GitHub.
</Update>

<Update label="v0.8.14" description="November 01, 2019">
  ## Bug Fix

  * Improve large object warning when values reach maximum size.
  * Warn when wandb.save() isn't passed a string.
  * Run stopping from the UI works since regressing in 0.8.12.
  * Restoring a file that already exists locally works.
  * Fixed TensorBoard incorrectly placing some keys in the wrong step since 0.8.10.
  * Wandb.Video only accepts uint8 instead of incorrectly converting to floats.
  * SageMaker environment detection is now more robust.
  * Resuming correctly populates config.
  * Wandb.restore respects root when run.dir is set #658.
  * Calling wandb.watch() multiple times properly namespaces histograms and graphs.

  ## Enhancement

  * Sweeps now work in Windows!
  * Added sweep attribute to Run in the public api.
  * Added sweep link to Jupyter and terminal output.
  * TensorBoard logging now stores proper timestamps when importing historic results.
  * TensorBoard logging now supports configuring rate\_limits and filtering event types.
  * Use simple output mirroring stdout doesn't have a file descriptor.
  * Write wandb meta files to the system temp directory if the local directory isn't writable.
  * Added beta api.reports to the public API.
  * Added wandb.unwatch to remove hooks from pytorch models.
  * Store the framework used in config.\_wandb.

  View the [v0.8.14 changelog](https://github.com/wandb/wandb/releases/tag/v0.8.14) on GitHub.
</Update>

<Update label="v0.8.13" description="October 15, 2019">
  ## Bug Fix

  * Create nested directory when videos are logged from tensorboard namespaces.
  * Fix race when using wandb.log() `async=True`.
  * Run.summary acts like a proper dictionary.
  * Run.summary sub dictionaries properly render.
  * Handle None when passing class\_colors for segmentation masks.
  * Handle tensorflow2 not having a SessionHook.
  * Properly escape args in windows.
  * Fix hanging login when in anonymode.
  * Tf2 keras patch now handles missing callbacks args.

  ## Enhancement

  * Updates documentation autogenerated from docstrings in /docs.
  * Wandb.init(config=config\_dict) does not update sweep specified parameters.
  * Wandb.config object now has a setdefaults method enabling improved sweep support.
  * Improved terminal and jupyter message incorporating emojii!
  * Allow wandb.watch() to be called multiple times on different models.
  * Improved support for watching multiple tfevent files.
  * Windows no longer requires `wandb run` simply run `python script_name.py`.
  * `wandb agent` now works on windows.
  * Nice error message when wandb.log() is called without a dict.
  * Keras callback has a new `log_batch_frequency` for logging metrics every N batches.

  View the [v0.8.13 changelog](https://github.com/wandb/wandb/releases/tag/v0.8.13) on GitHub.
</Update>

<Update label="v0.8.12" description="September 20, 2019">
  ## Bug Fix

  * Fix compatibility issue with python 2.7 and old pip dependencies.

  ## Enhancement

  * Improved onboarding flow when creating new accounts and entering api\_key.

  View the [v0.8.12 changelog](https://github.com/wandb/wandb/releases/tag/v0.8.12) on GitHub.
</Update>

<Update label="v0.8.11" description="September 19, 2019">
  ## Bug Fix

  * Fix public api returning incorrect data when config value is 0 or False.
  * Resumed runs no longer overwrite run names with run id.

  ## Enhancement

  * Added recording of spell.run id in config.

  View the [v0.8.11 changelog](https://github.com/wandb/wandb/releases/tag/v0.8.11) on GitHub.
</Update>

<Update label="v0.8.10" description="September 13, 2019">
  ## Bug Fix

  * Wandb magic handles the case of tf.keras and keras being loaded.
  * Tensorboard logging won't drop steps if multiple loggers have different global\_steps.
  * Keras gradient logging works in the latest tf.keras.
  * Keras validation\_data is properly set in tensorflow 2.
  * Wandb pull command creates directories if they don't exist, thanks @chmod644.
  * File upload batching now asserts a minimum size.
  * Sweeps works in python2 again.
  * Scan\_history now iterates the full set of points.
  * Jupyter will run local mode if credentials can't be obtained.

  ## Enhancement

  * Sweeps can now be run from within jupyter / directly from python! [https://docs.wandb.com/sweeps/python](https://docs.wandb.com/sweeps/python).
  * New openai gym integration will automatically log videos, enabled with the monitor\_gym keyword argument to wandb.init().
  * Ray Tune logging callback in wandb.ray.WandbLogger.
  * New global config file in \~/.config/wandb for global settings.
  * Added tests for fastai, thanks @borisdayma.
  * Public api performance enhancements.
  * Deprecated username in favor of entity in the public api for consistency.
  * Anonymous login support enabled by default.
  * New wandb.login() method to be used in jupyter enabling anonymous logins.
  * Better dependency error messages for data frames.
  * Initial integration with spell.run.
  * All images are now rendered as PNG to avoid JPEG artifacts.
  * Public api now has a projects field.

  View the [v0.8.10 changelog](https://github.com/wandb/wandb/releases/tag/v0.8.10) on GitHub.
</Update>

<Update label="v0.8.9" description="August 19, 2019">
  ## Bug Fix

  * Run.summary updates work in jupyter before log is called.
  * Don't require numpy to be installed.
  * Setting nested keys in summary works.
  * Notebooks in nested directories are properly saved.
  * Don't retry 404's / better error messaging from the server.
  * Strip leading slashes when loading paths in the public api.

  ## Enhancement

  * Small files are batch uploaded as gzipped tarballs.
  * TensorBoardX gifs are logged to wandb.

  View the [v0.8.9 changelog](https://github.com/wandb/wandb/releases/tag/v0.8.9) on GitHub.
</Update>

<Update label="v0.8.8" description="August 13, 2019">
  ## Bug Fix

  * Wandb.init properly handles network failures on startup.
  * Keras callback only logs examples if data\_type or input\_type is set.
  * Fix edge case PyTorch model logging bug.
  * Handle patching tensorboard multiple times in jupyter.
  * Sweep picks up config.yaml from the run directory.
  * Dataframes handle integer labels.
  * Handle invalid JSON when querying jupyter servers.

  ## Enhancement

  * Fastai uses a fixed seed for example logging.
  * Increased the max number of images for fastai callback.
  * New wandb.Video tag for logging video.
  * Sync=False argument to wandb.log() moves logging to a thread.
  * New local sweep controller for custom search logic.
  * Anonymous login support for easier onboarding.
  * Calling wandb.init() multiple times in jupyter doesn't error out.

  View the [v0.8.8 changelog](https://github.com/wandb/wandb/releases/tag/v0.8.8) on GitHub.
</Update>

<Update label="v0.8.7" description="August 07, 2019">
  ## Bug Fix

  * Keras callback no longer guesses input\_type for 2D data.
  * Wandb.Image handles images with 1px height.

  ## Enhancement

  * Wandb Public API now has `run.scan_history` to return all history rows.
  * Wandb.config prints helpful errors if used before calling init.
  * Wandb.summary prints helpful errors if used before calling init.
  * Filestream api points to new url on the backend.

  View the [v0.8.7 changelog](https://github.com/wandb/wandb/releases/tag/v0.8.7) on GitHub.
</Update>

<Update label="v0.8.6" description="July 31, 2019">
  ## Bug Fix

  * Fastai callback uses the default monitor instead of assuming val\_loss.
  * Notebook introspections handles error cases and doesn't print stacktrace on failure.
  * Don't print description warning when setting name.
  * Fixed dataframe logging error with the keras callback.
  * Fixed line offsets in logs when resuming runs.
  * Wandb.config casts non-builtins before writing to yaml.
  * Vendored backports.tempfile to address missing package on install.

  ## Enhancement

  * Added `api.sweep` to the python export api for querying sweeps.
  * Added `WANDB_NOTEBOOK_NAME` for specifying the notebook name in cases we can't infer it.
  * Added `WANDB_HOST` to override hostnames.
  * Store if a run was run within jupyter.
  * Wandb now supports stopping runs from the web ui.
  * Handle floats passed as step to `wandb.log()`.
  * Wandb.config has full unicode support.
  * Sync the main file to wandb if code saving is enabled and it's untracked by git.
  * XGBoost callback: wandb.xgboost.wandb\_callback().

  View the [v0.8.6 changelog](https://github.com/wandb/wandb/releases/tag/v0.8.6) on GitHub.
</Update>

<Update label="v0.8.5" description="July 12, 2019">
  ## Bug Fix

  * Fixed plotly charts with large numpy arrays not rendering.
  * `wandb docker` works when nvidia is present.
  * Better error when non string keys are sent to log.
  * Relaxed pyyaml dependency to fix AMI installs.
  * Magic works in jupyter notebooks.

  ## Enhancement

  * New preview release of auto-dataframes for Keras.
  * Added input\_type and output\_type to the Keras callback for simpler config.
  * Public api supports retrieving specific keys and custom xaxis.

  View the [v0.8.5 changelog](https://github.com/wandb/wandb/releases/tag/v0.8.5) on GitHub.
</Update>

<Update label="v0.8.4" description="July 08, 2019">
  ## Bug Fix

  * WANDB\_IGNORE\_GLOBS is respected on the final scan of files.
  * Unified run.id, run.name, and run.notes across all apis.
  * Handle funky terminal sizes when setting up our pseudo tty.
  * Fixed Jupyter notebook introspection logic.
  * Run.summary.update() persists changes to the server.
  * Tensorboard syncing is robust to invalid histograms and truncated files.

  ## Enhancement

  * Preview release of magic, calling wandb.init(magic=True) should automatically track config and metrics when possible.
  * Cli now supports local installs of the backend.
  * Fastai callback supports logging example images.

  View the [v0.8.4 changelog](https://github.com/wandb/wandb/releases/tag/v0.8.4) on GitHub.
</Update>

<Update label="v0.8.3" description="June 26, 2019">
  ## Bug Fix

  * Image logging works in Windows.
  * Wandb sync handles tfevents with a single timestep.
  * Fix incorrect command in overview page for running runs.
  * Handle histograms with > 512 bins when streaming tensorboard.
  * Better error message when calling wandb sync on a file instead of a directory.

  ## Enhancement

  * New helper function for handling hyperparameters in sweeps `wandb.config.user_items()`.
  * Better mocking for improved testing.

  View the [v0.8.3 changelog](https://github.com/wandb/wandb/releases/tag/v0.8.3) on GitHub.
</Update>

<Update label="v0.8.2" description="June 20, 2019">
  ## Bug Fix

  * Entity is persisted on wandb.run when queried from the server.
  * Tmp files always use the temporary directory to avoid syncing.
  * Raise error if file shrinks while uploading.
  * Images log properly in windows.
  * Upgraded pyyaml requirement to address CVE.
  * No longer store a history of rows to prevent memory leak.

  ## Enhancement

  * Summary now supports new dataframe format.
  * WANDB\_SILENT environment variable writes all wandb messages to debug.log.
  * Improved error messages for windows and tensorboard logging.
  * Output.log is uploaded at the end of each run.
  * Metadata, requirements, and patches are uploaded at the beginning of a run.
  * When not running from a git repository, store the main python file.
  * Added WANDB\_DISABLE\_CODE to prevent diffing and code saving.
  * When running in jupyter store the name of the notebook.
  * Auto-login support for colab.
  * Store url to colab notebook.
  * Store the version of this library in config.
  * Store sys.executable in metadata.
  * Fastai callback no longer requires path.
  * Wandb.init now accepts a notes argument.
  * The cli replaced the message argument with notes and name.

  View the [v0.8.2 changelog](https://github.com/wandb/wandb/releases/tag/v0.8.2) on GitHub.
</Update>

<Update label="v0.8.1" description="May 23, 2019">
  ## Bug Fix

  * Wandb sync handles tensorboard embeddings.
  * Wandb sync correctly handles images in tensorboard.
  * Tf.keras correctly handles single input functional models.
  * Wandb.Api().runs returns an iterator that's reusable.
  * WANDB\_DIR within a hidden directory doesn't prevent syncing.
  * Run.files() iterates over all files.
  * Pytorch recursion too deep error.

  ## Enhancement

  * Wandb sync accepts an --ignore argument with globs to skip files.
  * Run.summary now has an items() method for iterating over all keys.

  View the [v0.8.1 changelog](https://github.com/wandb/wandb/releases/tag/v0.8.1) on GitHub.
</Update>

<Update label="v0.8.0" description="May 17, 2019">
  ## Bug Fix

  * Better error messages on access denied.
  * Better error messages when optional packages aren't installed.
  * Urls printed to the terminal are url-escaped.
  * Namespaced tensorboard events work with histograms.
  * Public API now retries on failures and re-uses connection pool.
  * Catch git errors when remotes aren't pushed to origin.
  * Moved keras graph collection to on\_train\_begin to handle unbuilt models.
  * Handle more cases of not being able to save weights.
  * Updates to summary after resuming are persisted.
  * PyTorch histc logging fixed in 0.4.1.
  * Fixed `wandb sync` tensorboard import.

  ## Enhancement

  * Wandb.init(tensorboard=True) works with Tensorflow 2 and Eager Execution.
  * Wandb.init(tensorboard=True) now works with tb-nightly and PyTorch.
  * Automatically log examples with tf.keras by adding missing validation\_data.
  * Socket only binds to localhost for improved security and prevents firewall warnings in OSX.
  * Added user object to public api for getting the source user.
  * Added run.display\_name to the public api.
  * Show display name in console output.
  * Added --tags, --job\_group, and --job\_type to `wandb run`.
  * Added environment variable for minimum time to run before considering crashed.
  * Added flake8 tests to CI, thanks @cclauss!

  View the [v0.8.0 changelog](https://github.com/wandb/wandb/releases/tag/v0.8.0) on GitHub.
</Update>

<Update label="v0.7.3" description="April 15, 2019">
  ## Bug Fix

  * Wandb-docker-run accepts image digests.
  * Keras callback works in tensorflow2-alpha0.
  * Keras model graph now puts input layer first.

  ## Enhancement

  * PyTorch log frequency added for gradients and weights.
  * PyTorch logging performance enhancements.
  * Wandb.init now accepts a name parameter for naming runs.
  * Wandb.run.name reflects custom display names.
  * Improvements to nested summary values.
  * Deprecated wandb.Table.add\_row in favor of wandb.Table.add\_data.
  * Initial support for a fast.ai callback thanks to @borisdayma!

  View the [v0.7.3 changelog](https://github.com/wandb/wandb/releases/tag/v0.7.3) on GitHub.
</Update>

<Update label="v0.7.2" description="March 19, 2019">
  ## Bug Fix

  * Run.get\_url resolves the default entity if one wasn't specified.
  * Wandb restore accepts run paths with only slashes.
  * Fixed PyYaml deprecation warnings.
  * Added entrypoint shell script to manifest.
  * Strip newlines from cuda version.

  View the [v0.7.2 changelog](https://github.com/wandb/wandb/releases/tag/v0.7.2) on GitHub.
</Update>

<Update label="v0.7.1" description="March 14, 2019">
  ## Bug Fix

  * Handle case insensitive docker credentials.
  * Fix app\_url for private cloud login flow.
  * Don't retry 404's when starting sweep agents.

  View the [v0.7.1 changelog](https://github.com/wandb/wandb/releases/tag/v0.7.1) on GitHub.
</Update>

<Update label="v0.7.0" description="February 28, 2019">
  ## Bug Fix

  * Ensure DNS lookup failures can't prevent startup.
  * Centralized debug logging.
  * Wandb agent waits longer to send a SIGKILL after sending SIGINT.

  ## Enhancement

  * Support for logging docker images with the WANDB\_DOCKER env var.
  * WANDB\_DOCKER automatically set when run in kubernetes.
  * New wandb-docker-run command to automatically set env vars and mount code.
  * Wandb.restore supports launching docker for runs that ran with it.
  * Python packages are now recorded and saved in a requirements.txt file.
  * Cpu\_count, gpu\_count, gpu, os, and python version stored in wandb-metadata.json.
  * The export api now supports docker-like paths, i.e. username/project:run\_id.
  * Better first time user messages and login info.

  View the [v0.7.0 changelog](https://github.com/wandb/wandb/releases/tag/v0.7.0) on GitHub.
</Update>

<Update label="v0.6.35" description="January 29, 2019">
  ## Bug Fix

  * Improve error reporting for sweeps.

  View the [v0.6.35 changelog](https://github.com/wandb/wandb/releases/tag/v0.6.35) on GitHub.
</Update>

<Update label="v0.6.35" description="January 29, 2019">
  ## Bug Fix

  * Improve error reporting for sweeps.

  View the [v0.6.35 changelog](https://github.com/wandb/wandb/releases/tag/v0.6.35) on GitHub.
</Update>

<Update label="v0.6.35" description="January 29, 2019">
  ## Bug Fix

  * Improve error reporting for sweeps.

  View the [v0.6.35 changelog](https://github.com/wandb/wandb/releases/tag/v0.6.35) on GitHub.
</Update>

<Update label="v0.6.35" description="January 29, 2019">
  ### Bug Fix

  * Improve error reporting for sweeps.

  View the [v0.6.35 changelog](https://github.com/wandb/wandb/releases/tag/v0.6.35) on GitHub.
</Update>

<Update label="v0.6.35" description="January 29, 2019">
  ## Bug Fix

  * Improve error reporting for sweeps.

  View the [v0.6.35 changelog](https://github.com/wandb/wandb/releases/tag/v0.6.35) on GitHub.
</Update>

<Update label="v0.6.35" description="January 29, 2019">
  ## Bug Fix

  * Improve error reporting for sweeps.

  View the [v0.6.35 changelog](https://github.com/wandb/wandb/releases/tag/v0.6.35) on GitHub.
</Update>

<Update label="v0.6.34" description="January 23, 2019">
  ## Bug Fix

  * Fixed Jupyter logging, don't change logger level.
  * Fixed resuming in Jupyter.

  ## Enhancement

  * Wandb.init now degrades gracefully if a user hasn't logged in to wandb.
  * Added a **force** flag to wandb.init() to require a machine to be logged in.
  * Tensorboard and TensorboardX logging is now automatically instrumented when enabled.
  * Added a **tensorboard** to wandb.init() which patches tensorboard for logging.
  * Wandb.save handles now accepts a base path to files in sub directories.
  * Wandb.tensorflow and wandb.tensorboard can now be accessed without directly importing.
  * `wandb sync` will now traverse a wandb run directory and sync all runs.

  View the [v0.6.34 changelog](https://github.com/wandb/wandb/releases/tag/v0.6.34) on GitHub.
</Update>

<Update label="v0.6.34" description="January 23, 2019">
  ## Bug Fix

  * Fixed Jupyter logging, don't change logger level.
  * Fixed resuming in Jupyter.

  ## Enhancement

  * Wandb.init now degrades gracefully if a user hasn't logged in to wandb.
  * Added a **force** flag to wandb.init() to require a machine to be logged in.
  * Tensorboard and TensorboardX logging is now automatically instrumented when enabled.
  * Added a **tensorboard** to wandb.init() which patches tensorboard for logging.
  * Wandb.save handles now accepts a base path to files in sub directories.
  * Wandb.tensorflow and wandb.tensorboard can now be accessed without directly importing.
  * `wandb sync` will now traverse a wandb run directory and sync all runs.

  View the [v0.6.34 changelog](https://github.com/wandb/wandb/releases/tag/v0.6.34) on GitHub.
</Update>

<Update label="v0.6.34" description="January 23, 2019">
  ## Bug Fix

  * Fixed Jupyter logging, don't change logger level.
  * Fixed resuming in Jupyter.

  ## Enhancement

  * Wandb.init now degrades gracefully if a user hasn't logged in to wandb.
  * Added a **force** flag to wandb.init() to require a machine to be logged in.
  * Tensorboard and TensorboardX logging is now automatically instrumented when enabled.
  * Added a **tensorboard** to wandb.init() which patches tensorboard for logging.
  * Wandb.save handles now accepts a base path to files in sub directories.
  * Wandb.tensorflow and wandb.tensorboard can now be accessed without directly importing.
  * `wandb sync` will now traverse a wandb run directory and sync all runs.

  View the [v0.6.34 changelog](https://github.com/wandb/wandb/releases/tag/v0.6.34) on GitHub.
</Update>

<Update label="v0.6.34" description="January 23, 2019">
  ### Bug Fix

  * fixed Jupyter logging, don't change logger level.
  * fixed resuming in Jupyter.

  ### Enhancement

  * wandb.init() now degrades gracefully if a user hasn't logged in to wandb.
  * added a **force** flag to wandb.init() to require a machine to be logged in.
  * Tensorboard and TensorboardX logging is now automatically instrumented when enabled.
  * added a **tensorboard** to wandb.init() which patches tensorboard for logging.
  * wandb.save() handles now accepts a base path to files in sub directories.
  * wandb.tensorflow and wandb.tensorboard can now be accessed without directly importing.
  * `wandb sync` will now traverse a wandb run directory and sync all runs.

  View the [v0.6.34 changelog](https://github.com/wandb/wandb/releases/tag/v0.6.34) on GitHub.
</Update>

<Update label="v0.6.34" description="January 23, 2019">
  ## Bug Fix

  * Fixed Jupyter logging, don't change logger level.
  * Fixed resuming in Jupyter.

  ## Enhancement

  * Wandb.init now degrades gracefully if a user hasn't logged in to wandb.
  * Added a **force** flag to wandb.init() to require a machine to be logged in.
  * Tensorboard and TensorboardX logging is now automatically instrumented when enabled.
  * Added a **tensorboard** to wandb.init() which patches tensorboard for logging.
  * Wandb.save handles now accepts a base path to files in sub directories.
  * Wandb.tensorflow and wandb.tensorboard can now be accessed without directly importing.
  * `wandb sync` will now traverse a wandb run directory and sync all runs.

  View the [v0.6.34 changelog](https://github.com/wandb/wandb/releases/tag/v0.6.34) on GitHub.
</Update>

<Update label="v0.6.34" description="January 23, 2019">
  ## Bug Fix

  * Fixed Jupyter logging, don't change logger level.
  * Fixed resuming in Jupyter.

  ## Enhancement

  * Wandb.init now degrades gracefully if a user hasn't logged in to wandb.
  * Added a **force** flag to wandb.init() to require a machine to be logged in.
  * Tensorboard and TensorboardX logging is now automatically instrumented when enabled.
  * Added a **tensorboard** to wandb.init() which patches tensorboard for logging.
  * Wandb.save handles now accepts a base path to files in sub directories.
  * Wandb.tensorflow and wandb.tensorboard can now be accessed without directly importing.
  * `wandb sync` will now traverse a wandb run directory and sync all runs.

  View the [v0.6.34 changelog](https://github.com/wandb/wandb/releases/tag/v0.6.34) on GitHub.
</Update>

<Update label="v0.6.33" description="January 22, 2019">
  ## Bug Fix

  * Fixed race where wandb process could hang at the end of a run.

  View the [v0.6.33 changelog](https://github.com/wandb/wandb/releases/tag/v0.6.33) on GitHub.
</Update>

<Update label="v0.6.33" description="January 22, 2019">
  ## Bug Fix

  * Fixed race where wandb process could hang at the end of a run.

  View the [v0.6.33 changelog](https://github.com/wandb/wandb/releases/tag/v0.6.33) on GitHub.
</Update>

<Update label="v0.6.33" description="January 22, 2019">
  ## Bug Fix

  * Fixed race where wandb process could hang at the end of a run.

  View the [v0.6.33 changelog](https://github.com/wandb/wandb/releases/tag/v0.6.33) on GitHub.
</Update>

<Update label="v0.6.33" description="January 22, 2019">
  ### Bug Fix

  * Fixed race where wandb process could hang at the end of a run.

  View the [v0.6.33 changelog](https://github.com/wandb/wandb/releases/tag/v0.6.33) on GitHub.
</Update>

<Update label="v0.6.33" description="January 22, 2019">
  ## Bug Fix

  * Fixed race where wandb process could hang at the end of a run.

  View the [v0.6.33 changelog](https://github.com/wandb/wandb/releases/tag/v0.6.33) on GitHub.
</Update>

<Update label="v0.6.33" description="January 22, 2019">
  ## Bug Fix

  * Fixed race where wandb process could hang at the end of a run.

  View the [v0.6.33 changelog](https://github.com/wandb/wandb/releases/tag/v0.6.33) on GitHub.
</Update>

<Update label="v0.6.32" description="December 22, 2018">
  ## Bug Fix

  * Fix resuming in Jupyter on kernel restart.
  * Wandb.save ensures files are pushed regardless of growth.

  ## Enhancement

  * Added replace=True keyword to init for auto-resuming.
  * New run.resumed property that can be used to detect if we're resuming.
  * New run.step property to use for setting an initial epoch on resuming.
  * Made Keras callback save the best model as it improves.

  View the [v0.6.32 changelog](https://github.com/wandb/wandb/releases/tag/v0.6.32) on GitHub.
</Update>

<Update label="v0.6.31" description="December 20, 2018">
  ## Bug Fix

  * Really don't require numpy.
  * Better error message if wandb.log() is called before wandb.init().
  * Prevent calling wandb.watch() multiple times.
  * Handle datetime attributes in logs / plotly.

  ## Enhancement

  * Add environment to sweeps.
  * Enable tagging in the public API and in wandb.init().
  * New media type wandb.Html for logging arbitrary html.
  * Add Public api.create\_run method for custom integrations.
  * Added glob support to wandb.save(), files save as they're written to.
  * Added wandb.restore() for pulling files on resume.

  View the [v0.6.31 changelog](https://github.com/wandb/wandb/releases/tag/v0.6.31) on GitHub.
</Update>

<Update label="v0.6.30" description="December 06, 2018">
  ## Bug Fix

  * Added a timeout for generating diffs on large repos.
  * Fixed edge case where file syncing could hang.
  * Ensure all file changes are captured before exit.
  * Handle cases of sys.exit where code isn't passed.
  * Don't require numpy.

  ## Enhancement

  * New `wandb sync` command that pushes a local directory to the cloud.
  * Support for syncing tfevents file during training.
  * Detect when running as TFJob and auto group.
  * New Kubeflow module with initial helpers for pipelines.

  View the [v0.6.30 changelog](https://github.com/wandb/wandb/releases/tag/v0.6.30) on GitHub.
</Update>

<Update label="v0.6.29" description="November 26, 2018">
  ## Bug Fix

  * Fixed history / summary bug.

  View the [v0.6.29 changelog](https://github.com/wandb/wandb/releases/tag/v0.6.29) on GitHub.
</Update>

<Update label="v0.6.28" description="November 24, 2018">
  ## Enhancement

  * Initial support for AWS SageMaker.
  * `hook_torch` renamed to `watch` with a deprecation warning.
  * Projects are automatically created if they don't exist.
  * Additional GPU memory\_allocated metric added.
  * Keras Graph stores edges.

  ## Bug Fix

  * PyTorch graph parsing is more robust.
  * Fixed PyTorch 0.3 support.
  * File download API supports WANDB\_API\_KEY authentication.

  View the [v0.6.28 changelog](https://github.com/wandb/wandb/releases/tag/v0.6.28) on GitHub.
</Update>

<Update label="v0.6.27" description="November 13, 2018">
  ## Enhancement

  * Sweeps work with new backend (early release).
  * Summary tracks all history metrics unless they're overridden by directly writing.
    to summary.
  * Files support in data API.

  ## Bug Fix

  * Show ongoing media file uploads in final upload progress.

  View the [v0.6.27 changelog](https://github.com/wandb/wandb/releases/tag/v0.6.27) on GitHub.
</Update>

<Update label="v0.6.26" description="November 09, 2018">
  ## Enhancement

  * Wandb.Audio supports duration.

  ## Bug Fix

  * Pass username header in filestream API.

  View the [v0.6.26 changelog](https://github.com/wandb/wandb/releases/tag/v0.6.26) on GitHub.
</Update>

<Update label="v0.6.25" description="November 08, 2018">
  ## Enhancement

  * New wandb.Audio data type.
  * New step keyword argument when logging metrics.
  * Ability to specify run group and job type when calling wandb.init() or via.
    environment variables. This enables automatic grouping of distributed training runs
    in the UI
  * Ability to override username when using a service account API key.

  ## Bug Fix

  * Handle non-tty environments in Python2.
  * Handle non-existing git binary.
  * Fix issue where sometimes the same image was logged twice during a Keras step.

  View the [v0.6.25 changelog](https://github.com/wandb/wandb/releases/tag/v0.6.25) on GitHub.
</Update>

<Update label="v0.6.23" description="October 19, 2018">
  ## Enhancement

  * PyTorch.
    * Added a new `wandb.hook_torch` method which records the graph and logs gradients & parameters of pytorch models
    * `wandb.Image` detects pytorch tensors and uses **torchvision.utils.make\_grid** to render the image.

  ## Bug Fix

  * `wandb restore` handles the case of not being run from within a git repo.

  View the [v0.6.23 changelog](https://github.com/wandb/wandb/releases/tag/v0.6.23) on GitHub.
</Update>

<Update label="v0.6.22" description="October 18, 2018">
  ## Bug Fix

  * We now open stdout and stderr in raw mode in Python 2 ensuring tools like bpdb work.

  View the [v0.6.22 changelog](https://github.com/wandb/wandb/releases/tag/v0.6.22) on GitHub.
</Update>

<Update label="v0.6.21" description="October 12, 2018">
  ## Enhancement

  * Catastrophic errors are now reported to Sentry unless WANDB\_ERROR\_REPORTING is set to false.
  * Improved error handling and messaging on startup.

  View the [v0.6.21 changelog](https://github.com/wandb/wandb/releases/tag/v0.6.21) on GitHub.
</Update>

<Update label="v0.6.20" description="October 05, 2018">
  ## Bug Fix

  * The first image when calling wandb.log() was not being written, now it is.
  * `wandb.log()` and `run.summary` now remove whitespace from keys.

  View the [v0.6.20 changelog](https://github.com/wandb/wandb/releases/tag/v0.6.20) on GitHub.
</Update>

<Update label="v0.6.19" description="October 05, 2018">
  ## Bug Fix

  * Vendored prompt\_toolkit \<  1.0.15 because the latest ipython is pinned > 2.0.
  * Lazy load wandb.h5 only if `summary` is accessed to improve Data API performance.

  ## Enhancement

  * Jupyter.
    * Deprecated `wandb.monitor` in favor of automatically starting system metrics after the first wandb.log() call
    * Added new **%%wandb** jupyter magic method to display live results
    * Removed jupyter description iframe
  * The Data API now supports `per_page` and `order` options to the `api.runs` method.
  * Initial support for wandb.Table logging.
  * Initial support for matplotlib logging.

  View the [v0.6.19 changelog](https://github.com/wandb/wandb/releases/tag/v0.6.19) on GitHub.
</Update>


# Supported W&B Server releases
Source: https://docs.wandb.ai/release-notes/server-releases

View release notes for W&B Server including new features, improvements, bug fixes, and upgrade instructions.

This page includes release notes for supported releases of W\&B Server.

* For releases that are no longer supported, see [Archived Server releases](/release-notes/server-releases-archived).
* For more information about the W\&B Server release schedule and support, see [W\&B Server release policies](/release-notes/release-policies).

<Note>For details, see [Supported Server releases](https://docs.wandb.ai/release-notes/server-releases) and [Archived Server releases](https://docs.wandb.ai/release-notes/server-releases-archived). A W\&B Server release is supported for 12 months from its initial release date. As a reminder, customers using [Self-Managed](/platform/hosting/hosting-options/self-managed/) are responsible to upgrade to a supported release in time to maintain support.</Note>

<Update label="v0.81.3" description="June 9, 2026">
  ## Fixes

  * Removed an unsupported internal API endpoint.
</Update>

<Update label="v0.81.2" description="June 8, 2026">
  **June 8, 2026**

  ## Fixes

  * Fixed a bug in Dedicated Cloud where an opt-in preview feature could incorrectly be turned off after upgrading to v0.81.1. Deployments are upgraded automatically and no action is required. Contact [support](mailto:support@wandb.com) if you notice problems.
</Update>

<Update label="v0.81.1" description="June 3, 2026">
  **June 3, 2026**

  ## Fixes

  * Improved panel search performance in workspaces, especially for regex-based searches.
  * Fixed a bug where artifact downloads could fail when an artifact was linked in multiple registries because permission checks were evaluated beyond the user’s current accessible context.
  * Fixed a bug where re-linking an already-linked artifact could trigger artifact-link automations unexpectedly.
  * Fixed a bug where storage engine reads could fail after a project rename with the error `project [storage_key] not found`.
  * Fixed a bug where Weave scoring calls could fail for OpenAI audio-enabled models due to an unsupported response format.
</Update>

<Update label="v0.81.0" description="June 2, 2026">
  **June 2, 2026**

  W\&B Server 0.81.0 makes debugging and comparing runs easier with deeper, searchable run logs, a new diff patch viewer for run code, more flexible run filtering, improved artifact download, new triggers for automations, and more. Weave adds configurable trace retention, custom monitor signals with faster evaluation workflows. To keep your deployment secure and stable, this release includes security improvements and mitigations, expanded SCIM support for Okta and Microsoft Entra, and reliability fixes across artifacts, workspaces, and reports.

  ## Support and end of life

  * W\&B Server v0.69 has reached end of life.

  ## Features

  * Connect your IDE or AI agent to the [W\&B MCP Server](/platform/mcp-server) to query and analyze your W\&B data and W\&B's official documentation in natural language. It can run in hosted or local mode.

  * Improved run log viewing helps you debug and analyze long-running and active runs:
    * Keyword search now supports wrap-around navigation, so you can quickly cycle forward or backward through all matches.
    * Logs now refresh every few seconds while a run is active, and new entries stream to the bottom for a seamless experience.
    * For longer runs, you are no longer limited to viewing only the most recent 10,000 log entries. Scroll up to load up to 100,000 older log entries automatically, making it easier to access important details from the beginning of a run, like model architecture and setup information. Click **Jump to bottom** to return to the newest entries efficiently.
    * In the **Runs** list or **Runs** table, selecting **View run logs** from the action menu opens the logs in a new browser tab.

  * When viewing code in a run's **Files** tab, a new diff patch viewer displays changes as GitHub-style diffs with syntax highlighting, per-file separation, unified and split views, and line-change counts. You can copy the patch to your clipboard, download it, or click **Reproduce** to get the exact terminal commands to restore your code state.

  * You can now hide pinned and baseline runs in the run selector. Click the visibility icon next to an individual run to hide it, or click the icon in the heading to show or hide all pinned and baseline runs.

  * When viewing a run in a project, the run status and tags now display in the heading near the run's name.

  * Run filters now support `OR` conditions and grouping with parentheses, so you can build complex expressions. In the filter editor, click **New Filter** to add additional `AND` or `OR` conditions, or click **New group** to combine conditions together, such as `job_type = "train" AND (tags = "baseline" OR tags = "production")`. See [Filter runs](/models/runs/filter-runs).

  * Run filters now support filtering by run ID. To display the run ID column, click the column selector button, then select **Run ID**. Then, enter a partial or full run ID in the search field.

  * To crop the Y-axis on line plots, you can now click and drag vertically in the plot. Cropping one axis preserves the viewport on the other axis.

  * Configure the video seek step to control the time to control the time increment when skipping forward or back in a video panel using keyboard arrow keys. You can customize the setting for an individual panel, a section, or a workspace.

  * You can now navigate full-screen videos using keyboard arrow keys. Click a video to focus it, then use left and right arrow keys to step through the video. This can help you efficiently spot issues like motion artifacts, flicker, or timing problems.

  * In gallery mode, you can now configure whether images are scaled to fit or display at the original dimensions, and you can customize the default fit for a workspace or section.

  * In the **Artifacts** page, you can now download the artifact from the **Files** tab.

  * In an artifact's [lineage graph](/models/artifacts/explore-and-traverse-an-artifact-graph), you can now filter lineage runs by job type.

  * Automations support additional triggers when created from the UI:
    * When a tag is removed from an artifact.
    * When a tag is added to a collection.
    * When a tag is removed from a collection.
    * When an artifact is removed from a collection.

  ### Weave

  * Project or organization admins can now configure how long Weave traces are retained:
    * Specify the default TTL for traces in a project's **Settings** page.
    * Specify the TTL for an individual trace using the **Expiration** field in the trace **Details** view.
  * You can now create custom signals from the Weave UI. In the **Monitors** page, click **Add Signal**, then specify a signal name, description, and prompt.
  * The Weave UI now lets you select or update a scorer and version when editing a monitor.
  * Improved the performance of the **Compare Evaluations** page:
    * The page renders only the first 20 evaluations in the results.
    * To refine the results further, the user's filters are applied before rendering the evaluations.
  * Improved the performance of queries on the **Evaluation Results** page by using server-side sorting, filtering, and pagination.
  * The Weave UI now displays cached tokens.

  ## Performance improvements

  * Improved the performance of **Sweeps** pages by loading agents only as needed, rather than always loading all agents.
  * For large workspaces, improved the performance of opening the editor, changing axes, previewing updates, and applying changes.
  * Improved performance when adding runs to a workspace by cloning a hot path shallowly.

  ## Security

  * Added a boolean environment variable `GRAPHQL_REJECT_UNAUTHED_REQUESTS` to reject unauthorized requests if set to `true`.
  * Improved [SCIM](/platform/hosting/iam/scim) API compatibility when Okta is the IdP.
  * SCIM API now supports automatic account provisioning when Microsoft Entra is the IdP:
    * Added support for complex expressions that mix `[bracket]` and `.value` sub-attributes.
    * Added support for multiple `PATCH` operations on `USER` and `GROUP` objects.
    * A previously deleted user can be re-provisioned via SCIM.
  * Upgraded `cryptography` to 46.0.5 to fix [CVE-2026-26007](https://github.com/advisories/GHSA-r6ph-v2qm-q3c2).
  * Upgraded `protobuf` to 5.29.6 to fix [CVE-2026-0994](https://github.com/advisories/GHSA-7gcm-g887-7qv7).
  * Upgraded `PyJWT` to 2.12.0 to fix [CVE-2026-32597](https://github.com/advisories/GHSA-752w-5fwx-jx9f).
  * Upgraded `urllib3` to 2.6.3 to fix [CVE-2026-21441](https://github.com/advisories/GHSA-38jv-5279-wg99).
  * Upgraded `aiohttp` to 3.13.3 to fix [CVE-2025-69223](https://github.com/advisories/GHSA-6mq8-rvhq-8wgg).
  * Upgraded `azure-core` to 1.38.0 to fix [CVE-2026-21226](https://github.com/advisories/GHSA-jm66-cg57-jjv5).
  * Upgraded `fickling` to 0.1.9 to fix [GHSA-5hwf-rc88-82xm](https://github.com/advisories/GHSA-5hwf-rc88-82xm), [GHSA-mxhj-88fx-4pcv](https://github.com/advisories/GHSA-mxhj-88fx-4pcv), and [GHSA-wccx-j62j-r448](https://github.com/advisories/GHSA-wccx-j62j-r448).
  * Upgraded `litellm` to 1.83.7 to fix the following CVEs: [CVE-2026-35030](https://github.com/advisories/GHSA-jjhc-v7c2-5hh6), [CVE-2026-35029](https://github.com/advisories/GHSA-53mr-6c8q-9789), [GHSA-69x8-hrgq-fjj8](https://github.com/advisories/GHSA-69x8-hrgq-fjj8), [GHSA-xqmj-j6mv-4862](https://github.com/advisories/GHSA-xqmj-j6mv-4862).
  * Upgraded `pillow` to 12.2.0 to fix [CVE-2026-25990](https://github.com/advisories/GHSA-cfh3-3jmp-rvhc) and [CVE-2026-40192](https://github.com/advisories/GHSA-whj4-6x5x-4v2j).
  * Upgraded `pyasn1` to 0.6.3 to fix [CVE-2026-23490](https://github.com/advisories/GHSA-63vm-454h-vhhq) and [CVE-2026-30922](https://github.com/advisories/GHSA-jr27-m4p2-rc6r).
  * Upgraded `python-multipart` to 0.0.22 to fix [CVE-2026-24486](https://github.com/advisories/GHSA-wp53-j4wj-2cfg).

  ## Fixes

  * Reduced the risk of deadlocks during artifact creation, deletion, and garbage collection.
  * Fixed a memory caching bug in the storage engine that could significantly under-report the actual heap retained by cache entries.
  * Fixed bugs where if the workspace or section's **Panel settings** configure a specific x-axis or step size, users could not override the settings for an individual panel. Instead, the workspace or section setting forced its value onto the panel.
  * Improved the display of long aliases and tags in the W\&B App.
  * Fixed a bug that caused jumbled characters to appear when filtering runs by regular expression.
  * Fixed an infinite loading page when opening a link to a nonexistent run group.
  * Fixed a bug where discarding a report draft would redirect to the list of reports instead of the source report.
  * Fixed a bug where a resized Markdown panel in a report would render with fixed height instead of the configured height.
  * In Weave, replaced unsafe dynamic types with safer handling instead of surfacing raw errors in some paths.
  * Fixed a bug where Weave traces could not be written to a project that was deleted and then subsequently restored.
</Update>

<Update label="v0.80.1" description="May 6, 2026">
  **May 6, 2026**

  ## Fixes

  * Fixed a critical authorization bug that allowed exposure of artifacts to unauthorized users.
  * Fixed a critical security bug in validation of serialized payloads, allowing execution of code.
  * Fixed a high-severity authorization bug that allowed any non-viewer organization member to become organization administrator.
  * Fixed a high-severity authorization bug where testing a generic webhook integration required only entity read permission while creating or updating webhooks required admin. Testing a webhook now requires admin, consistent with other webhook configuration actions.
  * Fixed a high-severity bug that allowed cross-project modification of automation triggers.
  * Fixed a high-severity bug in user input sanitization of labels and URLs that would have allowed cross-site scripting.
  * Fixed OIDC sign-in for deployments that use PKCE with a confidential client secret by reverting an earlier change that omitted the client secret on PKCE token requests. When a client secret is configured, the server no longer adds redundant client IDs to token URL parameters, which avoids conflicts with the `Authorization` header.
  * Fixed SCIM list fields for Microsoft Entra ID: the API returns empty JSON arrays instead of `null`, which restores connection tests and group flows.
  * Improved the reliability of SCIM Push operations sent to Okta:
    * Fixed a bug where a SCIM Group Push operation could fail if the Okta group name is different from the W\&B team name. These operations are now sent as pathless `PATCH` operations. **This is a breaking change for integrations that parse SCIM User JSON.** See [Breaking changes](#breaking-changes).
    * Added guardrails to prevent inadvertently removing all members from a team if a SCIM Group Push operation did not include all members.
    * Improved serialization of SCIM User objects in the payload sent to Okta. The payload now better aligns with what Okta expects.
    * Fixed a bug that could lead to incorrect casing of email addresses and names in a SCIM User object.
  * Fixed a bug when using SCIM to create a team with bring your own bucket (BYOB) on Dedicated Cloud when the request omits organization context.
  * Fixed a bug that could lead to incorrect run history in a forked run when using the parquet history store with the query executor. For example, the history of a forked run could include incorrect merges at the live store boundary or incorrect step metadata.
  * Improved line plot performance for projects with very large run counts by skipping expensive total-count queries. The legend shows a warning if the UI cannot tell whether additional runs match the panel filters, and these warnings may be less accurate.
  * Fixed incorrect redirects from the organization **Usage Alerts** page.
  * Fixed a JSON serialization bug in Weave when writing artifact metadata to the ClickHouse search index.
  * Fixed a bug in the Weave trace server where migrations could select an incorrect database engine.

  ## Breaking changes

  This release includes a breaking change for integrations that parse SCIM User JSON:

  * In Dedicated Cloud and Self-Managed v0.80.1+, responses from `/scim/Users` (including `GET` user, `GET` users, and `PATCH` responses that return a User) serialize `emails` as a JSON array of objects using lowercase field names (`value`, `primary`, and optional `type` or `display`), matching SCIM 2.0.
  * Deployments on older releases return `emails` as a single JSON object with PascalCase keys (`Value`, `Primary`, and similar).

  If your code reads `emails` from SCIM *responses*, treat `emails` as an array and read the primary entry (or the first element).

  Request bodies for creating or updating users already used the array form and are unchanged. The `list-users` filter `emails.value eq "..."` is also unchanged.
</Update>

<Update label="v0.80.0" description="April 28, 2026">
  **April 28, 2026**

  W\&B 0.80.0 includes new features to help you efficiently observe and analyze your experiments, bug fixes, and security improvements. The updated home page lets you view, search, and filter recent activity across your organization. Automation based on a run metric’s z-score and side-by-side media comparison are generally available. Table panels, parallel coordinate plots, and scatter plots have also been improved. Interacting with and sharing artifact lineage graphs is more efficient. To help streamline provisioning for enterprise deployments, SCIM API now supports managing Models seats and Weave roles.

  ## Support and end of life

  * W\&B Server v0.68 has reached end of life.

  Refer to [Release policies and processes](/release-notes/release-policies). For assistance or questions, contact [support](mailto:support@wandb.com).

  ## Features

  * [Compare mode](/models/app/features/panels/media#compare-mode) for media panels is generally available. Compare mode lets you select 2 to 4 images or videos from any combination of runs, steps, or indices and view them in a single grid, without downloading files or switching tabs.

    <Frame>
      <img alt="Media panel compare mode grid" />
    </Frame>

  * The W\&B App’s home page has been improved.

    * View, search, or filter recent activity, both your own and that of your colleagues.
    * Access recent reports directly in the home page.
    * Starred projects are no longer displayed on the home page, but appear at the top of the Projects page and in the left navigation.

    <Frame>
      <img alt="Home page with recent activity" />
    </Frame>

  * Programmatically pin selected config keys to the top of the run’s Overview tab with the new method `run.pin_config_keys()`. Pinned keys appear in a new **References** section above **Notes**, where you can see the run’s most important details at a glance, like hyperparameters, external links, or identifiers. Each key you pin appears separately with its value, and links are clickable. Requires [W\&B SDK v0.26.0](/release-notes/sdk-releases#0-26-0)+.

    <Frame>
      <img alt="Pinned config keys in References section" />
    </Frame>

  * Triggering an automation based on a metric’s z-score is generally available. A z-score measures how many standard deviations the value is from the mean for that metric across a configurable window of runs in the project. See [Automation events and scopes](/models/automations/automation-events#run-metrics-z-score-change).

  * When hovering over a run in the runs list, the relevant line is now highlighted in parallel coordinate plots.

  * Scatter plots support new display options related to the frontier line, which is a line connecting the highest and lowest Y-axis values so far. Configure these options in the **Annotations** tab of the plot’s settings.
    * **Show run labels**: Optionally display a label with the run name or group name.
    * **Dim non-frontier points**: Optionally reduce the weight of non-frontier points.

  * You can now play audio in a table panel while it is downloading.

  * For manually-added custom panels, position overrides are now preserved across reloads.

  * Access release notes for your deployment’s current version from within the W\&B App. In the top navigation, click **?** to open the **Resources & Help** menu, then click **Release Notes**.

  * Improved Lineage DAG navigation: you can now filter lineage by artifact type and upstream/downstream hops. Filtered or hidden nodes now remain visible as placeholders, which should make it easier to understand graph structure while iterating on filters.

  * When you share an artifact’s lineage view by URL, the recipient now sees the exact same graph configuration, including filters, hidden nodes, manual positions, viewports, and zoom levels.

  ### Weave

  * A new **Scorer dashboard** view in **Project overview** pages shows pass/fail rates and histogram distributions per scorer, a filterable list of examples, and per-example detail with trial scores.
    <img alt="Video demo of scorer dashboards" />
  * Time-binned **Feedback** and **Score** charts on the **Monitors** page show numeric trends boolean counts, filtered by monitor.
  * Playground now supports streaming evaluation results. When you run an evaluation from the Evaluation Playground, the compare view appears immediately with one column per model. Each column shows real data when that model’s evaluation has completed successfully, a skeleton when it’s still running, or shows that an evaluation failed. Results stream in as each evaluation completes instead of waiting for all of them. A progress bar updates with per-call status polling.
  * Create and manage classifier monitors from the **Monitors** page with suggested classifier cards.
  * Edit signal prompts in place in the **Monitors** page. Click the gear icon, then click a signal's pencil icon.
  * In the **Calls** table, user IDs now resolve to usernames.

  ## Security

  * Assign Models seats or Weave roles using [SCIM](/platform/hosting/iam/automate_iam).
  * When [fetching audit logs](/platform/hosting/monitoring-usage/audit-logging#fetch-audit-logs), the `startDate` parameter controls the newest logs to fetch.
  * Organization admins can disable GraphQL introspection using the environment variable `GORILLA_GRAPHQL_DISABLE_INTROSPECTION` in both **Dedicated Cloud** and **Self-Managed**. As an alternative on Self-Managed, you can set the Gorilla configuration field `graphql-disable-introspection` instead. See [Configure environment variables](/platform/hosting/env-vars). **Client applications need [W\&B SDK v0.26.0](/release-notes/sdk-releases#0-26-0)+** against deployments with introspection already disabled.
  * Organization admins on **Dedicated Cloud** and **Self-Managed** can require authentication for all GraphQL requests by setting `GRAPHQL_REJECT_UNAUTHED_REQUESTS=true` on the API service. The setting is opt-in and does not change behavior unless enabled. See [Configure environment variables](/platform/hosting/env-vars).
  * Fixed a bug where certain structured run filters were not consistently escaped when building SQL, which could allow a malicious filter shape to interfere with query generation beyond what an authorized client should be able to express. Filter handling now applies the same column escaping used elsewhere so filter keys cannot alter query structure.
  * Fixed a bug where run-queue operations tied to sweeps and direct queue deletes could, in specific cases, act on queues or queue items outside the caller’s intended scope. Authorization now verifies queue access and item ownership before enqueueing from sweeps or deleting queue entries.
  * Fixed a bug where run metadata key suggestions could incorporate unsanitized user input into database queries. Key paths are now constrained so only safe, expected column references feed the suggestion path.
  * Fixed a bug where the service-account creation API could accept an organization identifier that did not match the team the caller administers, allowing a mismatched org association. The server now rejects inconsistent organization input and derives organization membership from the entity’s actual organization.
  * Fixed a bug where dequeueing from a queue could treat a queue as an internal “public” launch queue based on naming alone, while enqueueing used stricter checks, and where a caller-supplied job author could be honored on push. Dequeue now uses the same public-queue validation as enqueue, and job author is aligned with the authenticated user so internal credential injection cannot be triggered for spoofed queues or forged authors.
  * Fixed a bug where custom LLM provider settings could target disallowed hosts or include unsafe HTTP headers when the trace server loads provider configuration. Base URLs and headers are now validated so providers cannot be pointed at sensitive metadata-style endpoints or carry blocked header patterns.
  * Fixed a bug where file paths supplied for in-memory trace artifacts could escape the intended temporary workspace during deserialization. Paths are now sanitized so artifact files are confined to the expected directory layout.
  * Fixed a bug where the evaluate-model worker could process unsafe custom serialized payloads during evaluation jobs. Those payloads are now rejected before deserialization, consistent with hardened paths elsewhere in the evaluation stack.

  ## Fixes

  * Table panels now default to being rendered as summary tables, with a button to switch to a stepper view if the table was logged across steps. This improves load times for workspaces with table panels and mitigates an incompatibility when viewing run history for a resumed run.
  * Fixed a bug that prevented the sweeps table from loading more than 500 sweeps.
  * The column editor now includes a dedicated **Show/Hide** button when you hover over a column.
  * The run ID (the value of `run.name`) is now included in fuzzy search when searching for runs.
  * Improved performance of loading audio in table panels.
</Update>

<Update label="v0.79.2" description="April 10, 2026">
  ## Fixes

  * Fixed a bug that could delay ingestion of metrics.
</Update>

<Update label="v0.79.1" description="April 1, 2026">
  ## Fixes

  * Fixed a bug where new runs, especially on new projects, would result in a 404 `run not found` error.
  * Fixed a bug that could prevent querying audit logs for a deployment that had more than 1000 days of audit logs.
</Update>

<Update label="v0.79.0" description="March 24, 2026">
  W\&B Server 0.79.0 adds zooming and panning for images, improved management of service accounts and API keys, and performance improvements. Weave monitors now support scoring all operations, and the Weave UI adds enhanced management of prompt tags and aliases, chat view for Claude agents, filtering by emoji feedback reactions or note contents, and more.

  ## Support and end of life

  * W\&B Server v0.67 has reached end of life.

  Refer to [Release policies and processes](/release-notes/release-policies). For assistance or questions, contact [support](mailto:support@wandb.com).

  ## Security

  * When creating a service account, you are now automatically guided to create an API key for it.
  * Confirmation is now required to delete a service account or an API key from the W\&B App.
  * The backend now escapes column names in filters to help prevent SQL injection.
  * Email addresses that contain [diacritical marks](https://en.wikipedia.org/wiki/Combining_Diacritical_Marks) are now blocked from user registration or login.

  ## Features

  * **Pan and zoom on an image** to inspect fine-grained image details directly in W\&B's full-screen image modal. Open any image in full-screen, then:

    * Use UI controls or keyboard shortcuts to set your zoom level, reset to 100%, or zoom to fit.
    * Click and drag to pan when zoomed in.

    <Frame>
      <img alt="Pan and zoom controls in the full-screen image modal" />
    </Frame>
  * For pinned and baseline runs, [summary metric deltas](/models/runs/compare-runs#summary-metric-deltas) show how each run compares to the baseline. To turn on semantic coloring for quick visual reference, you can set the Metric directionality for a column. This feature is now generally available.
    <Frame>
      <img alt="Summary metric deltas in runs table comparing to baseline" />
    </Frame>
  * The run selector and Runs list have been enhanced:
    * Added support for filtering by Git commit.
    * For runs that specify config parameters, added support for the `CONTAINS` operator.
  * Added defensive caching for project fields to reduce database load.

  ### Registry

  * Added new rate limits for artifact creation to reduce the risk of overloading the system.

  ### Weave

  <Note>For deployments using Weave, Clickhouse v0.25.8+ is required before upgrading to v0.78+.</Note>

  * Weave now supports chat view for Claude agents.
  * **Traces** and **Evaluations** pages now support filtering by note contents and emoji feedback reactions.
  * You can now manage prompt tags and aliases in the **Prompts** tab of the **Assets** page.
  * Tables on the **Calls** and **Traces** pages now support drag-and-drop reordering of columns.
  * Weave monitors now support scoring across all operations.

  ## Fixes

  * Fixed floating point rounding errors in backend smoothing.
  * Fixed a bug where typing in expression inputs caused the cursor to jump.
  * Fixed a bug where actual-sized images could overlap in gallery mode.
  * Fixed misalignment in bounding boxes with long captions.
  * Removed panel and hover overlay drop shadows.
  * Panel search results now include custom charts.
</Update>

<Update label="v0.78.2" description="March 10, 2026">
  <Note>For deployments using Weave, Clickhouse v0.25.8+ is required before upgrading to v0.78.</Note>

  ## Fixes

  Fixes a regression of a previously-addressed critical security fix that allows unauthorized access to an artifact. This regression is present in v0.78.0 and v0.78.1. Do not install or upgrade to these versions. Instead, install v0.78.2, which includes the fix.
</Update>

<Update label="v0.78.1" description="March 5, 2026">
  <Warning>
    This release is impacted by a critical security issue that allows unauthorized access to an artifact. Do not install or upgrade to this version. Instead, install [v0.78.2](#v0-78-2), which includes the fix.
  </Warning>

  ## Fixes

  * Fixed a bug where forking from a parent run that logged no data could fail.
  * Fixed a UI bug where the user's preference for light or dark mode was not used.
  * Performance improvements and bug fixes for history reads from Parquet.
</Update>

<Update label="v0.78.0" description="March 3, 2026">
  W\&B Server 0.78.0 strengthens security and access defaults, adds tools to compare experiments and control costs more efficiently, and more.

  <Note>For deployments using Weave, Clickhouse v0.25.8+ is required before upgrading to v0.78.</Note>

  * In **Models**, create multi-metric plots efficiently with regular expressions, pin important runs, compare experiments against a baseline, and streamline workspaces using enhanced workspace templates.
  * **Registry** includes enhanced search, redesigned cards, clearer and more responsive lineage graphs, and more.
  * In **Weave**, the **Trace Analytics** dashboard helps you spot bottlenecks and budget drivers from the Project Overview page. Saved views now support customized Traces, Evaluations, and Leaderboards. Weave Playground adds support for LoRA (Low-Rank Adaptation) and new models.

  <Warning>
    This release is impacted by a critical security issue that allows unauthorized access to an artifact. Do not install or upgrade to this version. Instead, install [v0.78.2](#v0-78-2), which includes the fix.
  </Warning>

  ## Support and end of life

  * v0.66.0 and below have reached end of life.

  Refer to [Release policies and processes](/release-notes/release-policies). For assistance or questions, contact [support](mailto:support@wandb.com).

  ## Security

  * A W\&B organization or deployment can now have a maximum of 100 Personal API Keys, 100 organization-level API keys per user, and 100 API keys per Service Account.
  * On the Enterprise plan, when you [invite a member](/platform/hosting/iam/access-management/manage-organization#invite-a-user) to the organization, they are assigned the **Member** role by default. On other plans, the default role is **Admin**.
  * When inviting a new user to a team who is not currently a member of the organization, their initial role is now **Member**.
  * The initial role for a user [invited](/platform/hosting/iam/access-management/manage-organization#invite-a-user) to an organization or team now displays in the invitation table.

  ## Features

  * Use [pinned and baseline runs](/models/runs/compare-runs) to keep track of important runs and efficiently evaluate model experiments.

    * Pin up to 6 runs to keep them visible in the workspace and at the top of the runs list. If you have a baseline run, you can pin up to 5 runs because the baseline is implicitly pinned.
    * Specify a baseline run as your reference point for comparisons. The baseline run is always visible in the workspace and at the top of the runs list.

    In the runs table, summary metric deltas (preview) show how each run compares to the baseline. This is available by invitation only. Contact \[mailto:[support@wandb.com](mailto:support@wandb.com)] for details.

    <Frame>
      <img alt="Summary metric deltas in runs table comparing to baseline" />
    </Frame>

    In line plots, the baseline appears with visually distinct styling to help with comparison.

    <Frame>
      <img alt="Baseline run with distinct styling in line plot" />
    </Frame>

  * [Create panels with multiple metrics](/models/app/features/panels/line-plot#multi-metric-line-plot) in bulk to quickly organize your workspace and spot patterns across large metric sets. For example, you can overlay gradient metrics across model layers or position specific layers side-by-side. [Watch a video demonstration](https://www.loom.com/embed/ebbb1ab3f858473cb24420bf3abf7166).

  * [Workspace templates](/models/app/features/cascade-settings#workspace-layout-options) have new settings. You can now configure your workspace template to hide empty sections, sort panels in a section alphabetically, or adjust how automatic panel sorting in a section works. See [Workspace layout options](/models/app/features/cascade-settings#workspace-layout-options).

  * The **Sweeps** table now includes a column with the **Sweep ID**.

  * Admins can now customize usage and spend alerts for your organization. View usage and spending and set up email notifications for storage, tracked hours, Weave, and Training in your organization settings under the new **Usage & Alerts** tab. See [Plan usage](/platform/app/settings-page/billing-settings#plan-usage).

  ### Registry

  * The Registry UI is faster to navigate and easier to scan.

    * Improved type-ahead search, redesigned Registry and Collection cards, and new Registry icons help visually distinguish assets using smart ranking initially, with AI ranking coming soon.
    * Registry icons are initially auto-generated and can be customized by Registry Admins at any time.

    <Frame>
      <img alt="Registry icons and redesigned cards" />
    </Frame>

    * In lineage graphs, when you hover over a node, it is highlighted and other nodes are dimmed.
    * You can now click the name of a collection to quickly view its artifact versions.
    * For convenience, you can now star a registry from its **Settings** page.

  ### Weave

  * The **Project Overview** page now includes a **Trace Analytics** dashboard with request counts, latency percentiles (P50/P95/P99), token usage by model, and cost breakdown. Spot which models are burning through your budget, or find the slow requests dragging down your P95. Click a time bucket to jump straight to the relevant traces, to quickly identify the cause of a latency spike.

    <Frame>
      <img alt="Trace Analytics dashboard on Project Overview" />
    </Frame>

  * Claude Opus 4.6 is now available in **Weave Playground**.

  * You can now re-order columns in **Traces** and **Evaluations** tables.

    * Drag and drop leaf columns directly in the table or move entire sections around from the column management menu.
    * Persist your changes in a [saved view](/weave/guides/tools/saved-views). View a video demonstration.

    <Frame>
      <img alt="Video: re-ordering columns in Traces and Evaluations tables" />
    </Frame>

  * You can now save a customized [Leaderboard](/weave/guides/evaluation/dynamic_leaderboards) in a saved view.

  * You can now evaluate audio and image content in your traces using [monitors](/weave/guides/evaluation/monitors).

  * Aggregated OpenTelemetry token counts now display in the trace detail sidebar for [OpenTelemetry traces](/weave/guides/tracking/otel#send-opentelemetry-traces-to-weave).

  * For models like Gemini that provide thinking tokens, they now display in **Token Usage** in the Weave UI. Thinking tokens are internal tokens used by LLMs to process complex reasoning steps, calculations, and self-reflection before generating a final answer.

  ## Fixes

  * Fixed a bug where a hidden team's reports could be shared.
  * Fixed a UI bug where copying text with newlines could truncate the selection.
  * When the Sweeps agent claims a preempted run for re-execution, the training script now has up to 60 minutes to start up and initialize the run before the run is sent to a different agent, increased from 60 seconds.
  * Fixed a race condition that could prevent a preempted Sweeps run from being enqueued for re-execution.
  * Fixed a bug where a Sweep could fail if its name started with a digit.
</Update>

<Update label="v0.77.1" description="January 28, 2026">
  **January 28, 2026**

  ## Changes

  * In **Self-Managed**, you can now add Weave panels to a W\&B workspace and reference W\&B Artifacts in a Weave trace. This functionality was previously available in **Dedicated Cloud** only. See [Use Weave with W\&B training runs](/weave/guides/tools/weave-in-workspaces).

  ## Fixes

  * Fixed a bug where a registry version query can return duplicates when there is an alias filter.
</Update>

<Update label="v0.77.0" description="January 21, 2026">
  **January 21, 2026**

  W\&B 0.77 introduces security and usability improvements designed to keep your organization safe and productive.

  This release includes security enhancements to API keys, streamlined navigation with shared configuration between single-run and multi-run workspaces, new line plot shading options, and general availability of media panel sync and looping, plus key quality-of-life improvements and bug fixes.

  ## Support and end of life

  * W\&B Server v0.65 has reached end of life.

  Refer to [Release policies and processes](/release-notes/release-policies). For assistance or questions, contact [support](mailto:support@wandb.com).

  ## Upcoming changes

  * Coming soon, the Registry UI will be faster to navigate and easier to scan.

    * Improved type-ahead search, redesigned Registry and Collection cards, and new Registry icons help visually distinguish assets using smart ranking initially, with AI ranking coming soon.
    * Registry icons will be auto-generated initially and can be customized by Registry Admins at any time.

    These updates are expected in the next release. Contact [support](mailto:support@wandb.com) or your AISE to learn more.

    <Frame>
      <img alt="Animation demonstrating Registry UI improvements" />
    </Frame>

  ## Security

  * We've rolled out updates to API keys across W\&B to improve security and key management for both personal and service accounts. These changes align with modern security best practices while giving teams more flexibility and safer automation.

    **What's important to know**

    * API keys are shown only once, and you must immediately copy and save after creation.
    * Existing keys can't be copied anymore, and you must generate new keys.
    * Rename or delete keys anytime.
    * Service accounts now support multiple API keys.

    See [Sign up and create an API key](/models/quickstart#sign-up-and-create-an-api-key).

  ## Features

  * **Updated workspace navigation.** We've aligned multi-run and single-run workspace views, so when you drill into a single run, you'll see the same layout and configuration of panels as you've configured for the multi-run view, including any custom charts, layouts, and settings. This helps you stay in context and avoid re-work around configuring your views. See this [video](https://www.loom.com/share/df232057aa2d4ee78026a4f9cf400499?sid=da3661eb-e8cd-4aa5-a666-db2c4091b02b) to understand the changes, including to the location of navigation buttons.
    <Frame>
      <img alt="Screenshot showing updated workspace navigation" />
    </Frame>
  * Section-level [video playback sync and looping](/models/app/features/panels/media#sync) are generally available. By default, videos in a section play in sync.
  * In full-fidelity line plots for grouped runs, you can now configure the shaded area to show **Min/Max**, **Standard Deviation**, **Standard Error**, or **None**. See [Sampling](/models/app/features/panels/line-plot/sampling).
  * In a parallel coordinate plot, you can now **Cmd+Click** (macOS) or **Ctrl+Click** (Linux / Windows) on a line to open the associated run details page in a new tab for further inspection. Not yet supported for grouped plots.
    <Frame>
      <img alt="Animation demonstrating clicking a line in a parallel coordinate plot to open run details" />
    </Frame>
  * To make setting up an automation more efficient, the app now suggests metrics on which to base it.

  ### Registry

  * [Artifact lineage graphs](/models/artifacts/explore-and-traverse-an-artifact-graph) in Registry now support drag-and-drop repositioning and hiding of items.
  * When viewing an artifact version, you can now create a new registry and link the artifact version to it in a single operation.

  ### Weave

  * Weave onboarding pages in the empty state in the W\&B App now include TypeScript example code alongside the Python example code.
  * Weave call logs now support continuous scrolling by default. Previously, you could view only the most recent 10,000 lines of logs.
  * Bar charts and radar charts in the [Evaluation](/weave/guides/core-types/evaluations) comparison page now support resizing and displaying in full-screen view, and when you hover over an area of the chart, a tooltip displays details.
  * [Weave Playground](https://wandb.ai/site/playground/) includes new models:
    * GPT 5.2
    * Grok 4.1
    * Opus 4.5
  * You can now create and save a new prompt from [Weave Playground](https://wandb.ai/site/playground/).

  ## Fixes

  * Fixed a bug where a run's **Notes** text field was too narrow when editing. Now, the text box uses the same width when viewing and editing.
  * Fixed a bug where the resolution of images in a PDF export of a W\&B Report was too low.
  * Fixed a bug that could prevent line charts in a Report from updating in sync with run selector changes.
  * Fixed a bug where outliers in a Report appeared for line plots without an option to hide them.
  * Fixed a bug where line plots with multiple runsets in a Report could show incorrect data for runs that shared the same name across runsets. Runs are now uniquely keyed by runset and name.

  ### Weave

  * Fixed a bug where the Weave UI in Dedicated Cloud showed links to Serverless Inference, which is not available in Dedicated Cloud.
  * Fixed a bug where bar charts in the [Evaluation](/weave/guides/core-types/evaluations) comparison page could be resized too narrowly if the details drawer is open.
  * Fixed a UI bug where the cursor could disappear from the visible portion of the screen when typing in [Weave Playground](https://wandb.ai/site/playground/).
  * Fixed a bug where you could not edit template variables when loading a call in [Weave Playground](https://wandb.ai/site/playground/).
  * Fixed a bug where the operation selector was not being set when navigating from the [Evaluation](/weave/guides/core-types/evaluations) overview to the detailed view of a child trace. This made it difficult to tell which trace was selected, and caused some columns to appear empty.
  * Fixed a bug where clicking **Delete** on an [Evaluation](/weave/guides/core-types/evaluations) comparison view page did not actually delete the comparison view.
  * Fixed a bug where [Evaluation](/weave/guides/core-types/evaluations) results with a custom type could display a blank `CellValue` field instead of the custom type.
  * Fixed a bug where left-pinned cells could appear transparent in **runs as rows** display mode.
</Update>

<Update label="v0.76.3" description="December 17, 2025">
  **December 17, 2025**

  ## Fixes

  * Fixed a display bug that could occur on macOS when scroll bars are configured to always display. If a workspace has a [Query panel](/models/app/features/panels/query-panels), zooming out in the workspace could trigger a resize loop where the query panel table size oscillates several times a second.
  * Fixed a bug where Serverless Inference erroneously appeared in the Dedicated Cloud UI. Inference is not available in Dedicated Cloud.
</Update>

<Update label="v0.76.2" description="December 10, 2025">
  **December 10, 2025**

  ## Fixes

  * Replaced inaccurate usage cards in the **Organization Dashboard** page with a link to the **Usage** page. This was a UI bug only, with no impact on billing.
  * Fixed a bug that could lead to a situation where Parquet reads were turned off but exports were turned on.
  * Fixed a nil pointer dereference when manually upserting a run without any run config in the pending state.
</Update>

<Update label="v0.76.1" description="December 4, 2025">
  **December 4, 2025**

  ## Fixes

  * Fixed a bug where importing a line plot from a workspace to a report could get stuck if extreme outliers were hidden in the plot.
  * Fixed a bug that could prevent a table panel from resizing correctly if its height was reduced.
  * Fixed a bug where mask overlays could overflow an image with a caption.
  * Fixed a bug where a run note's edit box's size was too narrow for comfortable editing. Now the edit box uses the same dimensions as the text in display mode.
  * Fixed a bug where clicking the **Service Accounts** tab in the Organization Dashboard could cause the page to crash.
  * Fixed a caching bug where the wrong run's logs could display when viewing run logs and switching from one run to another.
  * When a client cancels a GraphQL operation, it is now correctly reported as a `4xx` HTTP client error rather than a `5xx` HTTP server error.
</Update>

<Update label="v0.76.0" description="November 20, 2025">
  **November 20, 2025**

  W\&B v0.76 delivers a major boost to experiment insight and productivity, introducing smarter media tools, smoother workflows, and powerful new interfaces. Multi-video sync with autoplay and looping enables quick, side-by-side evaluation, while simplified workspace navigation keeps you moving effortlessly between views. Plus, the new LEET terminal UI brings fast, real-time run insights directly to your command line.

  ## Support and end of life

  * W\&B Server v0.63 has reached end of life.

  ## Features

  * **Multi-video Sync**: Play and control multiple video panels together for qualitative evaluation and comparison of training runs, analysis of model behavior across experiments, and evaluation of multimodal model checkpoints.

    With synchronized video playback, you can play, pause, scrub, and adjust the speed of multiple videos in sync, enabling true side-by-side comparison for quick visual QA to evaluate style, timing, motion, and more.

    <Frame>
      <img alt="Animation demonstrating video playback sync" />
    </Frame>

    To try it, go to **Settings** → **Media.** Currently available in Dedicated Cloud only.

  * **Autoplay and loop** media in video panels in a workspace or section. Currently available in Dedicated Cloud only.

  * Now in Dedicated Cloud, updates to workspace navigation simplify moving between multi-run and single-run workspaces and ensuring view changes are saved between both ([quick demo video](https://www.loom.com/share/df232057aa2d4ee78026a4f9cf400499?sid=da3661eb-e8cd-4aa5-a666-db2c4091b02b)).

    * Single-run and multi-run saved views now use shared configuration details to reduce repetitive setup tasks.
    * Existing single-run saved views are removed, and you will need to recreate the ones you still need.

    <Frame>
      <img alt="Screenshot showing navigation updates for single-run views" />
    </Frame>

    These updates are expected in W\&B Self-Managed in January 2026.

  * You can now copy a run’s name or path from the run selector menu. This update also fixed a bug where a copied run name would include the run name twice.

    <Frame>
      <img alt="Screenshot demonstrating copying a run's name or path" />
    </Frame>

  * [W\&B Lightweight Experiment Exploration Tool (LEET)](/models/ref/cli/wandb-beta/wandb-beta-leet) (Beta) is a fast terminal UI for observing your ML training runs, including stats, metrics, and system health. It reads and visualizes live log files to provide a fast, customizable, browser-free experience in real time.

    LEET presents an interactive, three-pane dashboard that updates live as your run progresses.

    <Frame>
      <iframe />
    </Frame>

    To try it, install [W\&B SDK version 0.23.0](/models/ref/python) or later.

  * We’ve recently released [Workspaces and Reports SDK v0.20.0](/models/ref/wandb_workspaces) v0.20.0, which includes the ability to pin sections and columns programmatically. See [Install Workspaces And Reports API](/models/ref/wandb_workspaces#installation-and-setup).

  ## Fixes

  * Min and max values now display on grouped line charts to help with comparisons.
  * Reports now truncate long media panel titles for cleaner presentation.
  * [Semantic legends](/models/runs/color-code-runs#set-a-configuration-key) now allow editing the upper bounds of buckets.
  * Fixed a bug when switching semantic colors from discrete to continuous values.
  * Fixed a bug that could cause flickering when resizing charts in full-screen mode.
  * Fixed a bug that could cause rendering issues when the y axis is set to `_step.`
  * Fixed a bug that could prevent the media step slider from updating to show the latest incoming data if it is set to the maximum value.
  * Fixed a bug where image smoothing settings were not always applied to image panels with smoothing turned on.
  * Fixed a bug where media panels in grid view could take up too much vertical space, causing excessive scrolling.
  * Fixed minor display issues when rendering and refreshing automations.
  * Fixed minor UI bugs when linking artifacts, including scrolling and count rendering issues.
  * Fixed a bug that could prevent non-ASCII characters from displaying in HTML table media panels.
  * Updated the **Edit run name** action to make it more clear whether you are updating the run's display name for the workspace or the underlying run name across the project.
</Update>

<Update label="v0.75.2" description="November 5, 2025">
  * Fixed a bug where a team service account's API key was not visible to team Admin users who were also organization Admins.
  * Fixed a bug where the (min, max) range shown in a tooltip for a plot with multiple run groups could show the same values for all groups.
  * Fixed a bug where Parquet files for deleted runs were not completely garbage collected and the error `delete parquet history does not currently support partial deletes` was logged repeatedly.
</Update>

<Update label="v0.75.0" description="October 28, 2025">
  W\&B v0.75 enhances Registry with a new Restricted Viewer role, the ability to star registries, and programmatic Registry access control over SCIM. The new automation execution history view helps you to understand and debug your automations. Visualize config-driven effects with semantic coloring by config properties such as hyperparameters, view full-fidelity plots for system metrics, and more.

  ## Support and end of life

  * W\&B Server v0.61 has reached end of life.

  Refer to [Release policies and processes](/release-notes/release-policies). For assistance or questions, contact [support](mailto:support@wandb.com).

  ## Upcoming changes

  In the next W\&B Server release, we'll roll out an update to workspace navigation that makes moving between multi-run and single-run workspaces easier—see this [preview video](https://www.loom.com/share/df232057aa2d4ee78026a4f9cf400499?sid=da3661eb-e8cd-4aa5-a666-db2c4091b02b).

  Part of this change is included in Server v0.75: the left sidebar now indicates whether you are viewing the **Project** or a single **Run**. This upcoming navigation change coincides with a unification of single-run and multi-run saved views, so that changes to plots and layouts in one carry over to the other to save repetitive configuration. As part of this change, existing single-run saved views will be lost, and you will need to recreate the ones you still need.

  <img alt="Upcoming navigation sidebar update" />

  ## Security

  * You can configure the [federated identity](/platform/hosting/iam/identity_federation) audience by setting the environment variable `FEDERATED_AUTH_AUDIENCES` to a comma-separated list of audiences. The audience specifies allowable recipients for the identity token.

  ## Features

  * **Registry access control updates:**
    * **Programmatic registry access control:** [Registry roles](/models/registry/configure_registry#details-about-registry-roles) are now included in the SCIM schema. While [creating](/platform/hosting/iam/scim#create-user) or updating a user, you can configure their Registry roles, and Registry roles are returned when you [retrieve a user](/platform/hosting/iam/scim#get-user).
    * **Restrict Registry access:** Where you don't want to grant full Registry access, you can now grant a new Registry role, [Restricted Viewer](/models/registry/configure_registry). This role provides read-only access to registry artifacts without the ability to create, update, or delete collections, automations, or other registry resources. This is useful for sensitive or regulated registry content.
    * [**Expanded Registry access:**](https://wandb.ai/wandb_fc/product-announcements-fc/reports/Registry-for-W-B-Models-viewers-Faster-discovery-cleaner-handoffs--VmlldzoxNDc4MzM5Mg) Now, any user with the Models **Viewer** role is also a full Registry **Member**, so they can browse, compare, and use models and artifacts by default. A Models **Viewer** can be assigned a different Registry role, like **Restricted Viewer**, **Viewer**, or **Admin**. This unlocks full Registry value for everyone in your organization.

  * **Starred registries:** To help you navigate Registry as you scale, you can now star registries. From the Registry landing page, hover over a registry's card, then click the star outline. From an individual registry page, click the star outline at the top of the page. Starred registries appear at the top of the Registry landing page in alphabetical order.

    <img alt="Starred registries" />

  * **Automation execution history:** View execution history for your project or registry [Automations](/models/automations/automation-events) from the **Automation history** tab. All automation runs display in reverse chronological order, with status and key details at a glance. To help you understand and debug automations, a detailed view shows the execution's conditions, actions, and results.

  * In the **Organization dashboard**, you can now invite multiple users to your organization at once by specifying a comma-separated list of email addresses, and you can optionally add the invited users to teams at the same time.

  * **Semantic legend updates:**
    * Semantic legends for runs now support coloring plots based on config values such as hyperparameters. Visualize config-driven effects by coloring runs based on a chosen hyperparameter like learning rate, batch size, or model family to visually spot config-driven effects.

      <img alt="Semantic legend config coloring" />

    * You can now configure semantic legends when viewing another user's workspace.

  * The new full-screen image browser simplifies inspection and comparison of logged images. Previously available only in **Dedicated Cloud**, the feature is now also available in **Self-Managed**.
    * **True full-Screen view:** Inspect image details more easily with edge-to-edge full-screen viewing. No more cramped panels.
    * **Step slider in full-screen view:** Move through steps efficiently with an embedded slider to see how outputs evolve.
    * **Quickly move through runs or steps:** In full-screen view, use the keyboard **left / right arrow** keys to move through runs, and use **Cmd + left / right arrow** (macOS) or **Ctrl + left / right arrow** (Windows / Linux) to move through steps of the same image.

  * **Full-fidelity rendering for system metrics:** Charts for system metrics, like GPU, CPU, or memory, have a new detailed rendering mode that preserves fine-grained signal for deep inspection and analysis.

  * Run history export to Parquet has been improved. Now run history export is asynchronous by default in all deployments. Also, when exporting run history, if a run has not been logged in 7 or more days and after history about the run has been completely exported, history about the run is automatically dropped from MySQL.

  ## Fixes

  * You can now toggle the **Show original** line plot setting for all plots in the workspace or a section.
  * Fixed a bug in Reports where pinning all columns in the run selector could break the report's layout.
  * Fixed a bug where a pinned column could be expanded beyond the boundaries of the run selector.
  * Fixed a bug where changing the run name while also editing the **Notes** field could revert the run name change.
  * Fixed a bug where incorrect values could appear when plotting grouped runs with the X axis set to **Relative Wall Time**.
  * Fixed a bug where group pagination controls were missing.
  * Fixed a bug where a report could not be published if BYOB storage is configured with S3-compatible storage configured with a bucket subpath. This impacts CoreWeave AI Object Storage, Amazon S3 storage, and S3-compatible storage like MinIO.
  * Fixed a bug that could prevent a system metrics chart from loading if smoothing is turned on.
  * Fixed a bug that could prevent a chart with multiple Y axes from loading if smoothing is turned on.
  * In full-fidelity mode, line plot tooltips now show min and max bounds.
  * The hover highlight interaction has been improved for grouped run plots.
  * Fixed a bug that could prevent a grouped plot from rendering if the X-axis name is a substring of the Y axis name.
  * Fixed a bug that could cause hidden or deselected runs to appear as gray points instead of being hidden.
  * Fixed a bug that could prevent a media panel from opening in full screen if you would need to scroll to view it.
  * To help you visualize spikes in line plots with dense comparisons more clearly, the min and max shading in line plots has been improved.
  * To efficiently recover from deep tweaks or experiments, you can now revert the X axis in the line plot editor.
  * Fixed a bug where the access type for a core registry could be set incorrectly if the registry was created during artifact linking. This bug could cause membership management issues and prevent the registry from functioning.
</Update>

<Update label="v0.74.1" description="October 2, 2025">
  * Fixed a bug that could prevent an organization admin from accessing the Organization Dashboard.
  * Fixed a bug introduced in 0.73.0 that allowed deletion of previously exported Parquet files.
  * Fixed a bug that could allow deleting an artifact file in Parquet after it has successfully committed.
  * Fixed a performance regression introduced in v0.73 when querying system metrics from MySQL.
  * Fixed a bug that could prevent editing of a project with no owner.
  * Historical metric histograms no longer display in summary pages by default. Instead, details about each metric display in summary pages, and histograms are available as workspace panels.
</Update>

<Update label="v0.74.0" description="October 8, 2025">
  W\&B v0.74 delivers SCIM API improvements for enterprise identity management, including automated group sync with Okta, streamlined user role configuration, and more. In Dedicated Cloud, a powerful new full-screen image browser helps you inspect and compare images within and between runs. Weave enhancements include in-Playground LLM judge evaluations, better visibility into multi-turn Thread tool calls, DSPy improvements, and a new integration with the Verifiers library for Reinforcement Learning workflows.

  ## Support and end of life

  * W\&B Server v0.60 has reached end of life.

  ## Features

  * The SCIM API has been enhanced:
    * Provisioning and deprovisioning operations are now more compatible with Okta.
    * In Multi-tenant Cloud deployments, you can now configure the organization role and teams when you [create a user](/platform/hosting/iam/scim#create-user). This aligns with the endpoint's behavior in Dedicated Cloud and Self-Managed.
    * In Dedicated Cloud and Self-Managed deployments, you can now update a user's username or email address. See [Update user email](/platform/hosting/iam/scim#update-user-email) and [Update user display name](/platform/hosting/iam/scim#update-user-display-name). Not applicable to Multi-tenant Cloud deployments.
    * You can now [search and retrieve a user](/platform/hosting/iam/scim#get-user) by email and username.
    * Okta groups are now synced automatically using [ETags](/platform/hosting/iam/scim#etag-support). For details about SCIM groups, see [Group resource](/platform/hosting/iam/scim#group-resource).
  * When [exporting the list of users](/platform/hosting/monitoring-usage/org_dashboard#export-user-details) in your organization, the CSV file now includes columns for **Models seats** and **Weave access**.
  * In **W\&B Dedicated Cloud**, the new full-screen image browser simplifies inspection and comparison of logged images.
    * **True full-screen view.** Inspect image details more easily with edge-to-edge full-screen viewing. No more cramped panels.
    * **Step slider in full-screen view.** Move through steps efficiently with an embedded slider to see how outputs evolve.
    * **Quickly toggle between runs using the keyboard.** In full-screen view, use the keyboard **left / right arrow** keys to move between runs, and use **Cmd + left / right arrow** (macOS) or **Ctrl + left / right arrow** (Windows / Linux) to move between steps of the same image.

  ### Weave

  * From the **Evaluations** sidebar tab, you can now update prompt values and run LLM judge evaluations to test the changes. Try it out in the [Playground](/weave/guides/tools/playground).
  * You can now integrate Weave into your Reinforcement Learning (RL) training workflows and [view the traces and evaluations](/weave/guides/tools/weave-in-workspaces) captured during training in the W\&B Workspace, together with run metrics panels.
  * Weave now includes an integration with [Verifiers](https://verifiers.readthedocs.io/en/latest/), a library of modular components for creating Reinforcement Learning (RL) environments and training LLM agents. Environments built with Verifiers can serve as LLM evaluations, synthetic data pipelines, agent harnesses for OpenAI-compatible endpoints, and RL training. [Learn more](/weave/guides/integrations/verifiers).
  * From the **Threads** sidebar tab, you can now view tool calls for a multi-turn Thread together with its message history.
  * A Team's **Project** page now shows the number of Weave traces for each project together with data about the project's runs.
  * The [DSPy integration](/weave/guides/integrations/dspy) has been improved. For details, see [`wandb/weave` #5184](https://github.com/wandb/weave/pull/5184).
  * If your Weave code calls `wandb.init()`, it no longer needs to explicitly call `weave.init()`.
  * New convenience method [`delete_all_object_versions`](/weave/reference/python-sdk/trace/weave_client).
  * The output from an OpenAI streaming endpoint call now includes a new field, `time_to_first_token`, expressed in milliseconds.

  ## Fixes

  * Fixed display problems in bounding boxes with captions.
  * When grouping runs in the Runs Table, you can now expand the `null` group to see the runs with null values for the grouping key.
  * Increased running average smoothing parameter maximum to **1000**.
  * Fixed a bug where clicking a run name could open an invalid URL if a runset included runs from multiple projects.
  * Fixed text display issues in dark mode.
  * Fixed a permission bug when viewing a public report that included Query Panels.
  * Fixed a UI bug that could prevent code from being copied from a report.
  * Fixed a bug in the order of min and max values in a line plot tooltip.
  * Fixed a bug that could prevent a grouped plot from rendering if the X-axis key was a substring of the Y-axis key.
</Update>

<Update label="v0.73.1" description="September 10, 2025">
  ## Fixes

  * Fixed a bug introduced in v0.73.0, where resizing or reordering workspace panels was not always reflected in saved views.
  * Fixed a bug in the Registry UI where paging through the list of registries could fail if an organization has hundreds of registries.
  * Fixed a bug where bounding boxes did not always display for media that was logged with bounding boxes.
  * Fixed a bug where export tasks could get stuck in a pending state and never execute.
  * Fixed a bug where `wandb verify` could fail when downloading files with special characters in their names.
  * Fixed a performance regression when fetching system events, especially when plotting a large number of system events in a chart. This regression was introduced in v0.73.0.
</Update>

<Update label="v0.73.0" description="September 10, 2025">
  W\&B v0.73 marks a significant milestone in the migration to W\&B Registry from team-level Model Registry; most organizations will be migrated automatically during the upgrade. UI improvements include colorblind-safe run color palettes, continuous color options metric-based run coloring, and smoother panel reorganization. This release also provides service principal authentication for BYOB (Bring Your Own Bucket) setups with CoreWeave AI Object Storage, along with improved console log handling and additional enhancements.

  ## Upcoming changes

  * In an upcoming release, we will align multi-run and single-run workspace views, so when you drill into a single run, you'll see the same layout and configuration of panels as you've configured for the multi-run view, including any custom charts, layouts, and settings. The goal is help you stay in context and avoid re-work around configuring your views. However, this will remove the ability to customize unique single-run views, isolated from the rest of the workspace. Reach out to your W\&B team or [support](mailto:support@wandb.ai) for any questions about the migration.
  * Over the next several releases, we will continue migrating existing team-level Model Registry to W\&B Registry. For details and a reminder about the timeline, see the [W\&B Registry GA announcement](#registry_ga_announcement) below.

  ## Known issues

  In v0.73.0, resizing or reordering workspace panels may not be reflected in saved views. This bug is fixed in v0.73.1. If you experience this issue, upgrade to v0.73.1, which includes the fix. If you can't upgrade, contact [support](mailto:support@wandb.ai) for a workaround.

  ## Features

  * <a aria-label="W&B Registry GA announcement" />We're migrating existing team-level Model Registry to W\&B Registry in stages over the next several releases. **No action is needed on your end; W\&B will automatically copy Model Registry assets for each team to a corresponding automatically-created team-level private registry.** Existing references to model collections in your old team-level Model Registry will still work.

    1. Server v0.73 (this release) includes an opt-out environment variable that will migrate the team's Model Registry automatically during the upgrade. Migration takes between 5 and 30 minutes for most organizations. During the migration, linking artifacts to either the old Model Registry or the new Registry will fail with an error. **Artifact logging and retrieval will not be affected**.
    2. In Server v0.74 (end of September), we will remove the opt-out flag and migration will happen during the upgrade.

    We're optimizing for a seamless upgrade to a powerful new experience. Contact your W\&B account team or [support](mailto:support@wandb.ai) if you have any questions about the migration.
  * Added a copy button to the [Run Logs console page](/models/app/console-logs) and improved paste behavior.
  * W\&B now offers two new colorblind-safe color palettes for runs. Set these in the **Runs** section of the [Workspace settings](/models/app/features/cascade-settings/#workspace-settings).
  * When [coloring runs by metrics or numeric configs](/models/runs/color-code-runs), you can now choose from five new continuous color palettes.
  * During a [W\&B run](/models/runs), [CoreWeave Mission Control](https://www.coreweave.com/mission-control) monitors your compute infrastructure. This integration is now available on Dedicated Cloud.
  * BYOB with CoreWeave AI Object Storage now supports authentication with service principals in Multi-tenant Cloud, Dedicated Cloud, and Self-Managed deployments. This helps you to avoid using static credentials in your deployment. Learn more at [Bring your own bucket (BYOB)](/platform/hosting/data-security/secure-storage-connector).

  ### Weave

  * You can now group sessions or conversations across multiple traces with [Weave Threads](/weave/guides/tracking/threads).
  * The **Trace view**'s new [Graph view](/weave/guides/tracking/trace-tree#graph-view) visualizes the call tree structure as a graph.
  * The **Trace view** includes new specialized views for [documents](/weave/guides/core-types/media#documents) retrieved with ChromaDB and Langchain improves handling of schemas such as chats. In **Trace view**, click a document to open its detailed view in a drawer to the right. Learn more at [Navigate the trace view](/weave/guides/tracking/trace-tree).
  * You can now attach [HTML](/weave/guides/core-types/media#html) to a trace as a media type.
  * From the **Assets** tab, you can now create and edit prompts. Click **Assets**, then in the navigation, click **Prompts**. Click **New prompt** or click the name of an existing prompt to view its details. From there, click the pencil icon to edit and republish the prompt.

  ## Fixes

  * Improved the smoothness and accuracy of [dragging and dropping panels](/models/app/features/panels/#move-a-panel) within a section and between sections.
  * In [full screen panel view](/models/app/features/panels/#view-a-panel-in-full-screen-mode), the **Esc** key now returns you to the workspace.
  * Fixed a bug that could cause a panel in a workspace to zoom unexpectedly after returning from full screen.
  * In [plot settings](/models/app/features/panels/line-plot/#edit-line-plot-settings), chart and legend settings have been consolidated to a single tab.
  * Fixed a bug where panel legends were ordered incorrectly after polling updates.
  * Improved the formatting of panels with long exponential Y-axis values and those with a large number of significant digits.
  * [Overlays for panels](/models/app/features/panels/media/#overlays) without matching runs are now hidden.
  * Fixed a bug where clicking on a run's tag in a workspace's **Runs** tab or **Runs** list did not filter by the tag.
  * Fixed a bug that could allow a user to bypass the max runs limit.
  * When you toggle runs on or off, W\&B no longer reloads the media for other runs in the panel.
  * Fixed a bug that could cause a report to crash when expanding or collapsing a section after pressing **Tab**.
  * Fixed a bug on mobile displays where scrolling could cause logged videos to automatically play in full screen.

  ## Other changes

  * The custom grid layout was removed from W\&B Workspaces with Server release v0.73.0. Previously, custom grid layout allowed you to adjust the size of individual panels within a section. Now, resizing a panel in a workspace section resizes all panels in that section, whether you configure the workspace using the [W\&B App](/models/app/features/panels/#manage-a-sections-panels) or the [Workspace API](/models/ref/wandb_workspaces/workspaces#class-sectionlayoutsettings). Custom grid layout remains available in [W\&B Reports](/models/reports) and the [Reports API](/models/ref/wandb_workspaces/reports).
</Update>

<Update label="v0.72.0" description="August 6, 2025">
  W\&B 0.72 includes fixes for line plots and media panels, as well as improvements to Weave integrations for OpenTelemetry and Google Agent Development Kit.

  ## Support and end of life

  * W\&B Server v0.58 has reached end of life.

  ## Upcoming changes

  * We’re optimizing our rendering behavior in the workspace to provide faster loading in workspaces with many panels, as well as more responsive and accurate drag-and-drop of panels in workspaces. **As part of this, we plan to remove the “Custom grid” layout in workspace sections in the next (0.73) release.** Custom grids will remain available in Reports. Reach out to your W\&B team or [support](mailto:support@wandb.ai) for any questions or concerns.
  * In an upcoming release, we will align multi-run and single-run workspace views, so when you drill into a single run, you’ll see the same layout and configuration of panels as you’ve configured for the multi-run view, including any custom charts, layouts, and settings. The goal is help you stay in context and avoid re-work around configuring your views. However, this will remove the ability to customize unique single-run views, isolated from the rest of the workspace. Reach out to your W\&B team or [support](mailto:support@wandb.ai) for any questions about the migration.
  * Over the next several releases, we will continue migrating existing team-level Model Registry to W\&B Registry.

  ## Fixes

  * Reverted the line plot behavior change announced in 0.71.0. Line plots with only 1 point for every line will convert to bar charts even if settings are changed.
  * Fixed a bug that could cause bounding boxes to render outside the media in a media panel instead of overlaying the media.
  * Fixed a bug where the Google Agent Development Kit integration did not support Weave Threads.
  * Fixed a bug where the **Attributes** column was missing from Weave traces submitted through OpenTelemetry.
</Update>

<Update label="v0.71.0" description="July 29, 2025">
  With W\&B Server 0.71, Registry is turned on by default for all organizations. For more details about the staged Registry rollout, see the relevant note in Features. This release brings several improvements and fixes to workspaces: Keep media panels with a common step slider in sync, navigate through full-screen panels more efficiently, and find more details about a run’s total duration and time actively running in the Run Overview page. Automations have been enhanced, and you can now trigger an automation based on an artifact version’s tags.&#x20;

  ## Support and end of life

  * W\&B Server v0.57 has reached end of life.

  ## Upcoming changes

  * We’re optimizing our rendering behavior in the workspace to provide faster loading in workspaces with many panels, as well as more responsive and accurate drag-and-drop of panels in workspaces. **As part of this, we plan to remove the “Custom grid” layout in workspace sections in the next (0.72) release.** Custom grids will remain available in Reports. Reach out to your W\&B team or [support](mailto:support@wandb.ai) for any questions or concerns.
  * In an upcoming release, we will align multi-run and single-run workspace views, so when you drill into a single run, you’ll see the same layout and configuration of panels as you’ve configured for the multi-run view, including any custom charts, layouts, and settings. The goal is help you stay in context and avoid re-work around configuring your views. However, this will remove the ability to customize unique single-run views, isolated from the rest of the workspace. Reach out to your W\&B team or [support](mailto:support@wandb.ai) for any questions about the migration.
  * Over the next several releases, we will continue migrating existing team-level Model Registry to W\&B Registry. For details and a reminder about the timeline, see the [W\&B Registry GA announcement](#registry_ga_announcement) below.

  ## Features

  * <a aria-label="W&B Registry GA announcement" /> We're migrating existing team-level Model Registry to W\&B Registry in stages over the next several releases. **No action is needed on your end; W\&B will automatically copy Model Registry assets for each team to a corresponding automatically-created team-level private registry.**  Existing references to model collections in your old team-level Model Registry will still work.

    1. In Server v0.71 (this release), Registry is now turned on by default for all organizations.
    2. Server v0.73 (August) will include an opt-out environment variable that will execute this migration automatically during the upgrade. Migration takes between 5 and 30 minutes for most organizations. During the migration, linking artifacts to either the old team’s Model Registry or the new Registry will fail with an error. **Artifact logging and retrieval will not be affected**.
    3. In Server v0.74 (September), we will remove this flag and migration will happen during the upgrade.

    We’re optimizing for a seamless, minimally-inconveniencing upgrade to a powerful new experience. Reach out to your W\&B for any questions about the migration.
  * An automation can now be triggered by adding a tag to an artifact. See [Artifact events](/models/automations/automation-events/#artifact-events).
  * When viewing a panel in full screen, view the next or previous panel in the section using the **Previous** and **Next** buttons or the left and right arrow keys.
  * When viewing a media panel in full screen, move the step slider with **CMD + left or right arrow** keys (macOS) or **Ctrl + left or right arrow** keys (Windows / Linux).
  * You can now synchronize media panels with the same step slider key so that when you advance the step slider in one media panel, the step sliders for other panels also advance. Open the workspace or section settings, go to **Media**, click the **Sync** tab, then turn on **Sync slider by key (Step)**.
  * The Run Comparer includes a new **Meta** section with metadata about the run, such as the command, Python version, and GPU type.

  ### Weave

  * The **Trace plots** tool allows you to explore, visualize, and debug trace-level metrics like latency, cost, or tokens over time using custom interactive charts. [Learn more](/weave/guides/tracking/trace-plots).
  * **Online Evaluations**: Monitor your traces in Dedicated Cloud by attaching Monitors to your incoming traces. Monitors run in the background as LLM judges and score a subset of calls that you specify. Use Monitors to track production behavior, catch regressions, collect real-world production data, and more. [Learn more](/weave/guides/tracking/trace-plots).
  * Added [AutoGen](/weave/guides/integrations/autogen) and [LlamaIndex](/weave/guides/integrations/llamaindex) integrations. *AutoGen* is a framework from Microsoft for building AI agents and applications, with components for conversational AI, core multi-agent functionalities, and integrations with external services, and tools for no-code agent prototyping. *LlamaIndex* is a powerful framework for building LLM-driven applications like RAG systems, chatbots, and agents.
  * Improved Integrations with OpenAI, LangChain, ChromaDB, Verdict, including:
    * Document view for Langchain and ChromaDB.
    * Chat view rendering for LangChain.
    * A new export format for OpenAI.
  * Added support to the TypeScript SDK for creating and publishing prompts. [Learn more](/weave/guides/core-types/prompts).
  * The new `Content` class allows you safely to upload data of any MIME type, with automatic Base-64 encoding, automatic metadata extraction, and more.

  ## Fixes

  * A run’s **Overview** page now displays the run’s uptime and tracked hours. A run’s *uptime* represents the run’s total duration from start to finish, while the run’s *tracked hours* represent the duration of time that the run was actively running experiments or computations.
  * When you zoom into a line plot, then open it in full screen mode, the full screen plot now correctly maintains the same zoomed-in view.
  * Fixed a bug where updating the number of groups shown in a plot did not update the plot.
  * When **Show Original** is selected in line plots, the original unsmoothed line is now dimmer than the smoothed line.
  * Full fidelity line plots now show the number of runs in the plot.
  * Fixed a bug that could unexpectedly convert a plot that is explicitly set to a line plot to a bar plot when only one value is present per run.
  * X-axis expressions now support nested config fields.
  * The **Registry** link has been removed from the sidebar on the **Artifacts** page. To access Registry, click **Home**, then click **Registry**.
  * Line plots in a workspace or section can optionally show the bin range in the tooltip when you hover over a point. Open the workspace or section settings, click **Line plots**, go to the **Display preferences** tab, then turn on **Show bin range in tooltip**.
  * Fixed a bug when clicking a link in the **Registry** artifact lineage page would update the URL and parameters in the browser but not update the page contents. The UI now updates when the URL changes.
  * Fixed a bug in **Registry** where collection tables were sorted as strings instead of floats.
  * Fixed a bug in **Registry** where the lineage info drawer would always show the latest version for the selected artifact, even when a different version was selected.
  * Fixed a mobile display bug where the panel search bar was missing.
  * When you sort the run selector by any attribute with coloring by key on, the top runs in the plot are now colored correctly.
  * When using grouped runs, parallel coordinate plots now correctly show the group name.
  * **State** is now correctly shown by default in the Sweeps table.
  * Sweeps now show correctly when grouping by custom display names.
  * Fixed wrapping of long names for Sweeps and Reports.
  * Improved the copy and paste behavior for console logs. Line numbers are no longer copied, and timestamps now appear inline instead of on a new line.
</Update>

<Update label="v0.70.1" description="July 9, 2025">
  ## Fixes

  * Fixed a bug where the status of crashed runs was not updated to **Crashed** after an upgrade.
  * Fixed a bug where the **State** column was missing from the **Sweeps** Runs table.
  * Fixed a Weave bug where the browser for a Registry collection could display files from an incorrect version because the cached path for an artifact manifest file depended only on the artifact's version index.
  * Fixed a Weave bug that could prevent JSON with valid syntax from being parsed correctly.

  These bugs were introduced in v0.70.0.
</Update>

<Update label="v0.70.0" description="July 7, 2025">
  W\&B Server v0.70 includes features, enhancements, and performance improvements to help you gain and share insights more efficiently. For example:

  ## Support and end of life

  * W\&B Server v0.56 has reached end of life.

  ## Upcoming changes

  * In an upcoming release, we will align multi-run and single-run workspace views, so when you drill into a single run, you’ll see the same layout and configuration of panels as you’ve configured for the multi-run view, including any custom charts, layouts, and settings. The goal is help you stay in context and avoid re-work around configuring your views. However, this will remove the ability to customize unique single-run views, isolated from the rest of the workspace. Reach out to your W\&B team or [support](mailto:support@wandb.ai) for any questions about the migration.
  * Over the next several releases, we will migrate existing team-level Model Registry W\&B Registry. For details and a timeline, see the [W\&B Registry GA announcement](#registry_ga_announcement) below.

  ## Features

  * <a aria-label="W&B Registry GA announcement" /> **W\&B Registry is now generally available**! Registry offers improved asset diversity, access management, and scalability, making it a true hub for collaborating on AI initiatives across large, diverse teams.
    * Registry can be turned on today with the `ENABLE_REGISTRY_UI` environment variable or by contacting your W\&B support team.
    * We’re migrating existing team-level Model Registry assets to W\&B Registry in stages over the next several releases. **No action is needed on your end. W\&B will automatically copy Model Registry assets for each team to a corresponding automatically-created team-level private registry.**  Existing references to model collections in your old team-level Model Registry will still work.

      1. In Server v0.71, Registry will be on by default for all organizations.
      2. Server v0.73 (August) will contain an opt-out environment variable that will execute this migration automatically during the upgrade. Migration takes between 5 and 30 minutes for most organizations. During the migration, linking artifacts to either the old team’s Model Registry or the new Registry will fail with an error. \*\***Artifact logging and retrieval** **will not be affected**.
      3. In Server v0.74 (September), we will remove this flag and migration will happen during the upgrade.

      We’re optimizing for a seamless, minimally-inconveniencing upgrade to a powerful new experience. Reach out to your W\&B team or [support](mailto:support@wandb.ai) for any questions about the migration.
  * **CoreWeave infrastructure monitoring**: During a [W\&B Run](/models/runs) on CoreWeave infrastructure, [CoreWeave Mission Control](https://www.coreweave.com/mission-control) monitors your compute infrastructure, allowing for observation of infrastructure alerts such as GPU failures, thermal violations, and more. If an error occurs, CoreWeave sends that information to W\&B. W\&B populates infrastructure information onto your run's plots in your project's workspace. CoreWeave attempts to resolve some issues automatically, and W\&B surfaces that information in the run's page. W\&B provides a link to the SLURM job's Grafana dashboard for system-level details about the run. Learn more at [Visualize CoreWeave infrastructure alerts](/models/app/features/panels/line-plot).
  * **CoreWeave external file tracking**: You can now track external files stored in CoreWeave AI Object Storage with reference artifacts. Learn more at [Track external files](/models/artifacts/track-external-files).
  * **CoreWeave BYOB support**: Dedicated Cloud and Self-Managed now support CoreWeave AI Object Storage for Instance and Team level BYOB. CoreWeave Team level BYOB on Multi-tenant Cloud is coming soon. Learn more at [Bring your own bucket (BYOB)](/platform/hosting/data-security/secure-storage-connector).
  * **Bulk media settings**: Just like line plots, you can now manage all your media panel settings at once—across an entire workspace or a specific section. Easily configure media panels to display by epoch or arrange them into customized grid patterns without adjusting each panel separately. Individual panel settings override the global settings.
  * Use **Workspace templates** to quickly create workspaces using the same settings as an existing workspace. Currently, a workspace template can define custom line plot settings. Learn more at [Workspace templates](/models/track/workspaces/#workspace-templates).
  * With **incremental table logging**, you can log batches of rows to a table during a machine learning experiment. This is ideal for monitoring long-running jobs or when working with large tables that would be inefficient to log during the run for updates. Within the UI, the table is updated with new rows as they are logged, so you can view the latest data without having to wait for the entire run to finish. You can step through the increments to view the table at different points in time. Learn more at [Log tables](/models/track/log/log-tables/#adding-data-incrementally).

  ## Fixes

  * Added a banner at the top of the log viewer if only a subset of lines is shown.
  * Fixed a bug in the running average calculation at data boundaries.
  * Clarified x-axis unit labels for relative time plots.
  * Fixed a bug in plots with smoothing where the original line would still display when **Show Original** was unchecked.
  * Fixed a bug that could unexpectedly convert a grouped full-fidelity line plot with expressions into a bar chart.
  * Hovering on histograms now shows the x-axis step in a tooltip.
  * Fixed a bug where data exported from a panel could sort incorrectly. Exported data now sorts numerically.
  * In the run selector, you can now filter and sort on **updated at**.
  * Added right-handed system setting option to point cloud panels.
  * Trailing slashes are now stripped from full screen panel URLs.
  * Fixed a bug where a deleted saved view could reappear if you refreshed immediately after deleting it.
  * Fixed a display bug that caused a flash when reloading a page with dark mode turned on.
  * Fixed a bug that caused incorrect slider steps when using non-monotonic values.
  * Added a convenience checkbox and notification when moving runs between groups.
  * Fixed a bug where using the wildcard character `*` for the metric name in `wandb.define_metric()` would fail to match metrics with `/` in the name.
  * W\&B now no longer deletes source artifacts when collection portfolios are garbage collected.
  * Fixed a bug that incorrectly allowed typing input in **Allowed types** and **Registry visibility** selection boxes.
  * Collection cards now display the full artifact type name instead of truncating it.
  * Fixed a bug where clicking **Action History** in a Registry collection would incorrectly load the **Versions** view.
  * Registry now supports adding `job` type artifacts.
  * Lineage tiles are now wider, displaying more text before truncating.
  * Clarified text in project-level automation setup to refer to "artifacts" instead of the Registry terminology "collections".
  * The artifact browser now searches all artifacts when searching the artifact browser by name, rather than the previous limit of the first 500 artifacts.
</Update>


# Archived W&B Server releases
Source: https://docs.wandb.ai/release-notes/server-releases-archived

Browse archived release notes for older W&B Server versions that have reached end-of-life support status.

This page includes release notes for W\&B Server releases that have reached end of life and are no longer supported.

* For supported Server releases, see [W\&B Server release notes](/release-notes/server-releases).
* For more information about the W\&B Server release schedule and support, see [W\&B Server release policies](/release-notes/release-policies).

<Note>For details, see [Supported Server releases](https://docs.wandb.ai/release-notes/server-releases) and [Archived Server releases](https://docs.wandb.ai/release-notes/server-releases-archived). A W\&B Server release is supported for 12 months from its initial release date. As a reminder, customers using [Self-Managed](/platform/hosting/hosting-options/self-managed/) are responsible to upgrade to a supported release in time to maintain support.</Note>

<Update label="v0.69.1" description="June 10, 2025">
  ## Fixes

  * You can now set the initial run state when creating a run with `Run.create()` by setting the `state` parameter to `pending` or `running`.
  * Fixed a bug where clicking **Action History** incorrectly loaded the **Version** view.
  * Improved memory performance of the Parquet store service.
</Update>

<Update label="v0.69.0" description="May 28, 2025">
  ## Support and end of life

  * W\&B Server v0.56 has reached end of life.

  ## Upgrading

  To [upgrade](/platform/hosting/server-upgrade-process/#update-with-helm) to W\&B v0.69.x, you must use v0.31.4+ of the `operator-wandb` Helm chart. Otherwise, after the upgrade, the `weave-cache-clear` container can fail to start. Ensure that your deployment uses these values:

  ```yaml theme={null}
  chart:
    url: https://charts.wandb.ai
    name: operator-wandb
    version: 0.31.4
  ```

  If you have questions or are experiencing issues with an upgrade, contact [support](mailto:support@wandb.com).

  ## Features

  * You can now set a custom display name for a run directly in the workspace. Customized run names show up in all plots and tables but only in your workspace, with no impact on your teammates' views. This provides a clearer and cleaner view in your workspace, with no more labels like `*...v6-final-restart...`\* in every legend and plot.
  * When filtering or grouping runs, colors can sometimes overlap and become indistinct. The run selector's new **Randomize Colors** option reassigns random colors from the default palette to your current run selection or groups, helping to make the colors more distinguishable.
  * In line plots, you can now use **Cmd+Click** on a line to open a single-run view in a new tab.
  * Video media panels now provide more playback controls to play, pause, seek, view full screen, and adjust playback speed.
  * Settings for all types of media panels have been reorganized and improved.
  * You can now customize the point and background colors for point cloud panels.
  * Team-level and organization-level service accounts can now interact with Registry.
  * Improved Exponentially-weighted Moving Average (EMA) smoothing provides more reliable [smoothed lines](/models/app/features/panels/line-plot/smoothing) when operating on complete, unbinned data. In most cases, smoothing is handled at the back end for improved performance. This feature was in private preview in v0.68.x.

  ### Private preview

  Private preview features are available by invitation only. To request enrollment in a private preview, contact [support](mailto:support@wandb.com) or your AISE.

  * You can now color all of your runs based on a secondary metric, such as loss or custom efficiency metrics. This creates a clear gradient color scale across your runs in all plots, so you can spot patterns faster. [Watch a video demo](https://www.loom.com/share/c6ed484899324de991ef7147fd73785d).
  * [Personal workspace templates](/models/track/workspaces/#workspace-templates) allow you to save core line plot settings and automatically reapply them in new views. These settings include x-axis key, smoothing algorithm, smoothing factor, max number of lines, whether to use the run selector's grouping, and which aggregation to apply.

  ### Weave

  * [Saved views](/weave/guides/tools/saved-views) simplify team collaboration and allow you to persist filter and column settings.
  * PDFs and generic files are now supported.
  * The new [`EvaluationLogger` API](/weave/guides/evaluation/evaluation_logger) provides flexible imperative-style evaluation logging.
  * You can now import [human annotations](/weave/guides/tracking/feedback#add-human-annotations) into Weave datasets
  * [Playground](/weave/guides/tools/playground) now supports saved configurations and prompts.
  * Decorators are now supported in TypeScript.
  * Added support for [tracing generator functions](/weave/guides/tracking/tracing#trace-sync-%26-async-generator-functions).
  * The new [`dataset.add_rows`](/weave/reference/python-sdk/#method-add-rows) helper improves the efficiency of appending to an existing dataset.
  * To help you understand your usage, trace and object sizes are now shown through the UI.

  ## Performance

  * With [`wandb` SDK](/models/quickstart/#install-the-wandb-library-and-log-in) v0.19.11, artifacts now download 3-5x faster on average. For example, an artifact that previously downloaded at around 100 MB/sec may now download at 450 MB/sec or faster. Actual download speeds vary based on factors such as your network and storage infrastructure.
  * Improved caching on [Project](/models/track/project-page) and [User Settings](/platform/app/settings-page/user-settings) pages.

  ## Fixes

  * Improved the startup process for the `weave-cache-clear` container to ensure compatibility with Python virtual environments.
  * Added options for denser display of console logs.
  * Workspace loading screens are now more informative.
  * When adding a panel from a workspace to a report, the current project's reports are now shown first in the destination report list.
  * Fixed many cases where y-axes would over-round to a degree that caused duplicate values to display.
  * Fixed confusing behavior when entering invalid smoothing parameters.
  * Removed the **Partial Media** warning from media panels. This does not change the behavior of the media panels.
  * When adding a [run filter based on tags](/models/runs/filter-runs/#filter-runs-with-tags), the filter is now selected by default, as when filtering by other fields.
  * Removed the green bell icon that could appear on active runs in the run selector.
  * Removed the System page for individual runs.
  * The project description field now respects new lines.
  * Fixed URLs for legacy model registry collections.
  * Fixed a bug where the Netron viewer did not expand to fill all available space on the page.
  * When you click **Delete** on a project, the project name now displays in the confirmation modal.
</Update>

<Update label="v0.68.2" description="May 7, 2025">
  ## Fixes

  * You can now set the initial run state when creating a run with `Run.create()` by setting the `state` parameter to `pending` or `running`.
  * Fixed a bug where clicking **Action History** incorrectly loaded the **Version** view.
  * Improved memory performance of the Parquet store service.
</Update>

<Update label="v0.68.1" description="May 2, 2025">
  ## Fixes

  * Fixed a bug introduced in v0.68.0 that could prevent media from loading in media panels.
</Update>

<Update label="v0.68.0" description="April 29, 2025">
  <Note>
    v0.68.0 introduced a bug, fixed in [v0.68.1](#0_68_1), that could prevent media from loading in media panels. To avoid this bug, install or upgrade to a patch that contains the fix. If you need assistance, contact [support](mailto:support@wandb.com).
  </Note>

  ## Features

  * Registry admins can define and assign [*protected aliases*](/models/registry/aliases#protected-aliases) to represent key stages of your development pipeline. A protected alias can be assigned only by a registry admin. W\&B blocks other users from adding or removing protected aliases from versions in a registry using the API or UI.
  * You can now filter console logs based on a run's `x_label` value. During [distributed training](/models/track/log/distributed-training/#track-all-processes-to-a-single-run), this optional parameter tracks the node that logged the run.
  * You can now move runs between `Groups`, one by one or in bulk. Also, you can now create new `Groups` after the initial logging time.
  * Line plots now support **synchronized zooming** mode, where zooming to a given range on one plot automatically zooms into the same range on all other line plots with a common x-axis. Turn this on in the [workspace display settings for line plots](/models/app/features/panels/line-plot/#all-line-plots-in-a-workspace).
  * Line plots now support formatting custom metrics as timestamps. This is useful when synchronizing or uploading runs from a different system.
  * You can now slide through [media panels](/models/app/features/panels/media) using non-`_step` fields such as `epoch` or `train/global_step` (or anything else).
  * In Tables and plots in [Query Panels](/models/app/features/panels/query-panels) that use `runs` or `runs.history` expressions, a step slider allows you to step through the progress on your metrics, text, or media through the course of your runs. The slider supports stepping through non-`_step` metrics.
  * You can now customize [bar chart](/models/app/features/panels/bar-plot) labels using a font size control.

  ### Private preview

  Private preview features are available by invitation only. To request enrollment in a private preview, contact [support](mailto:support@wandb.com) or your AISE.

  * **Personal workspace templates** allow you to save your workspace setup so it is automatically applied to your new [projects](/models/track/project-page). Initially, you can configure certain line plot settings such as the default X axis metric, smoothing algorithm, and smoothing factor.
  * **Improved Exponentially-weighted Moving Average (EMA) smoothing** provides more reliable [smoothed lines](/models/app/features/panels/line-plot/smoothing) when operating on complete, unbinned data. In most cases, smoothing is handled at the back end for improved performance.

  ### Weave

  * Chat with fine-tuned models from within your W\&B instance. [Playground](/weave/guides/tools/playground) is now supported in Dedicated Cloud. Playground is a chat interface for comparing different LLMs on historical traces. Admins can add API keys to different model providers or hook up [custom hosted LLM providers](/weave/guides/tools/playground/#add-a-custom-provider) so your team can interact with them from within Weave.
  * Open Telemetry Support. Now you can log traces via OpenTelemetry (OTel). See [OpenTelemetry tracing](/weave/guides/tracking/otel?utm_source=beamer\&utm_medium=sidebar\&utm_campaign=OpenTelemetry-support-in-Weave\&utm_content=ctalink).
  * Weave [tracing](/weave/guides/tracking) has new framework integrations: CrewAI, OpenAI's Agent SDK, DSPy 2.x and Google's genai Python SDK.
  * Playground supports new [OpenAI models](/weave/guides/tools/playground/#openai): GPT‑4.1, GPT‑4.1 mini, and GPT‑4.1 nano.
  * Build labeled datasets directly from traces, with your annotations automatically converted into dataset columns. See [Dataset creation from traces](/weave/guides/core-types/datasets/#create-edit-and-delete-a-dataset-in-the-ui).

  ## Security

  * Registry admins can now designate a [service account](/platform/hosting/iam/service-accounts) in a registry as either a Registry Admin or a Member. Previously, the service account's role was always Registry Admin. See [Registry service account configuration](/models/registry/configure_registry).

  ## Performance

  * Improved the performance of many workspace interactions, particularly in large workspaces. For example, expanding sections and using the run selector are significantly more responsive.
  * Improved Fork and Rewind Performance.

    [Forking](/models/runs/forking) a run creates a new run that uses the same configuration as an existing run. Changes to the forked run do not the parent run, and vice versa. A pointer is maintained between the forked run and the parent. [Rewinding](/models/runs/rewind) a run lets you log new data from that point in time without losing the existing data.

    In projects with many nested forks, forking new runs is now much more efficient due to improvements in caching.

  ## Fixes

  * Fixed a bug that could prevent an organization service account from being added to new teams.
  * Fixed a bug that could cause hover marks to be missing for grouped lines.
  * Fixed a bug that could include invalid project names in the **Import** dropdown of a Report panel.
  * Fixed a display bug in the alignment of filters in the run selector.
  * Fixed a page crash when adding a timestamp **Within Last** filter
  * Fixed a bug that could prevent the X-axis from being set to **Wall Time** in global line plot settings.
  * Fixed a bug that could prevent image captions from appearing when they are logged to a Table.
  * Fixed a bug that could prevent sparse metrics from showing up in panels.
  * In **Run Overview** pages, the **Description** field is now named **Notes**.
</Update>

<Update label="v0.67.0" description="March 28, 2025">
  ## Features

  * In Reports, you can now give a run a custom display name per panel grid. This allows you to replace the run’s (often long and opaque) training-time name with one that is more meaningful to your audience. The report updates the name in all panel grids, helping you to explain your hard-won experimental insights to your colleagues in a concise and readable way. The original run name remain intact in the project, so doing this won’t disrupt your collaborators.
  * When you expand a panel in the workspace, it now opens in full screen mode with more space. In this view, line plots now render with more granular detail, using up 10,000 bins. The run selector appear next to the panel, letting you easily toggle, group, or filter runs in context.
  * From any panel, you can now copy a unique URL that links directly to that panel's full screen view. This makes it even easier to share a link to dig into interesting or pathological patterns in your plots.
  * Run Comparer is a powerful tool you can use to compare the configurations and key metrics of important runs alongside their loss curves. Run Comparer has been updated:
    * Faster to add a Run Comparer panel, as an expanded option in **Add Panels**.
    * By default, a Run Comparer panel takes up more space, so you can see the values right away.
    * Improved readability and legibility of a Run Comparer panel. You can use new controls to quickly change row and column sizes so you can read long or nested values.
    * You can copy any value in the panel to your clipboard with a single click.
    * You can search keys with regular expressions to quickly find exactly the subset of metrics you want to compare across. Your search history is saved to help you iterate efficiently between views.
  * Segmentation mask controls have been updated:
    * You can now toggle each mask type on or off in bulk, or toggle all masks or all images on or off.
    * You can now change each class’s assigned color, helping to avoid confusion if multiple classes use the same color.
  * When you open a media panel in full screen mode, you can now use the left or right arrows on your keyboard to step through the images, *without* first clicking on the step slider.
  * Media panels now color run names, matching the run selector. This makes it easier to associate a run’s media values with related metrics and plots.
  * In the run selector, you can now filter by whether a run has certain media key or not.
  * You can now move runs between groups in the W\&B App UI, and you can create new groups after the run is logged.
  * Automations can now be edited in the UI
  * An automation can now notify a Slack channel for artifact events. When creating an automation, select “Slack notification” for the Action type.
  * Registry now supports global search by default, allowing you to search across all registries by registry name, collection name, alias, or tag.
  * In Tables and Query panels that use the `runs` expression, you can use the new Runs History step slider and drop-down controls to view a table of metrics at each step of a run.
  * Playground in W\&B Weave supports new models: OpenAI's `gpt-4.5-preview` and Deepseek's `deepseek-chat` and `deepseek-reasoner`.
  * Weave tracing has two new agent framework integrations: CrewAI and OpenAI’s Agent SDK.
  * In the Weave UI, you can now build Datasets from traces. [Learn more](/weave/guides/core-types/datasets#create-edit-and-delete-a-dataset-in-the-ui)
  * The Weave Python SDK now provides a way to filter the inputs and outputs of your Weave data to ensure sensitive data does not leave your network perimeter. You can configure to redact sensitive data. Learn more: /weave/guides/tracking/redact-pii/
  * To streamline your experience, the System tab in the individual run workspace view will be removed in an upcoming release. View full information about system metrics in the System section of the workspace. For questions, contact [support@wandb.com](mailto:support@wandb.com).

  ## Security

  * `golang crypto` has been upgraded to v0.36.0.
  * `golang oauth2` has been upgraded to v0.28.0.
  * In Weave, `pyarrow` is now pinned to v17.0.0.

  ## Performance

  * Frontend updates significantly reduce workspace reload times by storing essential data in the browser cache across visits. The update optimizes loading of saved views, metric names, the run selector, run counts, W\&B’s configuration details, and the recomputation of workspace views.
  * Registry overview pages now load significantly faster.
  * Improved the performance of selecting metrics for the X, Y, or Z values in a scatter plot in a workspace with thousands of runs or hundreds of metrics.
  * Performance improvements to Weave evaluation logging.

  ## Fixes

  * Fixed a bug in Reports where following a link to a section in the report would not open to that section.
  * Improved the behavior of how Gaussian smoothing handles index reflection, matching SciPy's default "reflect" mode.
  * A Report comment link sent via email now opens directly to the comment.
  * Fixed a bug that could crash a workspace if a sweep takes longer than 2 billion compute seconds by changing the variable type for sweep compute seconds to `int64` rather than `int32`.
  * Fixed display bugs that could occur when a report included multiple run sets.
  * Fixed a bug where panels Quick Added to an alphabetically sorted section were sorted incorrectly.
  * Fixed a bug that generated malformed user invitation links.
</Update>

<Update label="v0.66.0" description="March 06, 2025">
  ## Features

  * In tables and query panels, columns you derive from other columns now persist, so you can use them for filtering or in query panel plots.

  ## Security

  * Limited the maximum depth for a GraphQL document to 20.
  * Upgraded pyarrow to v17.0.0.
</Update>

<Update label="v0.65.0" description="January 30, 2025">
  ## Features

  * From a registry's **Settings**, you can now update the owner to a different user with the Admin role. Select **Owner** from the user's **Role** menu.
  * You can now move a run to a different group in the same project. Hover over a run in the run list, click the **action (<Icon icon="ellipsis-vertical" />)** menu, and choose **Move to another group**.
  * You can now configure whether the **Log Scale setting** for line plots is enabled by default at the level of the workspace or section.
    * To configure the behavior for a workspace, click the **action (<Icon icon="ellipsis" />)** menu for the workspace, click **Line plots**, then toggle **Log scale** for the X or Y axis.
    * To configure the behavior for a section, click the gear icon for the section, then toggle **Log scale** for the X or Y axis.
</Update>

<Update label="v0.63.0" description="December 10, 2024">
  ## Features

  **[W\&B Weave](https://wandb.ai/site/weave/) is now generally available (GA) in Dedicated Cloud on AWS. Reach out to your W\&B team if your teams are looking to build Generative AI apps with confidence and putting those in production.**

  <Frame>
    ![Weave UI](https://github.com/user-attachments/assets/22786cbc-7d69-4505-b329-78cb87734d19)
  </Frame>

  The release includes the following additional updates:

  * W\&B Models now seamlessly integrates with ***Azure public cloud***. You could now create a Dedicated Cloud instance in an Azure region directly from your Azure subscription and manage it as an Azure ISV resource. [This integration is in private preview](https://wandb.ai/site/partners/azure).
  * Ability to assign x\_label, e.g. node-0, in run settings object to distinguish logs and metrics by label, e.g. node, in distributed runs. Enables grouping system metrics and console logs by label for visualization in the workspace.
  * ***Coming soon*** with a patch release this week, you will be able to use organization-level service accounts to automate your W\&B workloads across all teams in your instance. You would still be able to use existing team-level service accounts if you would like more control over the access scope of a service account.
    * Allow org-level service accounts to interact with Registry. Such service accounts can be invited to a registry using the invite modal and are displayed in the members table along with respective organization roles.

  ## Fixes

  * Fixed an issue where users creating custom roles including the `Create Artifact` permission were not able to log artifacts to a project.
  * Fixed the issue with metadata logging for files in instances that have subpath support configured for BYOB.
  * Block webhook deletion if used by organization registry automations.
</Update>

<Update label="v0.61.0" description="October 17, 2024">
  ## Features

  **This is a mini-feature and patch release, delivered at a different schedule than the monthly W\&B server major releases**

  * Organization admins can now configure Models seats and access control for both Models & [W\&B Weave](/weave/) in a seamless manner from their organization dashboard. This change allows for a efficient user management when [Weave](/weave/) is enabled for a Dedicated Cloud or Self-Managed instance.
    * [Weave](/weave/) pricing is consumption-based rather than based on number of seats used. Seat management only applies to the Models product.
  * You can now configure [access roles at the project level for team and restricted scoped projects](https://docs.wandb.ai/platform/hosting/iam/access-management/restricted-projects/). It allows assigning different access roles to a user within different projects in the same team, and thus adding another strong control to conform to enterprise governance needs.

  ## Fixes

  * Fixed an issue where underlying database schema changes as part of release upgrades could timeout during platform startup time.
  * Added more performance improvements to the underlying parquet store service, to further improve the chart loading times for users. Parquet store service is only available on Dedicated Cloud, and Self-Managed instances based on [W\&B kubernetes operator](https://docs.wandb.ai/platform/hosting/operator).
  * Addressed the high CPU utilization issue for the underlying parquet store service, to make the efficient chart loading more reliable for users. Parquet store service is only available on Dedicated Cloud, and Self-Managed instances based on [W\&B kubernetes operator](https://docs.wandb.ai/platform/hosting/operator).
</Update>

<Update label="v0.60.0" description="September 26, 2024">
  <Note>
    Due to a release versioning issue, 0.60.0 is the next major release after 0.58.0. The 0.59.0 was one of the patch releases for 0.58.0.
  </Note>

  ## Features

  * Final updates for 1.1.1 Compliance of Level AA 2.2 for Web Content Accessibility Guidelines (WCAG) standards.
  * W\&B can now turn off auto-version-upgrade for Self-Managed instances using the W\&B kubernetes operator. You can request this to your W\&B team.
    W\&B requires all instances to upgrade periodically to comply with the 6-month end-of-life period for each version. W\&B does not support versions older than 6 months.

  ## Fixes

  * Fixed a bug to allow instance admins on Dedicated Cloud and Self-Managed instances to access workspaces in personal entities.
  * SCIM Groups and Users GET endpoints now filter out service accounts from the responses. Only non service account users are now returned by those endpoints.
  * Fixed a user management bug by removing the ability of team admins to simultaneously delete a user from the overall instance while deleting them from a team. Instance or Org admins are responsible to delete a user from the overall instance / organization.

  ## Performance improvements

  * Reduced the latency when adding a panel by up to 90% in workspaces with many metrics.
  * Improved the reliability and performance of parquet exports to blob storage when runs are resumed often.
    Runs export to blob storage in parquet format is available on Dedicated Cloud and on Self-Managed instances that are enabled using the W\&B kubernetes operator.
</Update>

<Update label="v0.58.1" description="September 4, 2024">
  ## Features

  * W\&B now supports sub-path for **Secure storage connector i.e. Bring your own bucket** capability. You can now provide a sub-path when configuring a bucket at the instance or team level. This is only available for new bucket configurations and not for existing configured buckets.
  * W\&B-managed storage on newer Dedicated Cloud instances in Google Cloud & Azure will by default be encrypted with **W\&B managed cloud-native keys**. This is already available on AWS instances. Each instance storage is encrypted with a key unique to the instance. Until now, all instances on Google Cloud & Azure relied on default cloud provider-managed encryption keys.
  * Makes the fields in the run config and summary copyable on click.
  * If you're using W\&B kubernetes operator for a Self-Managed instance, you can now optionally use a custom CA for the controller manager.
  * We've modified the W\&B kubernetes operator to run in a non-root context by default, aligning with OpenShift's Security Context Constraints (SCCs). This change ensures smoother deployment of Self-Managed instances on OpenShift by adhering to its security policies.

  ## Fixes

  * Fixed an issue where exporting panels from a workspace to a report now correctly respects the panel search regex.
  * Fixed an issue where setting `GORILLA_DISABLE_PERSONAL_ENTITY` to `true` was not disabling users from creating projects and writing to existing projects in their personal entities.

  ## Performance improvements

  * We have significantly improved performance and stability for experiments with 100k+ logged points. On a Self-Managed instance, this is available if the deployment is managed using the W\&B kubernetes operator.
  * Fixed issue where saving changes in large workspaces would be very slow or fail.
  * Improved latency of opening workspace sections in large workspaces.
</Update>

<Update label="v0.57.2" description="July 24, 2024">
  ## Features

  **You can now use JWTs (JSON Web Tokens) to access your W\&B instance from the wandb SDK or CLI, using the identity federation capability. The feature is in preview.** Refer to [Identity federation](https://docs.wandb.ai/platform/hosting/iam/identity_federation) and reach out to your W\&B team for any questions.

  * New `Add to reports` drawer improvements for exporting Workspace panels into Reports.
  * Artifacts metadata filtering in the artifact project browser.
  * Pass in artifact metadata in webhook payload via `${artifact_metadata.KEY}`.
  * Added GPU memory usage panels to the RunSystemMetrics component, enhancing GPU metrics visualization for runs in the app frontend.
  * Mobile users now enjoy a much smoother, more intuitive Workspace experience.
  * If you're using W\&B Dedicated Cloud on Google Cloud or Azure, you can now enable private connectivity for your instance, thus ensuring that all traffic from your AI workloads and optionally browser clients only transit the cloud provider private network. Refer to [Private connectivity](https://docs.wandb.ai/platform/hosting/data-security/private-connectivity) and reach out to your W\&B team for any questions.
  * Team-level service accounts are now shown separately in a new tab in the team settings view. The service accounts are not listed in the Members tab anymore. Also, the API key is now hidden and can only be copied by team admins.
  * Dedicated Cloud is now available in Google Cloud's Seoul region.

  ## Fixes

  * Gaussian smoothing was extremely aggressive on many plots.
  * Fixed issue where pressing the `Ignore Outliers in Chart Scaling` button currently has no effect in the UI workspace.
  * Disallow inviting deactivated users to an organization.
  * Fixed an issue where users added to an instance using SCIM API could not onboard successfully.

  ## Performance improvements

  * Significantly improved performance when editing a panel's settings and applying the changes.
  * Improved the responsiveness of run visibility toggling in large workspaces.
  * Improved chart hovering and brushing performance on plots in large workspaces.
  * Reduced workspace memory usage and loading times in workspaces with many keys.
</Update>

<Update label="v0.56.0" description="June 29, 2024">
  <Note>
    Due to a release versioning issue, 0.56.0 is the next major release after 0.54.0. The 0.55.0 was a patch release for 0.54.0.
  </Note>

  ## Features

  * The new Full Fidelity line plot in W\&B Experiments enhances the visibility of training metrics by aggregating all data along the x-axis, displaying the minimum, maximum, and last value within each bucket (the last value draws the primary line), allowing users to easily spot outliers and zoom into high-fidelity details without downsampling loss.  [Learn more in our documentation](https://docs.wandb.ai/models/app/features/panels/line-plot/sampling).
  * You can now use [cross-cloud storage buckets for team-level BYOB (secure storage connector)](https://docs.wandb.ai/platform/hosting/data-security/secure-storage-connector#cross-cloud-or-s3-compatible-storage-for-team-level-byob) in Dedicated Cloud and Self-Managed instances. For example, in a W\&B instance on AWS, you can now configure Azure Blob Storage or Google Cloud Storage for team-level BYOB, and so on for each cross-cloud combination.
  * In the same vein, you can now use [S3-compatible storage buckets like MinIO for team-level BYOB (secure storage connector)](https://docs.wandb.ai/platform/hosting/data-security/secure-storage-connector#cross-cloud-or-s3-compatible-storage-for-team-level-byob) in Dedicated Cloud and Self-Managed instances. For example, in a W\&B instance on Google Cloud, you can configure a MinIO bucket hosted in cloud or on-prem for team-level BYOB.
  * Admins can now automate full deletion of users in their Dedicated Cloud or Self-Managed instances using the [SCIM API's DELETE User endpoint](https://docs.wandb.ai/platform/hosting/iam/scim#delete-user). The user deactivation operation has been reimplemented using the [PATCH User endpoint](https://docs.wandb.ai/platform/hosting/iam/scim#deactivate-user), along with the introduction of [user reactivation operation](https://docs.wandb.ai/platform/hosting/iam/scim#reactivate-user).
  * If you use the SCIM API, you will also see a couple of minor improvements:
    The API now has a more pertinent error message in case of authentication failures.
    Relevant endpoints now return the full name of a user in the SCIM User object if it's available.

  ## Fixes

  * The fix resolves an issue where deleting a search term from a runset in a report could delete the panel or cause the report to crash by ensuring proper handling of selected text during copy/paste operations.
  * The fix addresses a problem with indenting bulleted items in reports, which was caused by an upgrade of slate and an additional check in the normalization process for elements.
  * The fix resolves an issue where text could not be selected from a panel when the report was in edit mode.
  * The fix addresses an issue where copy-pasting an entire panel grid in a Report using command-c was broken.
  * The fix resolves an issue where report sharing with a magic link was broken when a team had the `Hide this team from all non-members` setting enabled.
  * The fix introduces proper handling for restricted projects by allowing only explicitly invited users to access them, and implementing permissions based on project members and team roles.
  * The fix allows instance admins to write to their own named workspaces, read other personal and shared workspaces, and write to shared views in private and public projects.
  * The fix resolves an issue where the report would crash when trying to edit filters due to an out-of-bounds filter index caused by skipping non-individual filters while keeping the index count incremental.
  * The fix addresses an issue where unselecting a runset caused media panels to crash in a report by ensuring only runs in enabled runsets are returned.
  * The fix resolves an issue where the parameter importance panel crashes on initial load due to a violation of hooks error caused by a change in the order of hooks.
  * The fix prevents chart data from being reloaded when scrolling down and then back up in small workspaces, enhancing performance and eliminating the feeling of slowness.
</Update>

<Update label="v0.54.0" description="May 24, 2024">
  ## Features

  * You can now configure Secure storage connector (BYOB) at team-level in Dedicated Cloud or Self-Managed instances on Microsoft Azure.
    Refer to [this Terraform Module](https://github.com/wandb/terraform-azurerm-wandb/tree/main/modules/secure_storage_connector) and reach out to your W\&B team for any questions.
  * Organization admins can now enforce privacy settings across all W\&B teams by setting those at the organization level, from within the `Settings` tab in the Organization Dashboard.
    W\&B recommends to notify team admins and other users before making such enforcement changes.
  * Enable direct lineage option for artifact lineage DAG
  * It's now possible to restrict Organization or Instance Admins from self-joining or adding themselves to a W\&B team, thus ensuring that only Data & AI personas have access to the projects within the teams.
    W\&B advises to exercise caution and understand all implications before enabling this setting. Reach out to your W\&B team for any questions.
  * Dedicated Cloud on AWS is now also available in the Seoul (S. Korea) region.

  ## Fixes

  * Fix issue where Reports where failing to load on Mobile.
  * Fix link to git diff file in run overview.
  * Fixed the intermittently occurring issue related to loading of Organization Dashboard for certain users.
</Update>

<Update label="v0.52.2" description="April 25, 2024">
  ## Features

  * You can now enforce username and full name for users in your organization, by using OIDC claims from your SSO provider. Reach out to your W\&B team or support if interested.
  * You can now disable use of personal projects in your organization to ensure that all projects are created within W\&B teams and governed using admin-enforced guidelines. Reach out to your W\&B team or support if interested.
  * Option to expand all versions in a cluster of runs or artifacts in the Artifacts Lineage DAG view.
  * UI improvements to Artifacts Lineage DAG - the type will now be visible for each entry in a cluster.

  ## Fixes

  * Added pagination to image panels in media banks, displaying up to 32 images per page with enhanced grid aesthetics and improved pagination controls, while introducing a workaround for potential offset inconsistencies.
  * Resolved an issue where tooltips on system charts were not displaying by enforcing the `isHovered` parameter, which is essential for the crosshair UI visibility.
  * Unset the max-width property for images within media panels, addressing unintended style constraints previously applied to all images.
  * Fixed broken config overrides in launch drawer.
  * Fixed Launch drawer's behavior when cloning from a run.
</Update>

<Update label="v0.51.0" description="March 20, 2024">
  ## Features

  * You can now save multiple views of any workspace by clicking "Save as a new view" in the **action (<Icon icon="ellipsis" />)** menu of the workspace bar.
    See [Saved workspace views](https://docs.wandb.ai/models/app/pages/workspaces#create-saved-workspace-views) to learn more.

    <Frame>
      ![Image showing saved views](https://github.com/wandb/server/assets/7208315/862a36ac-8ce4-49e7-8677-d87d54ab1e54)
    </Frame>
  * You can now [set a project's visibility scope to Restricted](https://docs.wandb.ai/platform/hosting/iam/access-management/restricted-projects/) if you want to collaborate on AI workflows related to sensitive or confidential data.
    When you create a **restricted project** within a team, you can add specific members from the team. Unlike other project visibility scopes, all members of a team do not get implicit access to a restricted project.
  * **Enhanced Run Overview page performance**: now 91% faster on load, with search functionality improved by 99.9%. Also enjoy RegEx search for Config and Summary data.
  * **New UX for Artifacts Lineage DAG** introduces clustering for 5+ nodes at the same level, preview window to examine a node's details, and a significant speedup in the graph's loading time.
  * The template variable values used for a run executed by launch, for example GPU type and quantity, are now shown on the queue's list of runs.  This makes it  easier to see which runs are requesting which resources.
  * Cloning a run with Launch now pre-selects the overrides, queue, and template variable values used by the cloned run.
  * Instance admins will now see a `Teams` tab in the organization dashboard. It can be used to join a specific team when needed, whether it's to monitor the team activity as per organizational guidelines or to help the team when team admins are not available.
  * SCIM User API now returns the `groups` attribute as part of the GET endpoint, which includes the id of the groups / teams a user is part of.
  * All Dedicated Cloud instances on Google Cloud are now managed using the new [W\&B Kubernetes Operator](https://docs.wandb.ai/platform/hosting/operator/). With that, the new Parquet Store service is also available.
    Parquet store allows performant & cost efficient storage of run history data in parquet format in the blob storage. Dedicated Cloud instances on AWS & Azure are already managed using the operator and include the parquet store.
  * Dedicated Cloud instances on AWS have been updated to use the latest version of the relational data storage, and the compute infrastructure has been upgraded to a newer generation with better performance.

  > **Advanced Notice**: We urge all customers who use Webhooks with Automations to add a valid A-record for their endpoints as we are going to disallow using IP address based Webhook URLs from the next release onwards. This is being done to protect against SSRF vulnerability and other related threat vectors.

  ## Fixes

  * Fixed issue where expressions tab was not rendering for line plots.
  * Use display name for sweeps when grouped by sweeps in charts and runs table.
  * Auto navigation to runs page when selecting job version.
</Update>

<Update label="v0.50.2" description="February 26, 2024">
  ## Feature

  * Add panel bank setting to auto-expand search results
  * Better visibility for run queue item issues
  * Dedicated Cloud customers on AWS can now use Privatelink to securely connect to their deployments.
    The feature is in private preview and will be part of an advanced pricing tier at GA. Reach out to your W\&B team if interested.
  * You can now automate user role assignment for organization or team scopes using the [SCIM role assignment API](https://docs.wandb.ai/platform/hosting/iam/scim/#assign-organization-level-role-to-user).
  * All Dedicated Cloud instances on AWS & Azure are now managed using the new W\&B Kubernetes Operator. With that, the new Parquet Store service is also available. The service allows for performant & cost efficient storage of run history data in parquet format in the blob storage. That in turn leads to faster loading of relevant history data in charts & plots that are used to evaluate the runs.
  * W\&B Kubernetes Operator and along with that the Parquet Store service are now available for use in Self-Managed instances. We encourage customers that already use Kubernetes to host W\&B, to reach out to their W\&B team on how they can use the operator. And we highly recommend others to migrate to Kubernetes in order to receive the latest performance improvements and new services in future via operator. We're happy to assist with planning such a migration.

  ## Fixes

  * Properly pass template variables through sweep scheduler
  * Scheduler polluting sweep yaml generator
  * Display user roles correctly on team members page when search or sort is applied
  * Org admins can again delete personal projects in their Dedicated Cloud or Self-Managed server instance
  * Add validation for SCIM GET groups API for pending users
</Update>

<Update label="v0.49.0" description="January 18, 2024">
  ## Features

  * Set a default TTL (time-to-live or scheduled deletion) policy for a team in the team settings page.
    Restrict setting or editing of a TTL policy to either of admin only or admin plus members.
  * Test and debug a webhook during webhook creation or after in the team settings UI.
    W\&B will send a dummy payload and display the receiving server's response.
  * View Automation properties in the View Details slider.
    This includes a summary of the triggering event and action, action configs, creation date, and a copy-able curl command to test webhook automations.
  * Replace agent heartbeat with last successful run time in launch overview.
  * Service accounts can now use the Report API to create reports.
  * Use the new role management API to automate managing the custom roles.
  * Enable Kubernetes Operator for Dedicated Cloud deployments on AWS.
  * Configure a non-conflicting IP address range for managed cache used in Dedicated Cloud deployments on Google Cloud.

  ## Fixes

  * Update the add runset button clickable area in reports
  * Show proper truncate grouping message
  * Prevent flashing of publish button in reports
  * Horizonal Rule get collapsed in report section
  * Add section button hidden in certain views
  * Allow things like semantic versioning in the plot as string
  * Remove requirements for quotes when using template variables in queue config definitions
  * Improve Launch queue sorting order
  * Don't auto-open panel sections when searching large workspaces
  * Change label text for grouped runs
  * Open/close all sections while searching
</Update>

<Update label="v0.48.0" description="December 20, 2023">
  ## Features

  * All required frontend changes for launch prioritization
    Refer to [this blog](https://wandb.ai/wandb_fc/launch-releases/reports/Introducing-Priority-Based-Job-Management-with-W-B-Launch--Vmlldzo2MzE2NjI2) on how you can run more important jobs than others using Launch.
  * Refer to below changes for access control and user attribution behavior of team service accounts:
    * When a team is configured in the training environment, a service account from that team can be used to log runs in either of private or public projects within that team, and additionally attribute those runs to a user only if `WANDB_USERNAME` or `WANDB_USER_EMAIL` variable is configured in the environment and the user is part of that team.
    * When a team is not configured in the training environment and a service account is used, the runs would be logged to the named project within the service account's team, and those would be attributed to a user only if `WANDB_USERNAME` or `WANDB_USER_EMAIL` variable is configured in the environment and the user is part of that team.
    * A team service account **can not** log runs in a private project in another team, but it can log runs to public projects in other teams.

  ## Fixes

  * Reduce column widths for oversized runs selectors
  * Fix a couple of bugs related to Custom Roles preview feature
</Update>

<Update label="v0.47.2" description="December 1, 2023">
  ## Features

  * Use custom roles with specific permissions to customize access control within a team\*\*
    Available in preview to enterprise customers. Please reach out to your W\&B account team or support for any questions.
  * Minor runs search improvements
  * Auto-resize runs search for long texts
  * View webhook details, including URL, secrets, date created directly from the automations table for webhook automations

  ## Fixes

  * Grouping of runs when group value is a string that looks like a number
  * Janky report panel dragging behavior
  * Update bar chart spec to match the one on public cloud
  * Clean up panel padding and plot margins
  * Restores workspace settings beta
</Update>

<Update label="v0.46.0" description="November 15, 2023">
  ## Features

  * Deployments on AWS can now use W\&B Secrets with Webhooks and Automations
    Secrets are stored securely in AWS Secret Manager. Use the terraform-aws-wandb module to provision one and
  * Update webhooks table to display more information
  * Better truncation of long strings to improve the usability of strings in the UI
  * Reduce delay for scroll to report section
  * Add white background to weave1 panels
  * Allow deep link for weave1 panels in reports
  * Allow weave1 panel resizing in reports
  * Homepage banner will now show CLI login instructions
  * User invites will now succeed if invite email can't be sent for some reason
  * Add list of associated queues to agent overview page

  ## Fixes

  * Copy function on panel overlay was dropping values
  * CSS cleanup for import modal when creating report
  * Fixes regression to hide legend when toggled off
  * Report comment highlighting
  * Remove all caching for view's LoadMetadataList()
  * Let run search stretch
  * Associate launch agents with user id from `X-WANDB-USERNAME` header
</Update>

<Update label="v0.45.0" description="October 25, 2023">
  ## Features

  * Enable artifact garbage collection using environment variable `GORILLA_ARTIFACT_GC_ENABLED=true` and cloud object versioning or soft deletion.
  * The terraform module terrraform-azurerm-wandb now supports Azure Key Vault as a secrets store.
    Deployments on Azure can now use W\&B Secrets with Webhooks and Automations. Secrets are stored securely in Azure Key Vault.

  ## Fixes

  * Remove invalid early exit preventing history deletion
  * When moving/copying runs, don't drop key-set info
  * Update mutations to no longer use defunct storage plan or artifacts billing plan at all
  * Respect skip flag in useRemoteServer
</Update>

<Update label="v0.44.1" description="October 12, 2023">
  ## Features

  **Add OpenAI proxy UI to W\&B**

  <Frame>
    ![Image showing the new OpenAI proxy UI](https://github.com/wandb/server/assets/7208315/8bc78df9-e0e7-455e-8c74-975caa96eccf)
  </Frame>

  **Also:**

  * New version v1.19.0 of our AWS Terraform module terraform-google-wandb is available
  * Add support for AWS Secret Manager for Customer Secret Store, which can be enabled after the terraform module terrraform-aws-wandb is updated and released
  * Add support for Azure Key Vault for Customer Secret Store, which can be enabled after the terraform module terrraform-azurerm-wandb is updated and released

  ## Fixes

  * Quality-of-life improvements in the model registry ui
  * int values no longer ignored when determining if a run achieved a sweep's optimization goal
  * Cache runs data to improve workspace loading perf
  * Improve TTL rendering in collection table
  * Allow service accounts to be made workflow (registry) admins
  * Add tooltip for truncated run tags in workspaces
  * Fix report page scrolling
  * Copy `y` data values for chart tooltip
  * Query secrets for webhooks in local
  * Fixing broken domain zoom in panel config
  * Hide Customer Secret Store UI if `GORILLA_CUSTOMER_SECRET_STORE_SOURCE` env var not set
  * Bump langchain to latest
  * Adding W\&B Prompts to quickstart
  * Update AWS MIs to use terraform-kubernetes-wandb v1.12.0
  * Show correct Teams Plan tracked hours teams settings page and hide on usage page
</Update>

<Update label="v0.43.0" description="October 2, 2023">
  Server 0.43.0 contains a number of minor bug fixes and performance improvements, including fixing the bottom of runs tables when there's a scrollbar. Check out the other fixes below:

  <Frame>
    ![Demo of fixed Runs table](https://github.com/wandb/server/assets/117778861/e4fe9656-b382-4328-80e4-edc4fcae1c88)
  </Frame>

  ## Fixes

  * Dramatically improve workspace loading perf
  * Fixing broken docs link in disabled add panel menu
  * Render childPanel without editor in report
  * Copying text from a panel grid when editing
  * Run overview crashing with 'length' key
  * Padding for bottom of runs table when there's a scrollbar
  * Eliminate unnecessary history key cache read
  * Error handling for Teams Checkout modal
  * Memory leak, excess filestream sending, and orphaned processes in Weave Python autotracer
</Update>

<Update label="v0.42.0" description="September 14, 2023">
  ## Features

  * W\&B Artifacts now supports time-to-live (TTL) policies.

    <Frame>
      ![Image illustrating TTL policies for artifacts](https://github.com/wandb/server/assets/117778861/fcfe9484-5adb-4ace-8e88-9c9a344d94ef)
    </Frame>

    Users can now gain more control over deleting and retention of Artifacts logged with W\&B, with the ability to set retention and time-to-live (TTL) policies! Determine when you want specific Artifacts to be deleted, update policies on existing Artifacts, and set TTL policies on upstream or downstream Artifacts.
  * Use Launch drawer when creating Sweeps
  * Delete run queue items
  * Min/max aggregations nested dropdown
  * Allow users to connect multiple S3-compatible buckets
  * Add disk i/o system metrics
  * Use the legacy way to set permissions
  * Enable CustomerSecretStore
  * Add Kubernetes as a backend for CustomerSecretStore

  ## Fixes

  * Disable storage and artifact invoices for ongoing storage calculations refractors
  * Panel deletion bug
  * Remove link-version event type from project automation slider
  * Remove upper case styling for artifact type names
  * Keep uncolored tags from changing color on render
  * Stale defaults stuck in Launch drawer on reopen
  * Trigger alias automations while creating artifact
  * Edge case failure in infinite loading tag filters
</Update>

<Update label="v0.41.0" description="August 28, 2023">
  ## Features

  * New Launch landing page

    <Frame>
      ![Image showing the new Launch landing page](https://github.com/wandb/server/assets/97066933/3bd0ecfe-1fcb-412f-92d3-6d3ce8f849d6)
    </Frame>

    We’ve updated the Launch homepage, so users looking to get started with Launch will have a much easier way to get setup quickly. Easily access detailed documentation, or simply follow the three Quickstart steps to create a Launch queue, agent, and start launching jobs immediately.
  * Add new reverse proxy to track OpenAI requests and responses
  * Show agent version on agent overview page
  * New model registry workflow removed from feature flag for all users

  ## Fixes

  * Empty projects causing infinite load on storage explorer
  * Runs marked failed when run queue items are failed
  * Use correct bucket for storing OpenAI proxy artifacts
  * SEO tags not properly rendered by host
  * Trigger export in background, on context deadline as well
  * Transition runs in pending state to running when run is initialized
  * Query so Launch queues show most recent completed and failed jobs
</Update>

<Update label="v0.40.0" description="August 18, 2023">
  ## Features

  * Webhooks enable a seamless model CI/CD workflow using Webhook Automations to trigger specific actions within the CI/CD pipeline when certain events occur. Use webhooks to facilitate a clean hand-off point between ML engineering and devops. To see this in practice for [Model Evaluation](https://www.youtube.com/watch?v=7j-Mtbo-E74) and [Model Deployment](https://www.youtube.com/watch?v=g5UiAFjM2nA), check out the linked demo videos. Learn more in [our docs](/models/automations/create-automations/webhook/).
    <Frame>
      ![Image showing webhook configuration](https://github.com/wandb/server/assets/97066933/88ebcb78-a2e6-42ed-8a37-1be0b7556a3a)
    </Frame>
  * New user activity dashboard for all customers.

  ## Fixes

  * Removed limit on number of registered models an organization could have.
  * Added search history to workspaces to make it easier to find commonly used plots.
  * Changed reports “like” icon from hearts to stars.
  * Users can now change the selected run in a workspace view with a group of runs.
  * Fixed issue causing duplicate panel grids.
  * Users can now pass in per-job resource config overrides for Sweeps on Launch
  * Added redirect from */admin/users* to new organization dashboard.
  * Fixed issues with LDAP dropping connections.
  * Improvements to run permadeletion.
</Update>

<Update label="v0.39.0" description="July 27, 2023">
  ## Features

  **Revamped Organization Dashboard**

  <Frame>
    ![Revamped Organization Dashboard showing user activity, run creation statistics, and member management features](https://github.com/wandb/server/assets/47005026/ecfda350-2fc2-4023-a98c-d8eb67498957)
  </Frame>

  We've made it easier to see who's making the most W\&B with our overhauled Organization Dashboard, accessible to W\&B admins.  You can now see details on who's created runs and reports,  who's actively using W\&B, who's invites are pending--and you can export all this in CSV to share across your organization.  Learn more about [access management](https://docs.wandb.ai/platform/hosting/iam/access-management-intro/).

  For Dedicated Cloud customers, this feature has been turned on.  For Self-Managed W\&B customers, contact W\&B support and we'll be happy to work with you to enable it.

  ## Fixes

  * Restrict service API keys to team admins
  * Launch agent configuration is now shown on the Agents page
  * Added navigation panel while viewing a single Launch job.
  * Automations can now show configuration parameters for the associated job.
  * Fixed issue with grouped runs not live updating
  * Removed extra `/` in magic and normal link url
  * Check base for incremental artifacts
  * Inviting a user into multiple teams will no longer take up too many seats in the org
</Update>

<Update label="v0.38.0" description="July 13, 2023">
  ## Features

  * Metric visualization enhancements

    <Frame>
      ![Enhanced metric visualization interface supporting up to 100 metrics with time-weighted exponential moving averages](https://github.com/wandb/server/assets/47005026/1bb0b16c-eaae-4348-be0d-cc788b67853e)
    </Frame>

    We're continuing to enhance our core metric visualization experience.   You can now define which metrics from regular expressions to render in your plots, up to 100 metrics at once.   And to more accurately represent data at high scale, we've add a new time-weighted exponential moving average smoothing algorithm for plots ([check out all of our supported algorithms](https://docs.wandb.ai/models/app/features/panels/line-plot/smoothing#docusaurus_skipToContent_fallback)).

  ## Fixes

  * Major improvement to artifact download speed: over a 6x speedup on our 1-million-file artifact benchmark.  Please upgrade to SDK version 0.15.5+.
  * (Launch) Optuna is now available as [a sweeps scheduler with Sweeps on Launch](https://docs.wandb.ai/guides/launch/sweeps-on-launch#create-a-custom-sweep-scheduler), allowing more efficient exploration of hyperparameters.
  * Run data permadeletion is now available (default off).  This can be enabled with the GORILLA\_DATA\_RETENTION\_PERIOD environment variable, specified in hours.  Please take care before updating this variable and/or chat with W\&B Support, since the deletion is permanent.  Artifacts will not be deleted by this setting.
  * Updated report sharing emails to include a preview.
  * Relaxed HTML sanitation rules for reports in projects; this had been causing rare problems with report rendering.
  * Expanded the maximum number of metrics that can be matched by a regex in chart configuration; previously this had been always 10, the maximum is now 100.
  * Fixed issue with media panel step slider becoming unsynced with the media shown.
  * Added time-weighted exponential moving average as an option for smoothing in plots.
  * The "Search panels" textbox in workspaces now preserves the user's last search.
  * Applying a username filter when runs are grouped will no longer error.
  * (Launch) The loading of the Launch tab should now be much faster, typically under two seconds.
  * (Launch) There’s now an option to edit queue configs using YAML instead of JSON.  It’s also now more clear how to edit queue configs.
  * (Launch) Runs will now show error messages in the UI when they crash or fail.
  * (Launch) If you don’t specify a project when creating a job, we’ll now use the value for WANDB\_PROJECT from your wandb.init().
  * (Launch) Updated support for custom accelerator images—these will run in noninteractive mode when building, which had been blocking some images.
  * (Launch) Fixed issue where the run author for sweeps was the agent service account, rather than the real author
  * (Launch) Clicking outside the Launch drawer will no longer close the drawer automatically.
  * (Launch) Fixed issue where training jobs that had been enqueued by a sweep but not run yet were not correctly removed from the queue if you later stopped the sweep.
  * (Launch) The Launch navigation link is now hidden for users who aren't part of the team.
  * (Launch) Fixed formatting and display issues on Agent logs.
  * Fixed scrolling, resizing, and cloning issues in Automations panel.
  * Fixed pagination on artifact action history.
  * Added support for pre-signed URLs using a VPC endpoint URL if the AWS\_S3\_ENDPOINT\_URL env var is set and passed in from the SDK side.
  * Fixed enterprise dashboard link when organization name contains "&"
  * Updated tag colors to be consistent
</Update>

<Update label="v0.36.0" description="June 14, 2023">
  ## Features

  * Clone Runs with Launch

    <Frame>
      ![Run cloning interface in W\&B Launch allowing hyperparameter adjustments and infrastructure selection](https://github.com/wandb/server/assets/47005026/4aebe22a-70c9-4186-be74-feaf73c1b2d2)
    </Frame>

    If you want to repeat a run but tweak a couple hyperparameters--say bump the batch size to take advantage of a larger machine--it's now easy to clone a run using W\&B Launch. Go to the run overview, click Clone, and you'll be able to select new infrastructure to execute the job on, with new hyperparameters.  Learn more in the [Launch documentation](https://docs.wandb.ai/guides/launch).

  ## Fixes

  * Added report creation and update action to audit logs.
  * Artifacts read through the SDK will now be captured in the [audit logs](https://docs.wandb.ai/platform/hosting/audit-logging/).
  * In report creation, added button to select all plots to add to the new report
  * New view-only users signing up via a report link will now be fast tracked to the report, rather than going through the normal signup process.
  * Team admins can now add protected aliases.
  * Improved media panel handling of intermediate steps.
  * Removed inactive 'New Model' button from Model Registry homepage for anonymous users
  * Ability to copy data from plot legends has been rolled out to all users.
  * Fixed incorrect progress indicator in Model Registry onboarding checklist.
  * Fixed issue where the Automations page could crash when job name had slashes.
  * Fixed issue where a user could update the wrong user profiles.
  * Added option to permanently delete runs and their associated metrics after a duration specified in an environment variable.
</Update>

<Update label="v0.35.0" description="June 7, 2023">
  ## Security

  Fixed issue where API keys were logged for recently logged in users. Check for `FetchAuthUserByAPIKey` in the logs which you can find in `gorilla.log` from a debug bundle and rotate any keys that are found.

  ## Features

  * Launch agent logs are now available in the W\&B App.

    <Frame>
      ![Image showing Launch agent logs in GUI](https://github.com/wandb/server/assets/97066933/9d68dc0d-54c9-4024-a108-ffb5a1500fa8)
    </Frame>

    [W\&B Launch](https://docs.wandb.ai/guides/launch) allows you to push machine learning jobs to a wide range of specialized compute environments.  With this update, you can now use W\&B to monitor and debug jobs running in these remote environments, without needing to log into your AWS or Google Cloud console.

  ## Fixes

  * Logs tab is no longer trimmed to 1000 rows.
  * Fixed scenario where artifact files pagination could get into an infinite loop
  * Fixed bug where success toast messages were not appearing
  * The Runs table will now correctly show the git commit value
</Update>

<Update label="v0.34.0" description="May 31, 2023">
  ## Features

  * New Model Registry UI

    <Frame>
      ![New Model Registry UI featuring list view, model filtering, version management, and automation controls](https://github.com/wandb/server/assets/97066933/3fedbdb8-aa79-4fb6-aab8-bddd8899e8dc)
    </Frame>

    We’re making it easier for users to manage a long list of models, and navigate seamlessly between entities in the model registry.  With this new UI, users can:
  * Look at all your registered models
  * Filter to registered models within a specific team
  * With the new list view, users can expand each panel to see the individual versions inside of it, including each version's aliases, and metadata or run metrics. Clicking on a version from this quick view can take you to it's version-view
  * Look at an overview directly by clicking “View Details”
  * See a preview of how many version, consumers, and automations are present for each registered model
  * Create Automations directly
  * See some metadata columns and details in preview
  * Change Model Access Controls

  ## Fixes

  * Improved search functionality for better universal search ranking results.
  * Added functionality to add/delete multiple tags at once in the model registry
  * Enhanced the FileMarkdown feature to correctly scroll long content.
  * Made the default team selection dropdown scrollable.
  * Removed the UI access restriction for Tier 1/2/3 plans based on tracked hour usage.
  * Added tooltips to for LLM trace viewer spans
  * LLM trace timeline/detail now splits horizontally in fullscreen
  * Added entity / team badges to Model Registry entries.
  * Improved the navigation bar experience for logged out users
  * Disabled storage/artifact banners to avoid issue where UI blocks for orgs with excess artifacts.
  * Fixed issues where user avatars were not being displayed correctly.
  * Fixed issue using Launch with Azure Git URLs
  * Launch configuration boxes now work in airgapped environments
  * In Launch queue creation, show teams as disabled (rather than hidden) for non-admins.
  * Fixed issue with embedding projector rendering
  * Fixes issue that prevented users from being able to reset their password in some cases involving mixed-case usernames.
  * Files with special characters now show up in the media panel in Azure
  * Added the ability to override the inline display format for timestamps.
  * Reports with custom charts now load when not logged in.
  * Wide GIFs no longer overflow fullscreen view
  * Increase default automations limit from 20 to 200.
  * Fixed bug allowing the appearance of deleting the version alias of a registered model (in fact, this could not be deleted on the backend).
</Update>

<Update label="v0.33.0" description="May 10, 2023">
  ## Features

  * Prompts: Zoom and pan
    <Frame>
      ![Demo of zooming and panning](https://github.com/wandb/server/assets/97066933/c6580dff-f003-4767-86f2-ad26d40108fb)
    </Frame>
    Explore complex chains of LLM prompts more easily with new zoom and pan controls in our prompts tracer.
  * Model registry admin role
    <Frame>
      ![Image showing Model registry admin role](https://github.com/wandb/server/assets/97066933/b0fc6389-b8a6-437a-91a3-a991e69bca0d)
    </Frame>
    Control your model promotion process with a new role for model registry admins.   These users can manage the list of protected aliases (for example, “challenger” or “prod”), as well as apply or remove protected aliases for model versions.
  * Viewer role
    You can now share your W\&B findings with a broader audience with the introduction of a Viewer role for W\&B Server.  Users with this role can view anything their team(s) make, but not create, edit, or delete anything.   These seats are measured separately from traditional W\&B Server seats, so reach out your W\&B account team to request an updated license.
  * Improved sharing: optional magic link, and easier signup for viewers
    Team admins can now turn off magic link sharing for a team and its members.  Disable public sharing on the team setting allows you increase team privacy controls.  Meanwhile, it’s now easier for users who receive a report link to access the report in W\&B after signing up.
  * Improved report composition
    <Frame>
      ![Improved report composition interface with enhanced draft publication, editing, and collaboration features](https://github.com/wandb/server/assets/97066933/c2858fd5-d2f6-47e3-9e93-f8d048df4f59)
    </Frame>
    Reports help share your findings W\&B further throughout an organization, including with people outside the ML team. We’ve made several investments to ensure it’s as simple and frictionless as possible to create and share them—including an improved report drafting experience with enhanced draft publication, editing, management, and sharing UX to improve how teams collaborate with Reports.
  * Updated navigation
    As W\&B has expanded the parts of the ML workflow we cover, we’ve heard your feedback that it can be hard to move around the application.   So we’ve updated the navigation sidebar to include clearer labels on the product area, and added backlinks to certain detail screens.   We’ve also renamed “Triggers” to “Automations” to better reflect the power of the feature.

  ## Fixes

  * When hovering over a plot in workspaces or a report, you can now use Cmd+C or Ctrl+C to copy run names and plot values shown in the hover control.
  * Changes to default workspaces are now no longer auto-saved.
  * Metrics in the Overview → Summary section now are formatted with commas.
  * Added an install-level option to allow non-admin users to create teams (default off; contact W\&B support to enable it).
  * Weave plots now support log scales.
  * The Launch panel can now be expanded horizontally to give more space for viewing parameters.
  * The Launch panel now indicates whether a queue is active
  * The Launch panel now allows you to choose a project for the run to be logged in.
  * Launch queues can now only be created by team admins.
  * Improved Markdown support in Launch panel.
  * Improved error message on empty Launch queue configurations.
  * Filters on the Sweeps parallel coordinates plot will now apply to all selected runsets.
  * Sweeps now no longer require a metric.
  * Added support for tracking reference artifact files saved outside W\&B in Azure Blob Storage.
  * Fixed bug in Markdown editing in Reports
  * Fullscreen Weave panels can now share config changes with the original panel.
  * Improved display of empty tables
  * Fixed bug in which the first several characters of logs were cut off
</Update>


# Weave SDK releases
Source: https://docs.wandb.ai/release-notes/weave-sdk-releases

View release notes for the Weave Python SDK (`weave` package), including new features, bug fixes, and performance improvements by version.

This page includes release notes for the [Weave Python SDK](https://pypi.org/project/weave/) (`weave` package). For the W\&B Models Python SDK (`wandb` package), see [W\&B SDK releases](/release-notes/sdk-releases). Release packages and per-commit history are on [GitHub Releases for `wandb/weave`](https://github.com/wandb/weave/releases).

<Note>
  These entries summarize user-facing SDK and trace server changes, and omit details about  Internal-only test, CI, and refactor work. For every merged change, see the GitHub release entry for that tag.
</Note>

<Update label="0.52.37" description="April 17, 2026">
  ## Added

  * OpenTelemetry integrations follow the latest semantic conventions.
  * Cost calculations and provider integrations include cache token usage.

  ## Changed

  * Improved performance for some call queries by optionally skipping a nested subquery.
  * `ClassifierMonitor` is exported from the top-level `weave` package.
  * The calls API accepts an optional `query` parameter for richer filtering.

  ## Fixed

  * Fixed the evaluation results API SQL for some filter combinations.
  * Fixed a bug where the TypeScript SDK could drop the URL scheme from `WANDB_BASE_URL`.
  * Fixed a bug where calls with multiple feedback rows could appear multiple times in list results.
  * Fixed ISO-8601 timestamp handling for ClickHouse-backed queries.
  * Fixed distributed ClickHouse mutations that incorrectly appended a `_local` suffix to table names.
  * Fixed missing spans when tracing OpenAI Agents SDK flows.
  * Fixed Google GenAI tracing when responses include non-text parts.
  * Fixed path sanitization for in-memory trace file artifacts and provider hostname handling in HTTP clients.
</Update>

<Update label="0.52.36" description="April 1, 2026">
  ## Added

  * Call queries can resolve human-readable usernames.
  * `ref.get()` works without explicitly initializing the client in some flows.
  * APIs and storage paths for text-based evaluation results, including improved handling of large evaluation sets.

  ## Changed

  * LiteLLM is pinned for compatibility with bundled integrations.
  * Moonshot is available as a model provider.
  * Underlying write-ahead logging infrastructure improves durability of trace writes.

  ## Fixed

  * Fixed a bug where abandoning a streaming generator could surface `GeneratorExit` incorrectly.
  * Fixed deserialization when stored objects include extra metadata fields.
  * Fixed feedback filters when a call has multiple feedback rows.
  * Fixed SQLite-backed call cost tracking and invalid `trace_id` handling in batch upserts.
  * Improved ClickHouse migration behavior, including retries on transient errors and clearer migrator exit status.
  * Fixed a bug where evaluation runs with more than 1000 calls could fail to load for prediction and scoring.
</Update>

<Update label="0.52.35" description="March 19, 2026">
  ## Added

  * Write-ahead log support for trace ingestion.
  * Feedback statistics queries for analytics over feedback data.

  ## Changed

  * `Ref.uri` can be read as a property (`ref.uri`) without calling `ref.uri()`.
  * You can configure the maximum size of debounced scoring history via environment variables.

  ## Fixed

  * Fixed ClickHouse casting for negative numeric filter values.
  * Fixed replicated database engine errors during on-cluster migrations.
  * Fixed `DelegatingTraceServerMixin` not forwarding some `ServiceInterface` methods.
  * Fixed retries when calling the W\&B API during `weave.init()`.
  * Fixed a bug where `EvaluationLogger` could crash when `WEAVE_DISABLED` is set.
  * Fixed `RefJSONEncoder` edge cases and classmethod instantiation for subclasses.
</Update>

<Update label="0.52.33" description="March 12, 2026">
  ## Added

  * Python SDK methods and HTTP models for object tags and aliases.
  * Instrumentation for the OpenAI Realtime API, including tool calls plus optional audio, text, and voice capture (see the TypeScript examples in the upstream release notes).

  ## Fixed

  * Fixed accumulation of Anthropic streaming completions in traces.
  * Fixed authenticated `PUT` requests in `RemoteHTTPTraceServer`.
  * Fixed cost query handling for `calls_complete` projections.
</Update>

<Update label="0.52.31" description="March 10, 2026">
  ## Added

  * Trace server support for tags and aliases on stored objects.
  * Claude Agents tracing integration.
  * Timestamps and time-to-first-token metrics for Realtime sessions.
  * Monitors can use merged scorers.

  ## Changed

  * Improved OpenTelemetry performance with a cross-request operation reference cache.

  ## Fixed

  * Fixed multiple issues in cost query construction, including escaping of internal fields and distributed ClickHouse setups.
  * Fixed LangChain integration handling for Pydantic v2 `Run` objects.
  * Fixed edge cases in prediction and scorer resolvers when inputs or metadata are missing.
  * Fixed Vertex AI text accumulation and thread visibility in `calls_complete` queries.
</Update>

<Update label="0.52.30" description="March 10, 2026">
  ## Added

  * Score backfill API for recomputing stored scores.
  * Gemini request tracking in the TypeScript SDK.

  ## Changed

  * Sharded distributed calls tables by `trace_id` or `project_id` for better query performance.
  * Simplified `calls_complete` query plans for lower latency.

  ## Fixed

  * Fixed a bug where Gemini media could fail to render in the Weave UI.
  * OpenTelemetry batch inserts now use async ClickHouse inserts.
  * Fixed `NO_PROXY` handling in HTTP clients.
  * Fixed entity versus team naming in some error messages.
</Update>

<Update label="0.52.29" description="February 27, 2026">
  ## Added

  * Usage APIs expose metadata for unfinished calls.
  * Optional `python-magic` integration for richer MIME detection.
  * Realtime threads participate in usage summaries.
  * Schema support for tags and aliases on Weave objects.
  * Structured `eval_results` query API for evaluation tables.

  ## Changed

  * Improved indexing by storing timestamps as strings in ClickHouse.
  * Reduced duplicate work during batched file creation on call upsert.

  ## Fixed

  * Fixed a bug where generators did not respect configured sampling rates.
  * Fixed buffering so streams flush immediately when a call ends.
  * Fixed sort-query index errors and deterministic JSON serialization for digest computation.
  * Fixed OpenTelemetry display names, `make_safe_name` handling, and LangChain serialization for Pydantic models.
</Update>

<Update label="0.52.28" description="February 14, 2026">
  ## Added

  * OpenTelemetry resource attributes can carry W\&B run and project variables.
  * The ORM supports `$lt` and `$lte` comparisons.
  * OpenTelemetry projects can write directly into the `calls_complete` table.
  * General availability improvements for OpenAI Realtime tracing.

  ## Changed

  * Improved performance for table scans and call statistics queries.

  ## Fixed

  * Fixed filtering calls by thread ID, including cases where filters were incorrectly optimized away.
  * Fixed idempotent annotation queue state updates.
  * Fixed a bug where `weave.finish()` did not always flush pending client data.
  * Fixed iterator typing for `PaginatedIterator`, `Dataset.select` metadata preservation, and large trace size queries that could run out of memory.
</Update>

<Update label="0.52.26" description="February 3, 2026">
  ## Added

  * Usage statistics APIs plus `/trace/usage` and `/calls/usage` endpoints for aggregated usage.
  * Optional performance mode flag for high-throughput deployments.
  * Anthropic structured-parse patching.
  * Saved views support a `column_order` field.

  ## Changed

  * Improved `PREWHERE` optimization for distributed cluster queries.

  ## Fixed

  * Fixed a bug where failed ClickHouse inserts could leak buffered rows.
  * Fixed a bug where importing IPython at import time could slow cold starts.
  * Fixed synchronous mutation migrations, summary filtering on `calls_complete`, and Google GenAI token overcounting.
  * Fixed OpenTelemetry spans with monitors and duplicate upload handling for Google Cloud Storage.
</Update>

<Update label="0.52.25" description="January 20, 2026">
  ## Fixed

  * Removed redundant HTTP response capture in some integrations.
  * Google GenAI tracing now records system instructions.
  * Google GenAI tracing now records thinking tokens separately from completion tokens.
</Update>

<Update label="0.52.24" description="January 15, 2026">
  ## Added

  * TypeScript helper APIs for working with prompts in the Node SDK.
  * Trace server safely autoconverts Base64 payloads where appropriate.
  * Leaderboard schema updates for upcoming comparison features.
  * `redact_pii_exclude_fields` setting to fine-tune PII redaction.
  * Audio inputs in `LLMAsAJudgeScorer` and richer op metadata (kinds and colors) for integrations.

  ## Fixed

  * Fixed invalid characters blocking op creation in edge cases.
  * Fixed HTTP and HTTPS proxy handling for the HTTPX client.
  * Fixed nested tracing when wrapping generators.
</Update>

<Update label="0.52.23" description="January 8, 2026">
  ## Added

  * Parsing helpers for Logfire Pydantic AI instrumentation inputs and outputs.

  ## Fixed

  * Fixed non-deterministic ordering for large-table evaluations.
  * Fixed cost queries that omitted input and output tokens.
  * Fixed distributed replicated ClickHouse tables and guarded Kafka flush behavior behind configuration.
</Update>

<Update label="0.52.22" description="January 8, 2026">
  ## Added

  * Prompts and template variables on `LLMStructuredCompletionModels` persist through completion calls.

  ## Fixed

  * Fixed ClickHouse import compatibility issues in some environments.
</Update>

<Update label="0.52.20" description="November 26, 2025">
  ## Added

  * Completions streaming APIs accept prompts and template variables.
  * `ObjectRef.from_uri` reconstructs objects from Weave URIs.
  * OpenAI Responses API tracing records `x-request-id` headers.
  * Bedrock Agents integration coverage.
  * TypeScript SDK `withAttributes` helper for span metadata.

  ## Fixed

  * Fixed a memory leak in the OpenAI Agents tracing processor.
  * Fixed a bug where refs could still publish when Weave was disabled.
  * Fixed iterator behavior when migrating HTTP client code from `requests` to `httpx`.
</Update>

<Tip>
  For releases older than 0.52.20, see [GitHub Releases for `wandb/weave`](https://github.com/wandb/weave/releases). Those entries use the default GitHub compare format (pull request links and commit prefixes). This docs page is updated for newer versions with customer-facing wording.
</Tip>


# API overview
Source: https://docs.wandb.ai/serverless-training/api-reference

Browse the Serverless Training API endpoints for chat completions, models, training jobs, and health checks with authentication details.

<Note>
  The Serverless Training API provides endpoints for managing training jobs, including serverless reinforcement learning (RL) and supervised fine-tuning (SFT). The API is OpenAI-compatible for chat completions.
</Note>

Use this reference to look up authentication requirements, the base URL, and the available endpoints for chat completions, models, training jobs, and health checks.

## Authentication

All API requests require authentication using your W\&B API key. Create an API key at [wandb.ai/settings](https://wandb.ai/settings).

Include your API key in the `Authorization` header:

```http theme={null}
Authorization: Bearer YOUR_API_KEY
```

## Base URL

```text theme={null}
https://api.training.wandb.ai/v1
```

## Available endpoints

The following sections describe the available endpoints, grouped by resource type.

### Chat completions

* **[POST /v1/chat/completions](https://docs.wandb.ai/serverless-training/api-reference/chat-completions/create-chat-completion-1)** - Create Chat Completion
* **[POST /v1/chat/completions/](https://docs.wandb.ai/serverless-training/api-reference/chat-completions/create-chat-completion)** - Create Chat Completion

### Models

* **[POST /v1/preview/models](https://docs.wandb.ai/serverless-training/api-reference/models/create-model)** - Create Model
* **[DELETE /v1/preview/models/](https://docs.wandb.ai/serverless-training/api-reference/models/delete-model)** - Delete Model
* **[DELETE /v1/preview/models//checkpoints](https://docs.wandb.ai/serverless-training/api-reference/models/delete-model-checkpoints)** - Delete Model Checkpoints
* **[GET /v1/preview/models//checkpoints](https://docs.wandb.ai/serverless-training/api-reference/models/list-model-checkpoints)** - List Model Checkpoints
* **[POST /v1/preview/models//log](https://docs.wandb.ai/serverless-training/api-reference/models/log)** - Log

### Training jobs

* **[POST /v1/preview/sft-training-jobs](https://docs.wandb.ai/serverless-training/api-reference/training-jobs/create-sft-training-job)** - Create Sft Training Job
* **[POST /v1/preview/training-jobs](https://docs.wandb.ai/serverless-training/api-reference/training-jobs/create-rl-training-job)** - Create Rl Training Job
* **[GET /v1/preview/training-jobs/](https://docs.wandb.ai/serverless-training/api-reference/training-jobs/get-training-job)** - Get Training Job
* **[GET /v1/preview/training-jobs//events](https://docs.wandb.ai/serverless-training/api-reference/training-jobs/get-training-job-events)** - Get Training Job Events

### Health

* **[GET /v1/health](https://docs.wandb.ai/serverless-training/api-reference/health/health-check)** - Health Check
* **[GET /v1/system-check](https://docs.wandb.ai/serverless-training/api-reference/health/system-check)** - System Check

## Related resources

* [Serverless Training overview](/serverless-training)
* [Prerequisites](/serverless-training/prerequisites)
* [Serverless SFT](/serverless-training/sft)
* [Use your trained models](/serverless-training/use-trained-models)
* [Available models](/serverless-training/available-models)
* [Usage limits](/serverless-training/usage-limits)


# Create Chat Completion
Source: https://docs.wandb.ai/serverless-training/api-reference/chat-completions/create-chat-completion

/serverless-training/api-reference/openapi.json post /v1/chat/completions/
Create a new chat completion.



# Create Chat Completion
Source: https://docs.wandb.ai/serverless-training/api-reference/chat-completions/create-chat-completion-1

/serverless-training/api-reference/openapi.json post /v1/chat/completions
Create a new chat completion.



# Health Check
Source: https://docs.wandb.ai/serverless-training/api-reference/health/health-check

/serverless-training/api-reference/openapi.json get /v1/health



# System Check
Source: https://docs.wandb.ai/serverless-training/api-reference/health/system-check

/serverless-training/api-reference/openapi.json get /v1/system-check
Check health of all system components.

Returns:
    JSON with status of:
    - api: Always true (if endpoint is reachable)
    - database: Whether DB connection works
    - cpu_queue: Success, duration, and any errors
    - gpu_queue: Success, duration, and any errors

Returns HTTP 503 if any checks fail.



# Create Model
Source: https://docs.wandb.ai/serverless-training/api-reference/models/create-model

/serverless-training/api-reference/openapi.json post /v1/preview/models
Create a new model.



# Delete Model
Source: https://docs.wandb.ai/serverless-training/api-reference/models/delete-model

/serverless-training/api-reference/openapi.json delete /v1/preview/models/{model_id}
Delete a model, all its checkpoints, artifacts, and the associated W&B run.



# Delete Model Checkpoints
Source: https://docs.wandb.ai/serverless-training/api-reference/models/delete-model-checkpoints

/serverless-training/api-reference/openapi.json delete /v1/preview/models/{model_id}/checkpoints
Delete specific checkpoints for a model.



# List Model Checkpoints
Source: https://docs.wandb.ai/serverless-training/api-reference/models/list-model-checkpoints

/serverless-training/api-reference/openapi.json get /v1/preview/models/{model_id}/checkpoints



# Log
Source: https://docs.wandb.ai/serverless-training/api-reference/models/log

/serverless-training/api-reference/openapi.json post /v1/preview/models/{model_id}/log
Log trajectories and calculate metrics.



# Create Rl Training Job
Source: https://docs.wandb.ai/serverless-training/api-reference/training-jobs/create-rl-training-job

/serverless-training/api-reference/openapi.json post /v1/preview/training-jobs
Create a new RL (Reinforcement Learning) training job.



# Create Sft Training Job
Source: https://docs.wandb.ai/serverless-training/api-reference/training-jobs/create-sft-training-job

/serverless-training/api-reference/openapi.json post /v1/preview/sft-training-jobs
Create a new SFT (Supervised Fine-Tuning) training job.



# Get Training Job
Source: https://docs.wandb.ai/serverless-training/api-reference/training-jobs/get-training-job

/serverless-training/api-reference/openapi.json get /v1/preview/training-jobs/{training_job_id}
Get a training job by ID.



# Get Training Job Events
Source: https://docs.wandb.ai/serverless-training/api-reference/training-jobs/get-training-job-events

/serverless-training/api-reference/openapi.json get /v1/preview/training-jobs/{training_job_id}/events
Get events for a training job.



# Support Home
Source: https://docs.wandb.ai/support



<div>
  <h1>Support Home</h1>

  <Banner title="How can we help?">
    Search for help from support articles, product documentation,<br />
    and the W\&B community.
  </Banner>

  <HelpQuestionForm />

  <h2>Browse support articles by product</h2>

  <CardGroup>
    <Card title="W&B Models" href="/support/models" icon="https://mintcdn.com/wb-21fd5541/NC1-xMMdE5yuKjPX/icons/cropped-models.svg?fit=max&auto=format&n=NC1-xMMdE5yuKjPX&q=85&s=c23ce5a772e5247c36f9f2a734177019">
      191 articles · 33 tags
    </Card>

    <Card title="W&B Weave" href="/support/weave" icon="https://mintcdn.com/wb-21fd5541/NC1-xMMdE5yuKjPX/icons/cropped-weave.svg?fit=max&auto=format&n=NC1-xMMdE5yuKjPX&q=85&s=ae6c5f30b6a30aa030c28d10bb2feca7">
      18 articles · 11 tags
    </Card>

    <Card title="Serverless Inference" href="/support/inference" icon="https://mintcdn.com/wb-21fd5541/NC1-xMMdE5yuKjPX/icons/cropped-inference.svg?fit=max&auto=format&n=NC1-xMMdE5yuKjPX&q=85&s=57f50ec28879b79645577e85c4a78494">
      7 articles · 5 tags
    </Card>
  </CardGroup>

  ## Featured articles

  ### W\&B Models

  <Card title="How can I configure the name of the run in my training code?" href="/support/models/articles/how-can-i-configure-the-name-of-the-run-">
    At the beginning of the training script, call wandb.init with a run name. For example: wandb.init(name="my\_awesome\_run") ...

    <Badge>[Experiments](/support/models/tags/experiments)</Badge>
  </Card>

  <Card title="How do I use custom CLI commands with sweeps?" href="/support/models/articles/how-do-i-use-custom-cli-commands-with-sw">
    You can use W\&B Sweeps with custom CLI commands if training configuration passes command-line arguments. This lets you r ...

    <Badge>[Sweeps](/support/models/tags/sweeps)</Badge>
  </Card>

  <Card title="Is it possible to save metrics offline and sync them to W&B later?" href="/support/models/articles/is-it-possible-to-save-metrics-offline-a">
    By default, wandb.init starts a process that syncs metrics in real time to the cloud. If you need to run experiments wit ...

    <Badge>[Experiments](/support/models/tags/experiments)</Badge> <Badge>[Environment Variables](/support/models/tags/environment-variables)</Badge> <Badge>[Metrics](/support/models/tags/metrics)</Badge>
  </Card>

  <Card title="What does wandb.init do to my training process?" href="/support/models/articles/what-does-wandbinit-do-to-my-training-pr">
    When wandb.init() runs in a training script, an API call creates a run object on the W\&B servers. A new process starts t ...

    <Badge>[Environment Variables](/support/models/tags/environment-variables)</Badge> <Badge>[Experiments](/support/models/tags/experiments)</Badge>
  </Card>

  <Card title="Still can't find what you are looking for?" href="mailto:support@wandb.com" icon="circle-info">
    Contact support
  </Card>
</div>


# Support: Serverless Inference
Source: https://docs.wandb.ai/support/inference



## Browse by category

<Card title="Administrator" href="/support/inference/tags/administrator">
  1 article
</Card>

<Card title="Authentication & Access" href="/support/inference/tags/authentication-access">
  3 articles
</Card>

<Card title="Billing" href="/support/inference/tags/billing">
  1 article
</Card>

<Card title="Quotas & Rate Limits" href="/support/inference/tags/quotas-rate-limits">
  2 articles
</Card>

<Card title="Server Errors" href="/support/inference/tags/server-errors">
  2 articles
</Card>


# Administrator
Source: https://docs.wandb.ai/support/inference/tags/administrator



<Card title="API error code 403: The inference gateway is not enabled for your organization" href="/support/inference/articles/api-error-code-403-the-inference-gateway">
  A 403 error with the message The inference gateway is not enabled for your organization means your organization doesn't  ...
</Card>


# Support: W&B Models
Source: https://docs.wandb.ai/support/models



## Featured articles

<Card title="How can I configure the name of the run in my training code?" href="/support/models/articles/how-can-i-configure-the-name-of-the-run-">
  At the beginning of the training script, call wandb.init with a run name. For example: wandb.init(name="my\_awesome\_run") ...

  <Badge>[Experiments](/support/models/tags/experiments)</Badge>
</Card>

<Card title="How do I use custom CLI commands with sweeps?" href="/support/models/articles/how-do-i-use-custom-cli-commands-with-sw">
  You can use W\&B Sweeps with custom CLI commands if training configuration passes command-line arguments. This lets you r ...

  <Badge>[Sweeps](/support/models/tags/sweeps)</Badge>
</Card>

<Card title="Is it possible to save metrics offline and sync them to W&B later?" href="/support/models/articles/is-it-possible-to-save-metrics-offline-a">
  By default, wandb.init starts a process that syncs metrics in real time to the cloud. If you need to run experiments wit ...

  <Badge>[Experiments](/support/models/tags/experiments)</Badge> <Badge>[Environment Variables](/support/models/tags/environment-variables)</Badge> <Badge>[Metrics](/support/models/tags/metrics)</Badge>
</Card>

<Card title="What does wandb.init do to my training process?" href="/support/models/articles/what-does-wandbinit-do-to-my-training-pr">
  When wandb.init() runs in a training script, an API call creates a run object on the W\&B servers. A new process starts t ...

  <Badge>[Environment Variables](/support/models/tags/environment-variables)</Badge> <Badge>[Experiments](/support/models/tags/experiments)</Badge>
</Card>

## Browse by category

<Card title="AWS" href="/support/models/tags/aws">
  2 articles
</Card>

<Card title="Academic" href="/support/models/tags/academic">
  2 articles
</Card>

<Card title="Administrator" href="/support/models/tags/administrator">
  25 articles
</Card>

<Card title="Alerts" href="/support/models/tags/alerts">
  2 articles
</Card>

<Card title="Anonymous" href="/support/models/tags/anonymous">
  2 articles
</Card>

<Card title="Artifacts" href="/support/models/tags/artifacts">
  14 articles
</Card>

<Card title="Billing" href="/support/models/tags/billing">
  4 articles
</Card>

<Card title="Charts" href="/support/models/tags/charts">
  4 articles
</Card>

<Card title="Connectivity" href="/support/models/tags/connectivity">
  4 articles
</Card>

<Card title="Environment Variables" href="/support/models/tags/environment-variables">
  12 articles
</Card>

<Card title="Experiments" href="/support/models/tags/experiments">
  39 articles
</Card>

<Card title="Hyperparameter" href="/support/models/tags/hyperparameter">
  3 articles
</Card>

<Card title="Inference" href="/support/models/tags/inference">
  5 articles
</Card>

<Card title="Logs" href="/support/models/tags/logs">
  7 articles
</Card>

<Card title="Metrics" href="/support/models/tags/metrics">
  19 articles
</Card>

<Card title="Notebooks" href="/support/models/tags/notebooks">
  3 articles
</Card>

<Card title="Outage" href="/support/models/tags/outage">
  3 articles
</Card>

<Card title="Privacy" href="/support/models/tags/privacy">
  2 articles
</Card>

<Card title="Projects" href="/support/models/tags/projects">
  4 articles
</Card>

<Card title="Python" href="/support/models/tags/python">
  6 articles
</Card>

<Card title="Reports" href="/support/models/tags/reports">
  15 articles
</Card>

<Card title="Resuming" href="/support/models/tags/resuming">
  2 articles
</Card>

<Card title="Run Crashes" href="/support/models/tags/run-crashes">
  10 articles
</Card>

<Card title="Runs" href="/support/models/tags/runs">
  19 articles
</Card>

<Card title="Security" href="/support/models/tags/security">
  7 articles
</Card>

<Card title="Storage" href="/support/models/tags/storage">
  2 articles
</Card>

<Card title="Sweeps" href="/support/models/tags/sweeps">
  19 articles
</Card>

<Card title="Tables" href="/support/models/tags/tables">
  2 articles
</Card>

<Card title="Team Management" href="/support/models/tags/team-management">
  12 articles
</Card>

<Card title="TensorBoard" href="/support/models/tags/tensorboard">
  2 articles
</Card>

<Card title="User Management" href="/support/models/tags/user-management">
  14 articles
</Card>

<Card title="Workspaces" href="/support/models/tags/workspaces">
  7 articles
</Card>

<Card title="Wysiwyg" href="/support/models/tags/wysiwyg">
  5 articles
</Card>


# Academic
Source: https://docs.wandb.ai/support/models/tags/academic



<Card title="Can I get an academic plan as a student?" href="/support/models/articles/can-i-get-an-academic-plan-as-a-student">
  Students can apply for an academic plan by following these steps: Visit the Pricing page on wandb.com and apply for the  ...
</Card>

<Card title="How can I change my account from corporate to academic?" href="/support/models/articles/how-can-i-change-my-account-from-corpora">
  If you signed up for W\&B with a corporate email but qualify for an academic plan, you can switch your account. Link your ...
</Card>


# Administrator
Source: https://docs.wandb.ai/support/models/tags/administrator



<Card title="Can I get an academic plan as a student?" href="/support/models/articles/can-i-get-an-academic-plan-as-a-student">
  Students can apply for an academic plan by following these steps: Visit the Pricing page on wandb.com and apply for the  ...
</Card>

<Card title="How can I be removed from a team?" href="/support/models/articles/how-can-i-be-removed-from-a-team">
  A team admin can remove you from a team from the Users tab of the team settings.
</Card>

<Card title="How can I change my account from corporate to academic?" href="/support/models/articles/how-can-i-change-my-account-from-corpora">
  If you signed up for W\&B with a corporate email but qualify for an academic plan, you can switch your account. Link your ...
</Card>

<Card title="How can I remove projects from a team space without admin privileges?" href="/support/models/articles/how-can-i-remove-projects-from-a-team-sp">
  To remove projects from a team space without admin privileges, try one of the following approaches: Request that a curre ...
</Card>

<Card title="How can I rotate or revoke access?" href="/support/models/articles/how-can-i-rotate-or-revoke-access">
  You can rotate or revoke personal and service account keys. Create a new API key or service account user, then reconfigu ...
</Card>

<Card title="How can I see the bytes stored, bytes tracked and tracked hours of my organization?" href="/support/models/articles/how-can-i-see-the-bytes-stored-bytes-tra">
  Organization administrators can review bytes stored, bytes tracked, and tracked hours on the Usage page in organization  ...
</Card>

<Card title="How do I add more seats to my account?" href="/support/models/articles/how-do-i-add-more-seats">
  If you're an administrator who needs to expand your team's access to W\&B, you can request additional seats for your acco ...
</Card>

<Card title="How do I cancel my subscription?" href="/support/models/articles/how-do-i-cancel-my-subscription">
  To cancel your subscription, contact the W\&B support team with the details that identify your account: 1. Contact the su ...
</Card>

<Card title="How do I change my billing address?" href="/support/models/articles/how-do-i-change-my-billing-address">
  To change your billing address, contact the support team at [support@wandb.com](mailto:support@wandb.com).
</Card>

<Card title="How do I delete a team from my account?" href="/support/models/articles/how-do-i-delete-a-team-from-my-account">
  As a team admin, you can permanently delete a team from your W\&B account. Use this procedure when you need to remove a t ...
</Card>

<Card title="How do I delete my organization account?" href="/support/models/articles/how-do-i-delete-my-organization-account">
  The W\&B support team handles organization account deletion. To delete your organization account, contact support at supp ...
</Card>

<Card title="How do I downgrade my subscription plan?" href="/support/models/articles/how-do-i-downgrade-my-subscription-plan">
  To downgrade your subscription plan, contact the support team at [support@wandb.com](mailto:support@wandb.com) with your current plan details and th ...
</Card>

<Card title="How do I export a list of users from my W&B organization?" href="/support/models/articles/how-do-i-export-a-list-of-users-from-my-">
  To export a list of users from a W\&B organization as an administrator, use the SCIM API with the following code: Modify  ...
</Card>

<Card title="How do I fix service account errors like `Unauthorized` or missing runs?" href="/support/models/articles/service-account-unauthorized-or-runs-no">
  When you use a service account to automate W\&B workflows, you might encounter authorization errors or find that runs don ...
</Card>

<Card title="How do I get added to a team on W&B?" href="/support/models/articles/how-do-i-get-added-to-a-team-on-wb">
  To join a team, follow these steps: 1. Request an invitation from a team administrator or someone with administrative pr ...
</Card>

<Card title="How do I log metrics without saving my code?" href="/support/models/articles/how-do-i-log-metrics-without-saving-code">
  To keep proprietary source code or training data out of W\&B for privacy or compliance reasons, limit what W\&B captures s ...
</Card>

<Card title="How do I renew my expired license?" href="/support/models/articles/how-do-i-renew-my-expired-license">
  To renew an expired license, contact the support team at [support@wandb.com](mailto:support@wandb.com) to start the renewal process and get a new li ...
</Card>

<Card title="If I am the admin of my local instance, how should I manage it?" href="/support/models/articles/if-i-am-the-admin-of-my-local-instance-h">
  If you're the admin for your instance, review the User Management section for instructions on adding users and creating  ...
</Card>

<Card title="Is it possible to add the same service account to multiple teams?" href="/support/models/articles/is-it-possible-to-add-the-same-service-a">
  You can't add a service account to multiple teams in W\&B. Each service account is tied to a specific team.
</Card>

<Card title="Is it possible to change the username?" href="/support/models/articles/is-it-possible-to-change-the-username">
  You can't change the username after account creation. Create a new account with the desired username instead.
</Card>

<Card title="Is there a monthly subscription option for the Teams plan?" href="/support/models/articles/is-there-a-monthly-subscription-option-f">
  The Teams plan doesn't offer a monthly subscription option. It's billed annually.
</Card>

<Card title="On a local instance, which files should I check when I have issues?" href="/support/models/articles/on-a-local-instance-which-files-should-i">
  When you troubleshoot issues on a W\&B local instance, start by collecting the Debug Bundle. The bundle contains the logs ...
</Card>

<Card title="What is a service account, and why is it useful?" href="/support/models/articles/what-is-a-service-account-and-why-is-it-">
  A service account represents a non-human or machine identity that can automate common tasks across teams and projects. S ...
</Card>

<Card title="What is the difference between team and organization?" href="/support/models/articles/what-is-the-difference-between-team-and-_2">
  A team is a collaborative workspace for users on the same projects. An organization is a higher-level entity that can in ...
</Card>

<Card title="Why is my enterprise license not recognized?" href="/support/models/articles/enterprise-license-not-recognized">
  This page helps administrators of W\&B enterprise deployments diagnose common issues when W\&B doesn't recognize an enterp ...
</Card>


# Alerts
Source: https://docs.wandb.ai/support/models/tags/alerts



<Card title="Do 'Run Finished' alerts work in notebooks?" href="/support/models/articles/do-run-finished-alerts-work-in-notebooks">
  No. Run Finished alerts only work with Python scripts. Even if you've enabled the Run Finished setting in User Settings, ...
</Card>

<Card title="How can I send run alerts to Microsoft Teams?" href="/support/models/articles/how-can-i-send-run-alerts-to-microsoft-t">
  W\&B doesn't send alerts to Microsoft Teams directly, but you can route them through email so your team sees run notifica ...
</Card>


# Anonymous
Source: https://docs.wandb.ai/support/models/tags/anonymous



<Card title="Features not available to anonymous users" href="/support/models/articles/what-are-features-that-are-not-available">
  Anonymous accounts let you try W\&B without signing up, but several features are restricted. Review the following limitat ...
</Card>

<Card title="How does someone without an account see run results?" href="/support/models/articles/how-does-someone-without-an-account-see-">
  W\&B supports anonymous runs so that people without an account can view experiment results and optionally claim them late ...
</Card>


# Artifacts
Source: https://docs.wandb.ai/support/models/tags/artifacts



<Card title="How can I fetch these Version IDs and ETags in W&B?" href="/support/models/articles/how-can-i-fetch-these-version-ids-and-et">
  If you log an artifact reference with W\&B and enable versioning on the buckets, the version IDs appear in the Amazon S3  ...
</Card>

<Card title="How can I find the artifacts logged or consumed by a run? How can I find the runs that produced or consumed an artifact?" href="/support/models/articles/how-can-i-find-the-artifacts-logged-or-c">
  This page shows you how to trace the relationships between runs and artifacts in W\&B. You can find which run produced or ...
</Card>

<Card title="How do I best log models from runs in a sweep?" href="/support/models/articles/how-do-i-best-log-models-from-runs-in-a-">
  To log models in a sweep, create a model artifact for the sweep. Each version represents a different run from the sweep. ...
</Card>

<Card title="How do I disable W&B while testing my code?" href="/support/models/articles/how-do-i-disable-wandb-when-testing-my-code">
  To configure W\&B as a no-op for testing, use wandb.init(mode="disabled") or set WANDB\_MODE=disabled. Even when you use w ...
</Card>

<Card title="How do I find an artifact from the best run in a sweep?" href="/support/models/articles/how-do-i-find-an-artifact-from-the-best-">
  To retrieve artifacts from the best-performing run in a sweep, use the following code. It sorts the sweep's runs by vali ...
</Card>

<Card title="How do I log an artifact to an existing run?" href="/support/models/articles/how-do-i-log-an-artifact-to-an-existing-">
  Sometimes you need to mark an artifact as the output of a previously logged run. For example, you might generate a new m ...
</Card>

<Card title="How do I save code?‌" href="/support/models/articles/how-do-i-save-code">
  Saving code with each run helps you reproduce experiments and track which version of your code produced a given result.  ...
</Card>

<Card title="How do I set a retention or expiration policy on my artifact?" href="/support/models/articles/how-do-i-set-a-retention-or-expiration-p">
  To manage artifacts that contain sensitive data or to schedule the deletion of artifact versions, set a TTL (time-to-liv ...
</Card>

<Card title="How much storage does each artifact version use?" href="/support/models/articles/how-much-storage-does-each-artifact-vers">
  Only files that change between two artifact versions incur storage costs. Consider an image artifact named animals that  ...
</Card>

<Card title="Is it possible to recover an artifact after it has been deleted with a run?" href="/support/models/articles/is-it-possible-to-recover-an-artifact-af">
  No. When you delete a run, a prompt asks whether to delete the associated artifacts. If you choose this option, W\&B perm ...
</Card>

<Card title="Using artifacts with multiple architectures and runs?" href="/support/models/articles/using-artifacts-with-multiple-architectu">
  This article describes how to organize artifacts when a project explores multiple model architectures across many runs,  ...
</Card>

<Card title="Where are artifacts downloaded, and how can I control that?" href="/support/models/articles/where-are-artifacts-downloaded-and-how-c">
  By default, W\&B downloads artifacts to the artifacts/ folder. To change the location: Pass the path to wandb.Artifact(). ...
</Card>

<Card title="Who has access to my artifacts?" href="/support/models/articles/who-has-access-to-my-artifacts">
  This page explains how W\&B determines access to artifacts, so you can know who in your organization can read or modify t ...
</Card>

<Card title="Why can't I link my artifact to the Registry?" href="/support/models/articles/cannot-link-artifact-to-registry-from-p">
  If you can't link an artifact to a W\&B Registry, the most common cause is that the artifact was logged with a personal e ...
</Card>


# AWS
Source: https://docs.wandb.ai/support/models/tags/aws



<Card title="How do I use W&B Sweeps with SageMaker?" href="/support/models/articles/how-do-i-use-sweeps-with-sagemaker">
  You can use W\&B Sweeps with Amazon SageMaker to run hyperparameter searches on SageMaker training jobs. To authenticate  ...
</Card>

<Card title="How do I use W&B Sweeps with cloud infrastructure?" href="/support/models/articles/how-do-i-use-sweeps-with-cloud-infrastructure">
  To run a W\&B Sweep on cloud infrastructure such as AWS Batch or ECS, publish the sweep\_id so that any W\&B Sweep agent ca ...
</Card>


# Billing
Source: https://docs.wandb.ai/support/models/tags/billing



<Card title="How do I change my billing address?" href="/support/models/articles/how-do-i-change-my-billing-address">
  To change your billing address, contact the support team at [support@wandb.com](mailto:support@wandb.com).
</Card>

<Card title="How do I downgrade my subscription plan?" href="/support/models/articles/how-do-i-downgrade-my-subscription-plan">
  To downgrade your subscription plan, contact the support team at [support@wandb.com](mailto:support@wandb.com) with your current plan details and th ...
</Card>

<Card title="How do I update my payment method?" href="/support/models/articles/how-do-we-update-our-payment-method">
  To update your payment method, follow these steps: 1. Go to your profile page: Navigate to your user profile page. 2. Se ...
</Card>

<Card title="Is there a monthly subscription option for the Teams plan?" href="/support/models/articles/is-there-a-monthly-subscription-option-f">
  The Teams plan doesn't offer a monthly subscription option. It's billed annually.
</Card>


# Charts
Source: https://docs.wandb.ai/support/models/tags/charts



<Card title="How do I add Plotly or Bokeh Charts into Tables?" href="/support/models/articles/how-do-i-add-plotly-or-bokeh-charts-into">
  Direct integration of Plotly or Bokeh figures into tables isn't supported. Instead, export the figures to HTML and inclu ...
</Card>

<Card title="How do you delete a custom chart preset?" href="/support/models/articles/how-do-you-delete-a-custom-chart-preset">
  To delete a custom chart preset: 1. Access the custom chart editor. 2. Click the currently selected chart type to open a ...
</Card>

<Card title="How do you show a step slider in a custom chart?" href="/support/models/articles/how-do-you-show-a-step-slider-in-a-custo">
  A step slider lets viewers scrub through logged steps in a custom chart. To enable it, open the Other settings page of t ...
</Card>

<Card title="What is the difference between `.log()` and `.summary`?" href="/support/models/articles/what-is-the-difference-between-log-and-s">
  The summary displays in the table, while the log saves all values for future plotting. For example, call run.log() whene ...
</Card>


# Connectivity
Source: https://docs.wandb.ai/support/models/tags/connectivity



<Card title="How can I resolve the Filestream rate limit exceeded error?" href="/support/models/articles/how-can-i-resolve-the-filestream-rate-li">
  The Filestream rate limit exceeded error occurs when W\&B receives too many API requests in a short window, which can int ...
</Card>

<Card title="How do I deal with network issues?" href="/support/models/articles/how-do-i-deal-with-network-issues">
  This page helps you resolve SSL and network errors that prevent W\&B from syncing your training data, so you can keep exp ...
</Card>

<Card title="How do I resolve a run initialization timeout error in wandb?" href="/support/models/articles/how-do-i-resolve-a-run-initialization-ti">
  A run initialization timeout error means wandb.init() can't establish a connection to the W\&B service within the allotte ...
</Card>

<Card title="Is there a W&B outage?" href="/support/models/articles/is-there-a-wb-outage">
  To check whether the W\&B Multi-tenant Cloud at wandb.ai is experiencing an outage, visit the W\&B status page.
</Card>


# Environment Variables
Source: https://docs.wandb.ai/support/models/tags/environment-variables



<Card title="Do environment variables overwrite the parameters passed to wandb.init()?" href="/support/models/articles/do-environment-variables-overwrite-the-p">
  Arguments passed to wandb.init override environment variables. To respect a user-set environment variable while still pr ...
</Card>

<Card title="How can I define the local location for `wandb` files?" href="/support/models/articles/how-can-i-define-the-local-location-for-">
  You can control where wandb stores its files locally by setting the following environment variables or, where noted, by  ...
</Card>

<Card title="How do I handle the 'Failed to query for notebook' error?" href="/support/models/articles/how-do-i-handle-the-failed-to-query-for-">
  If you encounter the error message "Failed to query for notebook name, you can set it manually with the WANDB\_NOTEBOOK\_N ...
</Card>

<Card title="How do I run W&B offline?" href="/support/models/articles/how-do-i-run-wandb-offline">
  To train on a machine without internet access and upload your results to W\&B later, follow these steps: 1. Set the envir ...
</Card>

<Card title="How do I silence W&B info messages?" href="/support/models/articles/how-do-i-silence-wb-info-messages">
  W\&B emits info-level log messages by default, which can clutter notebook output. This page describes two ways to reduce  ...
</Card>

<Card title="How do I stop wandb from writing to my terminal or my Jupyter notebook output?" href="/support/models/articles/how-do-i-stop-wandb-from-writing-to-my-t">
  Set the environment variable WANDB\_SILENT to true.
</Card>

<Card title="How do I switch between accounts on the same machine?" href="/support/models/articles/how-do-i-switch-between-accounts-on-the-">
  To manage two W\&B accounts from the same machine, store both API keys in a file outside your repository, then load the k ...
</Card>

<Card title="How does wandb stream logs and writes to disk?" href="/support/models/articles/how-does-wandb-stream-logs-and-writes-to">
  W\&B queues events in memory and writes them to disk asynchronously. This approach handles failures and supports the WAND ...
</Card>

<Card title="Is it possible to save metrics offline and sync them to W&B later?" href="/support/models/articles/is-it-possible-to-save-metrics-offline-a">
  By default, wandb.init starts a process that syncs metrics in real time to the cloud. If you need to run experiments wit ...
</Card>

<Card title="What does wandb.init do to my training process?" href="/support/models/articles/what-does-wandbinit-do-to-my-training-pr">
  When wandb.init() runs in a training script, an API call creates a run object on the W\&B servers. A new process starts t ...
</Card>

<Card title="What happens if internet connection is lost while I'm training a model?" href="/support/models/articles/what-happens-if-internet-connection-is-l">
  If the library cannot connect to the internet, it enters a retry loop and continues to attempt to stream metrics until t ...
</Card>

<Card title="Where are artifacts downloaded, and how can I control that?" href="/support/models/articles/where-are-artifacts-downloaded-and-how-c">
  By default, W\&B downloads artifacts to the artifacts/ folder. To change the location: Pass the path to wandb.Artifact(). ...
</Card>


# Experiments
Source: https://docs.wandb.ai/support/models/tags/experiments



<Card title="Does W&B use the `multiprocessing` library?" href="/support/models/articles/does-wb-use-the-multiprocessing-library">
  Yes, W\&B uses the multiprocessing library. As a result, scripts that call W\&B directly need to guard their entry point s ...
</Card>

<Card title="Does logging block my training?" href="/support/models/articles/does-logging-block-my-training">
  "Is the logging function lazy? I don't want to depend on the network to send results to your servers while executing loc ...
</Card>

<Card title="Does your tool track or store training data?" href="/support/models/articles/does-your-tool-track-or-store-training-d">
  To associate a dataset with a training run, pass a SHA or unique identifier to wandb.Run.config.update(...). W\&B stores  ...
</Card>

<Card title="How can I access the data logged to my runs directly and programmatically?" href="/support/models/articles/how-can-i-access-the-data-logged-to-my-r">
  The history object tracks metrics logged with wandb.log. To access the history object, use the API:
</Card>

<Card title="How can I compare images or media across epochs or steps?" href="/support/models/articles/how-can-i-compare-images-or-media-across">
  To compare images or other media logged at different epochs or steps, expand the image panel and use the step slider to  ...
</Card>

<Card title="How can I configure the name of the run in my training code?" href="/support/models/articles/how-can-i-configure-the-name-of-the-run-">
  At the beginning of the training script, call wandb.init with a run name. For example: wandb.init(name="my\_awesome\_run") ...
</Card>

<Card title="How can I define the local location for `wandb` files?" href="/support/models/articles/how-can-i-define-the-local-location-for-">
  You can control where wandb stores its files locally by setting the following environment variables or, where noted, by  ...
</Card>

<Card title="How can I organize my logged charts and media in the W&B UI?" href="/support/models/articles/how-can-i-organize-my-logged-charts-and-">
  Organize logged charts and media into related groups by naming logged items with a shared prefix, which makes it easier  ...
</Card>

<Card title="How can I save the git commit associated with my run?" href="/support/models/articles/how-can-i-save-the-git-commit-associated">
  When you call wandb.init, W\&B automatically collects Git information, including the remote repository link and the SHA o ...
</Card>

<Card title="How can I see files that don't appear in the Files tab?" href="/support/models/articles/how-can-i-see-files-that-do-not-appear-i">
  The Files tab shows a maximum of 10,000 files. To download all files, use the public API:
</Card>

<Card title="How can I use W&B with multiprocessing, for example distributed training?" href="/support/models/articles/how-can-i-use-wandb-with-multiprocessing">
  This article explains how to use W\&B in training programs that span multiple processes, such as distributed training job ...
</Card>

<Card title="How do I add Plotly or Bokeh Charts into Tables?" href="/support/models/articles/how-do-i-add-plotly-or-bokeh-charts-into">
  Direct integration of Plotly or Bokeh figures into tables isn't supported. Instead, export the figures to HTML and inclu ...
</Card>

<Card title="How do I fix `Rate limit exceeded` errors when logging metrics?" href="/support/models/articles/rate-limit-exceeded-on-metric-logging">
  If you receive an HTTP 429 Rate limit exceeded error when you call wandb.log(), you're exceeding the rate limit quota fo ...
</Card>

<Card title="How do I get the random run name in my script?" href="/support/models/articles/how-do-i-get-the-random-run-name-in-my-s">
  Call a run object's .save() method to save the current run. Retrieve the name using the run object's name attribute.
</Card>

<Card title="How do I launch multiple runs from one script?" href="/support/models/articles/how-do-i-launch-multiple-runs-from-one-s">
  To log multiple runs within a single script, you must finish each previous run before you start a new one. The following ...
</Card>

<Card title="How do I log a list of values?" href="/support/models/articles/how-do-i-log-a-list-of-values">
  You can log a list of values, such as per-step losses, with wandb.Run.log(). The following examples show two common appr ...
</Card>

<Card title="How do I log metrics on two different time scales?" href="/support/models/articles/how-do-i-log-metrics-on-two-time-scales">
  To log metrics on two different time scales, log indices like batch and epoch alongside your metrics. For example, you m ...
</Card>

<Card title="How do I plot multiple lines on a plot with a legend?" href="/support/models/articles/how-do-i-plot-multiple-lines-on-a-plot-w">
  Create a multi-line custom chart with wandb.plot.line\_series(). Go to the project page to view the line chart. To add a  ...
</Card>

<Card title="How do I programmatically access the human-readable run name?" href="/support/models/articles/how-do-i-programmatically-access-the-hum">
  To access the .name attribute of a wandb.Run, use the following code:
</Card>

<Card title="How do I run W&B offline?" href="/support/models/articles/how-do-i-run-wandb-offline">
  To train on a machine without internet access and upload your results to W\&B later, follow these steps: 1. Set the envir ...
</Card>

<Card title="How do I set the run name to the run ID?" href="/support/models/articles/how-do-i-set-the-run-name-to-the-run-id">
  To overwrite the run name with the run ID, use the following code snippet:
</Card>

<Card title="I didn't name my run. Where is the run name coming from?" href="/support/models/articles/i-didnt-name-my-run-where-is-the-run-nam">
  If a run isn't explicitly named, W\&B assigns a random name to identify it in your project. Examples of random names are  ...
</Card>

<Card title="InitStartError: Error communicating with wandb process" href="/support/models/articles/initstarterror-error-communicating-with-">
  This error indicates that the W\&B library can't launch the process that synchronizes data to the server. To resolve the  ...
</Card>

<Card title="Is it possible to save metrics offline and sync them to W&B later?" href="/support/models/articles/is-it-possible-to-save-metrics-offline-a">
  By default, wandb.init starts a process that syncs metrics in real time to the cloud. If you need to run experiments wit ...
</Card>

<Card title="My run's state is `crashed` on the UI but is still running on my machine. What do I do to get my data back?" href="/support/models/articles/my-runs-state-is-crashed-on-the-ui-but-i">
  This state usually means you lost connection to your machine during training. To recover your data, run wandb sync PATH\_ ...
</Card>

<Card title="What does wandb.init do to my training process?" href="/support/models/articles/what-does-wandbinit-do-to-my-training-pr">
  When wandb.init() runs in a training script, an API call creates a run object on the W\&B servers. A new process starts t ...
</Card>

<Card title="What happens if I pass a class attribute into wandb.Run.log()?" href="/support/models/articles/what-happens-if-i-pass-a-class-attribute">
  Avoid passing class attributes into wandb.Run.log(). Attributes may change before the network call executes. When you st ...
</Card>

<Card title="What happens when I log millions of steps to W&B? How is that rendered in the browser?" href="/support/models/articles/what-happens-when-i-log-millions-of-step">
  The number of points sent affects the loading time of graphs in the UI. For lines exceeding 1,000 points, the backend sa ...
</Card>

<Card title="What if I want to integrate W&B into my project, but I don't want to upload any images or media?" href="/support/models/articles/what-if-i-want-to-integrate-wb-into-my-p">
  W\&B supports projects that log only scalars by letting you specify which files or data to upload. For an example of logg ...
</Card>

<Card title="What if I want to log some metrics on batches and some metrics only on epochs?" href="/support/models/articles/what-if-i-want-to-log-some-metrics-on-ba">
  To log specific metrics in each batch and standardize plots, log the desired x-axis values alongside the metrics. In the ...
</Card>

<Card title="What is the difference between wandb.init modes?" href="/support/models/articles/what-is-the-difference-between-wandbinit">
  When you call wandb.init, you can choose a mode that controls how the client communicates with the W\&B server. The follo ...
</Card>

<Card title="Why am I seeing fewer data points than I logged?" href="/support/models/articles/why-am-i-seeing-fewer-data-points-than-i">
  When you visualize metrics against an X-axis other than Step, expect to see fewer data points. You must log metrics at t ...
</Card>

<Card title="Why are steps missing from a CSV metric export?" href="/support/models/articles/why-are-steps-missing-from-a-csv-metric-">
  Export limits can prevent you from exporting the entire run history as a CSV or with the run.history API. To access the  ...
</Card>

<Card title="Why can't I sort or filter metrics with certain characters?" href="/support/models/articles/why-cant-i-sort-or-filter-metrics-with-c">
  Metric names in W\&B must follow GraphQL naming conventions so that you can sort and filter them in the UI. Valid metric  ...
</Card>

<Card title="Why does my process stop responding when using Hydra with W&B?" href="/support/models/articles/process-hangs-when-using-hydra-with-wan">
  This page explains how to resolve unresponsive processes that occur when you start a process with Hydra alongside W\&B. I ...
</Card>

<Card title="Why does my training hang with distributed training?" href="/support/models/articles/training-hangs-with-distributed-trainin">
  This article helps you resolve training hangs when you use W\&B with distributed training frameworks, so your runs can st ...
</Card>

<Card title="Why is nothing showing up in my graphs?" href="/support/models/articles/why-is-nothing-showing-up-in-my-graphs">
  If the message "No visualization data logged yet" appears, the script hasn't executed the first wandb.log call. This can ...
</Card>

<Card title="Why is the same metric appearing more than once?" href="/support/models/articles/why-is-the-same-metric-appearing-more-th">
  When you log multiple data types under the same key, W\&B splits them in the database. This results in multiple entries o ...
</Card>

<Card title="Will wandb slow down my training?" href="/support/models/articles/will-wandb-slow-down-my-training">
  W\&B has a minimal impact on training performance under normal usage conditions. Normal use includes logging at a rate of ...
</Card>


# Hyperparameter
Source: https://docs.wandb.ai/support/models/tags/hyperparameter



<Card title="Best practices to organize hyperparameter searches" href="/support/models/articles/best-practices-to-organize-hyperparamete">
  When you run many hyperparameter searches, tag your runs so you can group and find related experiments later. Tag runs i ...
</Card>

<Card title="How do I rerun a grid search?" href="/support/models/articles/how-do-i-rerun-a-grid-search">
  If a grid search completes but some W\&B Runs crashed, delete those runs to rerun them. Then select the Resume button on  ...
</Card>

<Card title="What is the `Est. Runs` column?" href="/support/models/articles/what-is-the-est-runs-column">
  W\&B provides an estimated number of runs generated when creating a W\&B Sweep with a discrete search space. This total re ...
</Card>


# Inference
Source: https://docs.wandb.ai/support/models/tags/inference



<Card title="How do I fix Invalid Authentication (401) errors with Serverless Inference?" href="/support/models/articles/how-do-i-fix-invalid-authentication-401-">
  A 401 Invalid Authentication error means your API key is invalid or your W\&B project entity or name is incorrect. Use th ...
</Card>

<Card title="How do I fix server errors (500, 503) with Serverless Inference?" href="/support/models/articles/how-do-i-fix-server-errors-500-503-with-">
  Server errors indicate temporary issues with the Serverless Inference service. This page helps you identify these errors ...
</Card>

<Card title="What are the best practices for handling Serverless Inference errors?" href="/support/models/articles/what-are-the-best-practices-for-handling">
  Follow these best practices to handle W\&B Serverless Inference errors and maintain reliable applications. Always impleme ...
</Card>

<Card title="Why am I getting insufficient quota errors (402) with Serverless Inference?" href="/support/models/articles/why-am-i-getting-insufficient-quota-erro">
  Insufficient quota errors (402) occur when you don't have remaining credits in your plan. Error: "You exceeded your curr ...
</Card>

<Card title="Why am I getting rate limit errors (429) with Serverless Inference?" href="/support/models/articles/why-am-i-getting-rate-limit-errors-429-w">
  This page explains why Serverless Inference returns 429 rate limit errors and how to resolve them so your requests succe ...
</Card>


# Logs
Source: https://docs.wandb.ai/support/models/tags/logs



<Card title="Does logging block my training?" href="/support/models/articles/does-logging-block-my-training">
  "Is the logging function lazy? I don't want to depend on the network to send results to your servers while executing loc ...
</Card>

<Card title="How can I overwrite the logs from previous steps?" href="/support/models/articles/how-can-i-overwrite-the-logs-from-previo">
  To overwrite logs from previous steps, use forking and rewind. The ability to fork or rewind a run is in active developm ...
</Card>

<Card title="How do I log a list of values?" href="/support/models/articles/how-do-i-log-a-list-of-values">
  You can log a list of values, such as per-step losses, with wandb.Run.log(). The following examples show two common appr ...
</Card>

<Card title="How do I log runs launched by continuous integration or internal tools?" href="/support/models/articles/how-do-i-log-runs-launched-by-continuous">
  To launch automated tests or internal tools that log to W\&B, create a Service Account on the team settings page. This ac ...
</Card>

<Card title="How do I log to the right wandb user on a shared machine?" href="/support/models/articles/how-do-i-log-to-the-right-wandb-user-on-">
  When multiple users share a machine, runs can log to the wrong W\&B account if you don't explicitly set credentials. To e ...
</Card>

<Card title="How do I turn off logging?" href="/support/models/articles/how-do-i-turn-off-logging">
  You can stop W\&B from logging data to the remote server, or keep logging enabled but suppress its warning messages. Use  ...
</Card>

<Card title="Which files should I check when my code crashes?" href="/support/models/articles/which-files-should-i-check-when-my-code-">
  For the affected run, check debug.log and debug-internal.log in wandb/run- DATE \_ TIME - RUN-ID /logs in the directory w ...
</Card>


# Metrics
Source: https://docs.wandb.ai/support/models/tags/metrics



<Card title="How can I change how frequently to log system metrics?" href="/support/models/articles/how-can-i-change-how-frequently-to-log-s">
  To configure the frequency to log system metrics, set \_stats\_sampling\_interval to a number of seconds, expressed as a fl ...
</Card>

<Card title="How can I disable logging of system metrics to W&B?" href="/support/models/articles/how-can-i-disable-logging-of-system-metr">
  To disable logging of system metrics, set \_disable\_stats to True:
</Card>

<Card title="How can I log additional metrics after a run completes?" href="/support/models/articles/how-can-i-log-additional-metrics-after-a">
  To add metrics to a run that has already finished, W\&B offers several approaches depending on the complexity of your wor ...
</Card>

<Card title="How can I overwrite the logs from previous steps?" href="/support/models/articles/how-can-i-overwrite-the-logs-from-previo">
  To overwrite logs from previous steps, use forking and rewind. The ability to fork or rewind a run is in active developm ...
</Card>

<Card title="How do I fix `Rate limit exceeded` errors when logging metrics?" href="/support/models/articles/rate-limit-exceeded-on-metric-logging">
  If you receive an HTTP 429 Rate limit exceeded error when you call wandb.log(), you're exceeding the rate limit quota fo ...
</Card>

<Card title="How do I fix an `anaconda 400 error` during a sweep?" href="/support/models/articles/anaconda-400-error-during-a-sweep">
  An anaconda 400 error during a sweep often means you didn't log the metric you're optimizing. You might see: The sweep c ...
</Card>

<Card title="How do I log metrics on two different time scales?" href="/support/models/articles/how-do-i-log-metrics-on-two-time-scales">
  To log metrics on two different time scales, log indices like batch and epoch alongside your metrics. For example, you m ...
</Card>

<Card title="How do I log metrics without saving my code?" href="/support/models/articles/how-do-i-log-metrics-without-saving-code">
  To keep proprietary source code or training data out of W\&B for privacy or compliance reasons, limit what W\&B captures s ...
</Card>

<Card title="How do I plot the maximum of a metric?" href="/support/models/articles/how-do-i-plot-the-maximum-of-a-metric">
  To plot the running maximum of a metric: 1. Create a scatter plot of the metric. 2. Open the Edit menu and select Annota ...
</Card>

<Card title="How often are system metrics collected?" href="/support/models/articles/how-often-are-system-metrics-collected">
  By default, system metrics are collected every 10 seconds. For higher-resolution metrics, email [contact@wandb.com](mailto:contact@wandb.com).
</Card>

<Card title="Is it possible to save metrics offline and sync them to W&B later?" href="/support/models/articles/is-it-possible-to-save-metrics-offline-a">
  By default, wandb.init starts a process that syncs metrics in real time to the cloud. If you need to run experiments wit ...
</Card>

<Card title="Optimizing multiple metrics" href="/support/models/articles/optimizing-multiple-metrics">
  W\&B Sweeps optimize a single objective metric. To optimize multiple metrics in a single run, combine them into a weighte ...
</Card>

<Card title="What happens when I log millions of steps to W&B? How is that rendered in the browser?" href="/support/models/articles/what-happens-when-i-log-millions-of-step">
  The number of points sent affects the loading time of graphs in the UI. For lines exceeding 1,000 points, the backend sa ...
</Card>

<Card title="What if I want to log some metrics on batches and some metrics only on epochs?" href="/support/models/articles/what-if-i-want-to-log-some-metrics-on-ba">
  To log specific metrics in each batch and standardize plots, log the desired x-axis values alongside the metrics. In the ...
</Card>

<Card title="Why am I seeing fewer data points than I logged?" href="/support/models/articles/why-am-i-seeing-fewer-data-points-than-i">
  When you visualize metrics against an X-axis other than Step, expect to see fewer data points. You must log metrics at t ...
</Card>

<Card title="Why can't I sort or filter metrics with certain characters?" href="/support/models/articles/why-cant-i-sort-or-filter-metrics-with-c">
  Metric names in W\&B must follow GraphQL naming conventions so that you can sort and filter them in the UI. Valid metric  ...
</Card>

<Card title="Why does my workspace load slowly?" href="/support/models/articles/workspace-loads-slowly-with-many-metric">
  Workspaces can load slowly when a project has many metrics, runs, or panels. Common causes include high metric counts, e ...
</Card>

<Card title="Why is nothing showing up in my graphs?" href="/support/models/articles/why-is-nothing-showing-up-in-my-graphs">
  If the message "No visualization data logged yet" appears, the script hasn't executed the first wandb.log call. This can ...
</Card>

<Card title="Why is the same metric appearing more than once?" href="/support/models/articles/why-is-the-same-metric-appearing-more-th">
  When you log multiple data types under the same key, W\&B splits them in the database. This results in multiple entries o ...
</Card>


# Notebooks
Source: https://docs.wandb.ai/support/models/tags/notebooks



<Card title="Do 'Run Finished' alerts work in notebooks?" href="/support/models/articles/do-run-finished-alerts-work-in-notebooks">
  No. Run Finished alerts only work with Python scripts. Even if you've enabled the Run Finished setting in User Settings, ...
</Card>

<Card title="How do I handle the 'Failed to query for notebook' error?" href="/support/models/articles/how-do-i-handle-the-failed-to-query-for-">
  If you encounter the error message "Failed to query for notebook name, you can set it manually with the WANDB\_NOTEBOOK\_N ...
</Card>

<Card title="How do I silence W&B info messages?" href="/support/models/articles/how-do-i-silence-wb-info-messages">
  W\&B emits info-level log messages by default, which can clutter notebook output. This page describes two ways to reduce  ...
</Card>


# Outage
Source: https://docs.wandb.ai/support/models/tags/outage



<Card title="How can I resolve the Filestream rate limit exceeded error?" href="/support/models/articles/how-can-i-resolve-the-filestream-rate-li">
  The Filestream rate limit exceeded error occurs when W\&B receives too many API requests in a short window, which can int ...
</Card>

<Card title="Is there a W&B outage?" href="/support/models/articles/is-there-a-wb-outage">
  To check whether the W\&B Multi-tenant Cloud at wandb.ai is experiencing an outage, visit the W\&B status page.
</Card>

<Card title="What happens if internet connection is lost while I'm training a model?" href="/support/models/articles/what-happens-if-internet-connection-is-l">
  If the library cannot connect to the internet, it enters a retry loop and continues to attempt to stream metrics until t ...
</Card>


# Privacy
Source: https://docs.wandb.ai/support/models/tags/privacy



<Card title="Can W&B team members see my data?" href="/support/models/articles/can-wb-team-members-see-my-data">
  Key engineers and support staff at W\&B access logged values for debugging purposes with user permission. All data stores ...
</Card>

<Card title="How can I change the privacy of my project?" href="/support/models/articles/how-can-i-change-the-privacy-of-my-proje">
  A project's privacy setting controls who can view, edit, or submit runs to the project. To change a project's privacy (v ...
</Card>


# Projects
Source: https://docs.wandb.ai/support/models/tags/projects



<Card title="How can I change the privacy of my project?" href="/support/models/articles/how-can-i-change-the-privacy-of-my-proje">
  A project's privacy setting controls who can view, edit, or submit runs to the project. To change a project's privacy (v ...
</Card>

<Card title="How can I delete multiple runs in bulk instead of one at a time?" href="/support/models/articles/how-can-i-delete-multiple-runs-in-bulk-i">
  From the Runs tab in your Workspace, select the checkbox next to each run to delete, then click Delete. To filter runs b ...
</Card>

<Card title="How do I rename a project?" href="/support/models/articles/how-do-i-rename-a-project">
  To rename a project: 1. Navigate to the Project overview. 2. Click Edit Project. If the project name is protected, such  ...
</Card>

<Card title="How many runs can I create per project?" href="/support/models/articles/how-many-runs-can-i-create-per-project">
  For optimal performance, limit each project to approximately 10,000 runs.
</Card>


# Python
Source: https://docs.wandb.ai/support/models/tags/python



<Card title="Does the W&B client support Python 2?" href="/support/models/articles/does-the-wb-client-support-python-2">
  The W\&B client library supported both Python 2.7 and Python 3 through version 0.10. Version 0.11 removed Python 2.7 supp ...
</Card>

<Card title="Does the W&B client support Python 3.5?" href="/support/models/articles/does-the-wb-client-support-python-35">
  The W\&B client library supported Python 3.5 until version 0.11. Support for Python 3.5 ended with version 0.12, which al ...
</Card>

<Card title="Does this only work for Python?" href="/support/models/articles/does-this-only-work-for-python">
  The library supports Python 2.7 and later, as well as Python 3.6 and later. The architecture allows integration with oth ...
</Card>

<Card title="How can I resume a sweep using Python code?" href="/support/models/articles/how-can-i-resume-a-sweep-using-python-co">
  To resume a sweep, pass the sweep\_id to the wandb.agent() function.
</Card>

<Card title="How do I install the wandb Python library in environments without gcc?" href="/support/models/articles/how-do-i-install-the-wandb-python-librar">
  This page is for users who want to install the wandb Python library in environments without gcc, such as minimal contain ...
</Card>

<Card title="Is there an anaconda package for Weights and Biases?" href="/support/models/articles/is-there-an-anaconda-package-for-weights">
  An Anaconda package is available, and you can install it with either pip or conda. For conda, get the package from the c ...
</Card>


# Reports
Source: https://docs.wandb.ai/support/models/tags/reports



<Card title="Adding multiple authors to a report" href="/support/models/articles/adding-multiple-authors-to-a-report">
  You can credit contributors by adding multiple authors to a report. This is useful when a report represents work from mo ...
</Card>

<Card title="Embed reports" href="/support/models/articles/embedding-reports">
  This page describes how to share a W\&B report by embedding it in another page, such as a blog post, internal wiki, or ex ...
</Card>

<Card title="Filter and delete unwanted reports" href="/support/models/articles/filter-and-delete-unwanted-reports">
  To keep a project organized, remove reports you no longer need. Use the search bar to filter the reports list, then do o ...
</Card>

<Card title="How do I delete a panel grid?" href="/support/models/articles/how-do-i-delete-a-panel-grid">
  To select the panel grid, click the drag handle in its top-right corner. To remove it, press Delete or Backspace.
</Card>

<Card title="How do I insert a table?" href="/support/models/articles/how-do-i-insert-a-table">
  Tables are a Markdown feature without a direct what you see is what you get (WYSIWYG) equivalent. To add a table, insert ...
</Card>

<Card title="How do I use Markdown in my reports?" href="/support/models/articles/how-do-i-use-markdown-in-my-reports">
  To insert a Markdown block, type /mark anywhere in the document and press Enter. The block lets you edit content in Mark ...
</Card>

<Card title="How to get multiple charts with different selected runs?" href="/support/models/articles/how-to-get-multiple-charts-with-differen">
  W\&B Reports lets you display charts for different sets of runs side by side in the same report by using separate panel g ...
</Card>

<Card title="I converted my report to WYSIWYG but want to revert back to Markdown" href="/support/models/articles/i-converted-my-report-to-wysiwyg-but-wan">
  If you converted a report from Markdown to WYSIWYG and want to undo the conversion, the available options depend on how  ...
</Card>

<Card title="Incorporating LaTeX" href="/support/models/articles/incorporating-latex">
  You can include LaTeX equations in reports to document mathematical notation alongside your notes, custom visualizations ...
</Card>

<Card title="My report is running slowly after the change to WYSIWYG" href="/support/models/articles/my-report-is-running-slowly-after-the-ch">
  Performance issues may arise on older hardware or with very large reports. To mitigate this, collapse sections of the re ...
</Card>

<Card title="My report looks different after converting from Markdown" href="/support/models/articles/my-report-looks-different-after-converti">
  The Markdown-to-WYSIWYG (what you see is what you get) conversion aims to maintain the original appearance of your repor ...
</Card>

<Card title="Refreshing data" href="/support/models/articles/refreshing-data">
  Workspaces automatically load updated data. Auto-refresh doesn't apply to reports. To refresh report data, reload the pa ...
</Card>

<Card title="Upload a CSV to a report" href="/support/models/articles/upload-a-csv-to-a-report">
  To upload a CSV file to a report, use the wandb.Table format. Load the CSV file in your Python script and log it as a wa ...
</Card>

<Card title="Upload an image to a report" href="/support/models/articles/upload-an-image-to-a-report">
  To upload an image to a report: 1. On a new line, press /. 2. Scroll to the Image option. 3. Drag and drop an image into ...
</Card>

<Card title="Who can edit and share reports?" href="/support/models/articles/who-can-edit-and-share-reports">
  Reports created within an individual's private project remain visible only to that user. The user can share their projec ...
</Card>


# Resuming
Source: https://docs.wandb.ai/support/models/tags/resuming



<Card title="How do I fix the error `resume='must' but run ([RUN-ID]) doesn't exist`?" href="/support/models/articles/how-do-i-fix-the-error-resumemust-but-ru">
  If you encounter the error resume='must' but run ( RUN-ID ) doesn't exist, the run you're attempting to resume doesn't e ...
</Card>

<Card title="How do I use the resume parameter when resuming a run in W&B?" href="/support/models/articles/how-do-i-use-the-resume-parameter-when-r">
  To use the resume parameter in W\&B, set the resume argument in wandb.init() with entity, project, and id specified. The  ...
</Card>


# Run Crashes
Source: https://docs.wandb.ai/support/models/tags/run-crashes



<Card title="How can I fix an error like `AttributeError: module 'wandb' has no attribute ...`?" href="/support/models/articles/how-can-i-fix-an-error-like-attributeerr">
  If you encounter an error like AttributeError: module 'wandb' has no attribute 'init' or AttributeError: module 'wandb'  ...
</Card>

<Card title="How do I fix `Cuda out of memory` during a sweep?" href="/support/models/articles/cuda-out-of-memory-during-a-sweep">
  If you see Cuda out of memory during a sweep, refactor your code to use process-based execution. Rewrite your code as a  ...
</Card>

<Card title="How do I kill a job with wandb?" href="/support/models/articles/how-do-i-kill-a-job-with-wandb">
  To stop a script instrumented with W\&B, press Ctrl+D. Use this method to cancel a training run you no longer need.
</Card>

<Card title="How do I resolve a run initialization timeout error in wandb?" href="/support/models/articles/how-do-i-resolve-a-run-initialization-ti">
  A run initialization timeout error means wandb.init() can't establish a connection to the W\&B service within the allotte ...
</Card>

<Card title="If wandb crashes, will it possibly crash my training run?" href="/support/models/articles/if-wandb-crashes-will-it-possibly-crash-">
  W\&B runs in a separate process, so if W\&B crashes, your training run continues. During an internet outage, W\&B keeps ret ...
</Card>

<Card title="InitStartError: Error communicating with wandb process" href="/support/models/articles/initstarterror-error-communicating-with-">
  This error indicates that the W\&B library can't launch the process that synchronizes data to the server. To resolve the  ...
</Card>

<Card title="My run's state is `crashed` on the UI but is still running on my machine. What do I do to get my data back?" href="/support/models/articles/my-runs-state-is-crashed-on-the-ui-but-i">
  This state usually means you lost connection to your machine during training. To recover your data, run wandb sync PATH\_ ...
</Card>

<Card title="Why does my process stop responding when using Hydra with W&B?" href="/support/models/articles/process-hangs-when-using-hydra-with-wan">
  This page explains how to resolve unresponsive processes that occur when you start a process with Hydra alongside W\&B. I ...
</Card>

<Card title="Why does my training hang with distributed training?" href="/support/models/articles/training-hangs-with-distributed-trainin">
  This article helps you resolve training hangs when you use W\&B with distributed training frameworks, so your runs can st ...
</Card>

<Card title="Why is a run marked crashed in W&B when it's training fine locally?" href="/support/models/articles/why-is-a-run-marked-crashed-in-wb-when-i">
  A crashed status usually indicates a connection problem. If the server loses internet access and data stops syncing, W\&B ...
</Card>


# Runs
Source: https://docs.wandb.ai/support/models/tags/runs



<Card title="Best practices to organize hyperparameter searches" href="/support/models/articles/best-practices-to-organize-hyperparamete">
  When you run many hyperparameter searches, tag your runs so you can group and find related experiments later. Tag runs i ...
</Card>

<Card title="Can you group runs by tags?" href="/support/models/articles/can-you-group-runs-by-tags">
  A run can have multiple tags, so grouping by tags isn't supported. Instead, add a value to the config object for these r ...
</Card>

<Card title="How can I categorize runs without the Group feature?" href="/support/models/articles/how-can-i-categorize-runs-without-the-group-feature">
  You can categorize runs by tags or custom metadata. To organize runs by these values, use the Group button in the projec ...
</Card>

<Card title="How can I change how frequently to log system metrics?" href="/support/models/articles/how-can-i-change-how-frequently-to-log-s">
  To configure the frequency to log system metrics, set \_stats\_sampling\_interval to a number of seconds, expressed as a fl ...
</Card>

<Card title="How can I change the colors of each run in the same group?" href="/support/models/articles/how-can-i-change-the-colors-of-each-run-">
  You can't change the colors of individual runs within a group. All runs in the same group share a common color.
</Card>

<Card title="How can I delete multiple runs in bulk instead of one at a time?" href="/support/models/articles/how-can-i-delete-multiple-runs-in-bulk-i">
  From the Runs tab in your Workspace, select the checkbox next to each run to delete, then click Delete. To filter runs b ...
</Card>

<Card title="How can I disable logging of system metrics to W&B?" href="/support/models/articles/how-can-i-disable-logging-of-system-metr">
  To disable logging of system metrics, set \_disable\_stats to True:
</Card>

<Card title="How can I log a metric that doesn't change over time such as a final evaluation accuracy?" href="/support/models/articles/how-can-i-log-a-metric-that-doesnt-chang">
  To record a single, unchanging value such as a final evaluation accuracy, log it once with run.log. For example, run.log ...
</Card>

<Card title="How can I log additional metrics after a run completes?" href="/support/models/articles/how-can-i-log-additional-metrics-after-a">
  To add metrics to a run that has already finished, W\&B offers several approaches depending on the complexity of your wor ...
</Card>

<Card title="How can I recover deleted runs?" href="/support/models/articles/how-can-i-recover-deleted-runs">
  Learn how to recover recently deleted runs in the W\&B App for up to 7 days.
</Card>

<Card title="How do I change the group of a completed run?" href="/support/models/articles/how-do-i-change-the-group-of-a-completed-run">
  You can change the group assigned to a completed run using the API. This feature doesn't appear in the web UI. To update ...
</Card>

<Card title="How do I fix `CommError, Run does not exist` during a sweep?" href="/support/models/articles/commerror-run-does-not-exist-during-swee">
  If you see both CommError, Run does not exist and ERROR Error uploading during a sweep, the most likely cause is that yo ...
</Card>

<Card title="How do I fix the error `resume='must' but run ([RUN-ID]) doesn't exist`?" href="/support/models/articles/how-do-i-fix-the-error-resumemust-but-ru">
  If you encounter the error resume='must' but run ( RUN-ID ) doesn't exist, the run you're attempting to resume doesn't e ...
</Card>

<Card title="How do I log runs launched by continuous integration or internal tools?" href="/support/models/articles/how-do-i-log-runs-launched-by-continuous">
  To launch automated tests or internal tools that log to W\&B, create a Service Account on the team settings page. This ac ...
</Card>

<Card title="How do I move a run to another project?" href="/support/models/articles/how-do-i-move-a-run-to-another-project">
  You can move a run from one project to another to reorganize your work, consolidate related experiments, or relocate a r ...
</Card>

<Card title="How do I rerun a grid search?" href="/support/models/articles/how-do-i-rerun-a-grid-search">
  If a grid search completes but some W\&B Runs crashed, delete those runs to rerun them. Then select the Resume button on  ...
</Card>

<Card title="How do I resolve permission errors when logging a run?" href="/support/models/articles/how-do-i-resolve-permission-errors-when-">
  Permission errors when logging a run typically indicate a mismatch between your code, your credentials, or your deployme ...
</Card>

<Card title="How many runs can I create per project?" href="/support/models/articles/how-many-runs-can-i-create-per-project">
  For optimal performance, limit each project to approximately 10,000 runs.
</Card>

<Card title="Why are steps missing from a CSV metric export?" href="/support/models/articles/why-are-steps-missing-from-a-csv-metric-">
  Export limits can prevent you from exporting the entire run history as a CSV or with the run.history API. To access the  ...
</Card>


# Security
Source: https://docs.wandb.ai/support/models/tags/security



<Card title="Can W&B team members see my data?" href="/support/models/articles/can-wb-team-members-see-my-data">
  Key engineers and support staff at W\&B access logged values for debugging purposes with user permission. All data stores ...
</Card>

<Card title="Do you have a bug bounty program?" href="/support/models/articles/do-you-have-a-bug-bounty-program">
  W\&B has a bug bounty program. For details, see the W\&B security portal.
</Card>

<Card title="How can I rotate or revoke access?" href="/support/models/articles/how-can-i-rotate-or-revoke-access">
  You can rotate or revoke personal and service account keys. Create a new API key or service account user, then reconfigu ...
</Card>

<Card title="How do I find my API key?" href="/support/models/articles/how-do-i-find-my-api-key">
  import ApiKeyFind from "/snippets/\_includes/api-key-find.mdx";
</Card>

<Card title="How do I fix service account errors like `Unauthorized` or missing runs?" href="/support/models/articles/service-account-unauthorized-or-runs-no">
  When you use a service account to automate W\&B workflows, you might encounter authorization errors or find that runs don ...
</Card>

<Card title="How do I resolve permission errors when logging a run?" href="/support/models/articles/how-do-i-resolve-permission-errors-when-">
  Permission errors when logging a run typically indicate a mismatch between your code, your credentials, or your deployme ...
</Card>

<Card title="How do I set up SSO for W&B Multi-tenant?" href="/support/models/articles/how-do-i-set-up-sso-for-multi-tenant">
  W\&B supports Single Sign-On (SSO) for the Multi-tenant offering through Auth0, so you can let users authenticate with yo ...
</Card>


# Storage
Source: https://docs.wandb.ai/support/models/tags/storage



<Card title="How much storage does each artifact version use?" href="/support/models/articles/how-much-storage-does-each-artifact-vers">
  Only files that change between two artifact versions incur storage costs. Consider an image artifact named animals that  ...
</Card>

<Card title="Why does the storage meter not update after deleting runs?" href="/support/models/articles/why-does-the-storage-meter-not-update-af">
  The storage meter doesn't update immediately after you delete runs. The backend takes time to process the deletion and u ...
</Card>


# Sweeps
Source: https://docs.wandb.ai/support/models/tags/sweeps



<Card title="Best practices to organize hyperparameter searches" href="/support/models/articles/best-practices-to-organize-hyperparamete">
  When you run many hyperparameter searches, tag your runs so you can group and find related experiments later. Tag runs i ...
</Card>

<Card title="How can I change the directory my sweep logs to locally?" href="/support/models/articles/how-can-i-change-the-directory-my-sweep-">
  Set the logging directory for run data by configuring the WANDB\_DIR environment variable. For example:
</Card>

<Card title="How can I resume a sweep using Python code?" href="/support/models/articles/how-can-i-resume-a-sweep-using-python-co">
  To resume a sweep, pass the sweep\_id to the wandb.agent() function.
</Card>

<Card title="How do I best log models from runs in a sweep?" href="/support/models/articles/how-do-i-best-log-models-from-runs-in-a-">
  To log models in a sweep, create a model artifact for the sweep. Each version represents a different run from the sweep. ...
</Card>

<Card title="How do I enable code logging with Sweeps?" href="/support/models/articles/how-do-i-enable-code-logging-with-sweeps">
  To enable code logging for sweeps, add wandb.log\_code() after initializing the W\&B Run. This step is necessary even when ...
</Card>

<Card title="How do I fix `CommError, Run does not exist` during a sweep?" href="/support/models/articles/commerror-run-does-not-exist-during-swee">
  If you see both CommError, Run does not exist and ERROR Error uploading during a sweep, the most likely cause is that yo ...
</Card>

<Card title="How do I fix `Cuda out of memory` during a sweep?" href="/support/models/articles/cuda-out-of-memory-during-a-sweep">
  If you see Cuda out of memory during a sweep, refactor your code to use process-based execution. Rewrite your code as a  ...
</Card>

<Card title="How do I fix an `anaconda 400 error` during a sweep?" href="/support/models/articles/anaconda-400-error-during-a-sweep">
  An anaconda 400 error during a sweep often means you didn't log the metric you're optimizing. You might see: The sweep c ...
</Card>

<Card title="How do I flag boolean variables as hyperparameters?" href="/support/models/articles/how-do-i-flag-boolean-variables-as-hyperparameters">
  To pass hyperparameters as boolean flags, use the macro in the command section of your sweep configuration. The macro au ...
</Card>

<Card title="How do I rerun a grid search?" href="/support/models/articles/how-do-i-rerun-a-grid-search">
  If a grid search completes but some W\&B Runs crashed, delete those runs to rerun them. Then select the Resume button on  ...
</Card>

<Card title="How do I set default hyperparameter values in a sweep?" href="/support/models/articles/how-do-i-set-default-hyperparameter-values-in-a-sweep">
  You don't need to provide values for every hyperparameter in a W\&B Sweep. You can set defaults and let the sweep overrid ...
</Card>

<Card title="How do I use W&B Sweeps with SageMaker?" href="/support/models/articles/how-do-i-use-sweeps-with-sagemaker">
  You can use W\&B Sweeps with Amazon SageMaker to run hyperparameter searches on SageMaker training jobs. To authenticate  ...
</Card>

<Card title="How do I use W&B Sweeps with cloud infrastructure?" href="/support/models/articles/how-do-i-use-sweeps-with-cloud-infrastructure">
  To run a W\&B Sweep on cloud infrastructure such as AWS Batch or ECS, publish the sweep\_id so that any W\&B Sweep agent ca ...
</Card>

<Card title="How do I use custom CLI commands with sweeps?" href="/support/models/articles/how-do-i-use-custom-cli-commands-with-sw">
  You can use W\&B Sweeps with custom CLI commands if training configuration passes command-line arguments. This lets you r ...
</Card>

<Card title="How should I run sweeps on SLURM?" href="/support/models/articles/how-should-i-run-sweeps-on-slurm">
  To use sweeps with the SLURM scheduling system, run wandb agent --count 1 SWEEP\_ID in each scheduled job. This command e ...
</Card>

<Card title="Is there a way to add extra values to a sweep, or do I need to start a new one?" href="/support/models/articles/is-there-a-way-to-add-extra-values-to-a-">
  Once a W\&B Sweep starts, you can't change the sweep configuration. However, you can navigate to any table view, select r ...
</Card>

<Card title="Optimizing multiple metrics" href="/support/models/articles/optimizing-multiple-metrics">
  W\&B Sweeps optimize a single objective metric. To optimize multiple metrics in a single run, combine them into a weighte ...
</Card>

<Card title="What happens if I edit my Python files while a sweep is running?" href="/support/models/articles/what-happens-if-i-edit-my-python-files-w">
  The behavior depends on which file you edit. While a sweep is running: If the train.py script that the sweep uses change ...
</Card>

<Card title="What is the `Est. Runs` column?" href="/support/models/articles/what-is-the-est-runs-column">
  W\&B provides an estimated number of runs generated when creating a W\&B Sweep with a discrete search space. This total re ...
</Card>


# Tables
Source: https://docs.wandb.ai/support/models/tags/tables



<Card title="How do I add Plotly or Bokeh Charts into Tables?" href="/support/models/articles/how-do-i-add-plotly-or-bokeh-charts-into">
  Direct integration of Plotly or Bokeh figures into tables isn't supported. Instead, export the figures to HTML and inclu ...
</Card>

<Card title="How do I insert a table?" href="/support/models/articles/how-do-i-insert-a-table">
  Tables are a Markdown feature without a direct what you see is what you get (WYSIWYG) equivalent. To add a table, insert ...
</Card>


# Team Management
Source: https://docs.wandb.ai/support/models/tags/team-management



<Card title="How can I be removed from a team?" href="/support/models/articles/how-can-i-be-removed-from-a-team">
  A team admin can remove you from a team from the Users tab of the team settings.
</Card>

<Card title="How do I delete a team from my account?" href="/support/models/articles/how-do-i-delete-a-team-from-my-account">
  As a team admin, you can permanently delete a team from your W\&B account. Use this procedure when you need to remove a t ...
</Card>

<Card title="How do I get added to a team on W&B?" href="/support/models/articles/how-do-i-get-added-to-a-team-on-wb">
  To join a team, follow these steps: 1. Request an invitation from a team administrator or someone with administrative pr ...
</Card>

<Card title="How do I log metrics without saving my code?" href="/support/models/articles/how-do-i-log-metrics-without-saving-code">
  To keep proprietary source code or training data out of W\&B for privacy or compliance reasons, limit what W\&B captures s ...
</Card>

<Card title="Is it possible to add the same service account to multiple teams?" href="/support/models/articles/is-it-possible-to-add-the-same-service-a">
  You can't add a service account to multiple teams in W\&B. Each service account is tied to a specific team.
</Card>

<Card title="Is there a monthly subscription option for the Teams plan?" href="/support/models/articles/is-there-a-monthly-subscription-option-f">
  The Teams plan doesn't offer a monthly subscription option. It's billed annually.
</Card>

<Card title="What is a team and where can I find more information about it?" href="/support/models/articles/what-is-a-team-and-where-can-i-find-more">
  For more information about teams, see the teams section.
</Card>

<Card title="What is the difference between team and entity? As a user - what does entity mean for me?" href="/support/models/articles/what-is-the-difference-between-team-and-">
  A team is a collaborative workspace for users working on the same projects. An entity is either a username or a team nam ...
</Card>

<Card title="What is the difference between team and organization?" href="/support/models/articles/what-is-the-difference-between-team-and-_2">
  A team is a collaborative workspace for users on the same projects. An organization is a higher-level entity that can in ...
</Card>

<Card title="What type of roles are available and what are the differences between them?" href="/support/models/articles/what-type-of-roles-are-available-and-wha">
  Visit the Team roles and permissions page for an overview of the available roles and permissions.
</Card>

<Card title="When should I log to my personal entity versus my team entity?" href="/support/models/articles/when-should-i-log-to-my-personal-entity-">
  Personal entities are unavailable for accounts created after May 21, 2024. We recommend that you log new projects to a t ...
</Card>

<Card title="Who can create a team? Who can add or delete people from a team? Who can delete projects?" href="/support/models/articles/who-can-create-a-team-who-can-add-or-del">
  For details about roles and permissions, see Team roles and permissions.
</Card>


# TensorBoard
Source: https://docs.wandb.ai/support/models/tags/tensorboard



<Card title="How is W&B different from TensorBoard?" href="/support/models/articles/how-is-wb-different-from-tensorboard">
  If you're evaluating experiment tracking tools or migrating from TensorBoard, this page summarizes the key differences t ...
</Card>

<Card title="What formula do you use for your smoothing algorithm?" href="/support/models/articles/what-formula-do-you-use-for-your-smoothi">
  The exponential moving average formula aligns with the one used in TensorBoard. Refer to this explanation on Stack Overf ...
</Card>


# User Management
Source: https://docs.wandb.ai/support/models/tags/user-management



<Card title="Can I get an academic plan as a student?" href="/support/models/articles/can-i-get-an-academic-plan-as-a-student">
  Students can apply for an academic plan by following these steps: Visit the Pricing page on wandb.com and apply for the  ...
</Card>

<Card title="How can I change my account from corporate to academic?" href="/support/models/articles/how-can-i-change-my-account-from-corpora">
  If you signed up for W\&B with a corporate email but qualify for an academic plan, you can switch your account. Link your ...
</Card>

<Card title="How can I delete my user account?" href="/support/models/articles/how-can-i-delete-my-user-account">
  To delete your user account, go to your user settings and click Delete account. This action is irreversible and takes ef ...
</Card>

<Card title="How can I log in to W&B Server?" href="/support/models/articles/how-can-i-log-in-to-wb-server">
  Set the login URL in one of these ways: Set the environment variable WANDB\_BASE\_URL to the Server URL. Set the --host fl ...
</Card>

<Card title="How can I regain access to my account if I cannot receive a password reset email?" href="/support/models/articles/how-can-i-regain-access-to-my-account-if">
  If you can't receive a password reset email, try the following steps to regain access to your account: 1. Check spam or  ...
</Card>

<Card title="How can I resolve sign-in issues with my account?" href="/support/models/articles/how-can-i-resolve-login-issues-with-my-a">
  If you can't sign in to your W\&B account, the following steps help you identify and resolve common sign-in issues: Accou ...
</Card>

<Card title="How do I add more seats to my account?" href="/support/models/articles/how-do-i-add-more-seats">
  If you're an administrator who needs to expand your team's access to W\&B, you can request additional seats for your acco ...
</Card>

<Card title="How do I export a list of users from my W&B organization?" href="/support/models/articles/how-do-i-export-a-list-of-users-from-my-">
  To export a list of users from a W\&B organization as an administrator, use the SCIM API with the following code: Modify  ...
</Card>

<Card title="How do I find my API key?" href="/support/models/articles/how-do-i-find-my-api-key">
  import ApiKeyFind from "/snippets/\_includes/api-key-find.mdx";
</Card>

<Card title="How do I log in to W&B Self-Managed?" href="/support/models/articles/how-do-i-log-in-to-wandb-self-managed">
  To authenticate to a W\&B Self-Managed instance, point the CLI and SDK at your instance URL before you log in: Set WANDB\_ ...
</Card>

<Card title="How do I request the complete deletion of my W&B account?" href="/support/models/articles/how-do-i-request-the-complete-deletion-o">
  To delete a W\&B account, go to the User settings page, scroll to the bottom, and click Delete Account.
</Card>

<Card title="Is it possible to change the username?" href="/support/models/articles/is-it-possible-to-change-the-username">
  You can't change the username after account creation. Create a new account with the desired username instead.
</Card>

<Card title="Is it possible to create a new account with an email that was previously used for a deleted account?" href="/support/models/articles/is-it-possible-to-create-a-new-account-w">
  Yes. You can create a new account with an email previously associated with a deleted account.
</Card>

<Card title="What type of roles are available and what are the differences between them?" href="/support/models/articles/what-type-of-roles-are-available-and-wha">
  Visit the Team roles and permissions page for an overview of the available roles and permissions.
</Card>


# Workspaces
Source: https://docs.wandb.ai/support/models/tags/workspaces



<Card title="How can I categorize runs without the Group feature?" href="/support/models/articles/how-can-i-categorize-runs-without-the-group-feature">
  You can categorize runs by tags or custom metadata. To organize runs by these values, use the Group button in the projec ...
</Card>

<Card title="How can I change the colors of each run in the same group?" href="/support/models/articles/how-can-i-change-the-colors-of-each-run-">
  You can't change the colors of individual runs within a group. All runs in the same group share a common color.
</Card>

<Card title="How do I enable dark mode?" href="/support/models/articles/how-do-i-enable-dark-mode">
  W\&B provides a dark mode for the UI as a public preview feature. Dark mode is in beta and not optimized for accessibilit ...
</Card>

<Card title="How do I fix the 'overflows maximum values of a signed 64 bits integer' error?" href="/support/models/articles/how-do-i-fix-the-overflows-maximum-value">
  To resolve this error, add ?workspace=clear to the end of the URL and press Enter. This loads a cleared version of the p ...
</Card>

<Card title="Refreshing data" href="/support/models/articles/refreshing-data">
  Workspaces automatically load updated data. Auto-refresh doesn't apply to reports. To refresh report data, reload the pa ...
</Card>

<Card title="What features are hidden, and where can I find them?" href="/support/models/articles/what-really-good-functionalities-are-hid">
  Some features are hidden under a feature flag in the Beta Features section of your team's settings.
</Card>

<Card title="Why does my workspace load slowly?" href="/support/models/articles/workspace-loads-slowly-with-many-metric">
  Workspaces can load slowly when a project has many metrics, runs, or panels. Common causes include high metric counts, e ...
</Card>


# Wysiwyg
Source: https://docs.wandb.ai/support/models/tags/wysiwyg



<Card title="How do I delete a panel grid?" href="/support/models/articles/how-do-i-delete-a-panel-grid">
  To select the panel grid, click the drag handle in its top-right corner. To remove it, press Delete or Backspace.
</Card>

<Card title="How do I insert a table?" href="/support/models/articles/how-do-i-insert-a-table">
  Tables are a Markdown feature without a direct what you see is what you get (WYSIWYG) equivalent. To add a table, insert ...
</Card>

<Card title="I converted my report to WYSIWYG but want to revert back to Markdown" href="/support/models/articles/i-converted-my-report-to-wysiwyg-but-wan">
  If you converted a report from Markdown to WYSIWYG and want to undo the conversion, the available options depend on how  ...
</Card>

<Card title="My report is running slowly after the change to WYSIWYG" href="/support/models/articles/my-report-is-running-slowly-after-the-ch">
  Performance issues may arise on older hardware or with very large reports. To mitigate this, collapse sections of the re ...
</Card>

<Card title="My report looks different after converting from Markdown" href="/support/models/articles/my-report-looks-different-after-converti">
  The Markdown-to-WYSIWYG (what you see is what you get) conversion aims to maintain the original appearance of your repor ...
</Card>


# Support: W&B Weave
Source: https://docs.wandb.ai/support/weave



## Browse by category

<Card title="Agents" href="/support/weave/tags/agents">
  2 articles
</Card>

<Card title="Client Info" href="/support/weave/tags/client-info">
  2 articles
</Card>

<Card title="Code Capture" href="/support/weave/tags/code-capture">
  2 articles
</Card>

<Card title="Data Capture" href="/support/weave/tags/data-capture">
  2 articles
</Card>

<Card title="Evaluation" href="/support/weave/tags/evaluation">
  2 articles
</Card>

<Card title="Performance" href="/support/weave/tags/performance">
  5 articles
</Card>

<Card title="System Info" href="/support/weave/tags/system-info">
  2 articles
</Card>

<Card title="Trace Data" href="/support/weave/tags/trace-data">
  4 articles
</Card>

<Card title="Tracing" href="/support/weave/tags/tracing">
  2 articles
</Card>

<Card title="UI Rendering" href="/support/weave/tags/ui-rendering">
  2 articles
</Card>

<Card title="weave.op" href="/support/weave/tags/weave-op">
  2 articles
</Card>


# Client Info
Source: https://docs.wandb.ai/support/weave/tags/client-info



<Card title="How can I disable client information capture?" href="/support/weave/articles/how-can-i-disable-client-information-cap">
  To disable client information capture, pass the setting during W\&B Weave client initialization: weave.init("entity/proje ...
</Card>

<Card title="What information does Weave capture for a function?" href="/support/weave/articles/what-information-does-weave-capture-for">
  A function can be designated as a Weave Op either manually through a decorator or automatically as part of an enabled in ...
</Card>


# Code Capture
Source: https://docs.wandb.ai/support/weave/tags/code-capture



<Card title="How can I disable code capture?" href="/support/weave/articles/how-can-i-disable-code-capture">
  To disable code capture, use one of the following methods: During W\&B Weave client initialization, set capture\_code to F ...
</Card>

<Card title="What information does Weave capture for a function?" href="/support/weave/articles/what-information-does-weave-capture-for">
  A function can be designated as a Weave Op either manually through a decorator or automatically as part of an enabled in ...
</Card>


# Data Capture
Source: https://docs.wandb.ai/support/weave/tags/data-capture



<Card title="How is Weave data ingestion calculated?" href="/support/weave/articles/how-is-weave-data-ingestion-calculated">
  Weave defines ingested bytes as bytes that Weave receives, processes, and stores on your behalf. Ingested bytes include  ...
</Card>

<Card title="What information does Weave capture for a function?" href="/support/weave/articles/what-information-does-weave-capture-for">
  A function can be designated as a Weave Op either manually through a decorator or automatically as part of an enabled in ...
</Card>


# Evaluation
Source: https://docs.wandb.ai/support/weave/tags/evaluation



<Card title="Long eval clean up times" href="/support/weave/articles/long-eval-clean-up-times">
  This page describes two methods to use together to reduce long clean-up times when you run W\&B Weave evaluations with la ...
</Card>

<Card title="What is pairwise evaluation and how do I do it?" href="/support/weave/articles/what-is-pairwise-evaluation-and-how-do-i">
  When you score models in a Weave evaluation, absolute value metrics (for example, 9/10 for Model A and 8/10 for Model B) ...
</Card>


# Performance
Source: https://docs.wandb.ai/support/weave/tags/performance



<Card title="Long eval clean up times" href="/support/weave/articles/long-eval-clean-up-times">
  This page describes two methods to use together to reduce long clean-up times when you run W\&B Weave evaluations with la ...
</Card>

<Card title="OS errors - Too many open files" href="/support/weave/articles/os-errors-too-many-open-files">
  Errno 24 : Too many open files This error occurs when the number of open files exceeds the limit set by your operating s ...
</Card>

<Card title="Server response caching" href="/support/weave/articles/server-response-caching">
  Weave provides server response caching to improve performance when you run repeated queries or work with limited network ...
</Card>

<Card title="Trace data loss in worker processes" href="/support/weave/articles/trace-data-loss-in-worker-processes">
  Weave uploads trace data in background threads to minimize impact on your application's performance. However, you might  ...
</Card>

<Card title="Will Weave affect my function's execution speed?" href="/support/weave/articles/will-weave-affect-my-function-s-executio">
  The overhead of Weave logging is typically negligible compared to making a call to an LLM. To minimize Weave's impact on ...
</Card>


# System Info
Source: https://docs.wandb.ai/support/weave/tags/system-info



<Card title="How can I disable system information capture?" href="/support/weave/articles/how-can-i-disable-system-information-cap">
  To disable system information capture, pass the setting during Weave client initialization: weave.init("entity/project", ...
</Card>

<Card title="What information does Weave capture for a function?" href="/support/weave/articles/what-information-does-weave-capture-for">
  A function can be designated as a Weave Op either manually through a decorator or automatically as part of an enabled in ...
</Card>


# Trace Data
Source: https://docs.wandb.ai/support/weave/tags/trace-data



<Card title="Trace data is truncated" href="/support/weave/articles/trace-data-is-truncated">
  Sometimes, large trace data is cut off in the Weave UI. This happens because the default trace output is a raw, custom P ...
</Card>

<Card title="Trace data loss in worker processes" href="/support/weave/articles/trace-data-loss-in-worker-processes">
  Weave uploads trace data in background threads to minimize impact on your application's performance. However, you might  ...
</Card>

<Card title="Trace pages load slowly" href="/support/weave/articles/trace-pages-load-slowly">
  If trace pages load slowly, reduce the number of rows displayed to improve load time. The default value is 50. You can r ...
</Card>

<Card title="Why doesn't `.call()` raise exceptions?" href="/support/weave/articles/weave-call-does-not-raise-exceptions">
  By default, Weave's .call() method captures exceptions and stores them in call.exception instead of raising them. This i ...
</Card>


# UI Rendering
Source: https://docs.wandb.ai/support/weave/tags/ui-rendering



<Card title="How do I render Markdown in the UI?" href="/support/weave/articles/how-do-i-render-markdown-in-the-ui">
  Before saving, wrap your string with weave.Markdown(...), then use weave.publish(...) to store it. The object's type det ...
</Card>

<Card title="How do I render Python datetime values in the UI?" href="/support/weave/articles/how-do-i-render-python-datetime-values-i">
  Use Python's datetime.datetime with timezone information, and publish the object using weave.publish(...). Weave recogni ...
</Card>


# Dataset Create
Source: https://docs.wandb.ai/weave/reference/service-api/datasets/dataset-create

/weave/reference/service-api/openapi.json post /v2/{entity}/{project}/datasets
Create a dataset object.



# Dataset Delete
Source: https://docs.wandb.ai/weave/reference/service-api/datasets/dataset-delete

/weave/reference/service-api/openapi.json delete /v2/{entity}/{project}/datasets/{object_id}
Delete a dataset object. If digests are provided, only those versions are deleted. Otherwise, all versions are deleted.



# Dataset List
Source: https://docs.wandb.ai/weave/reference/service-api/datasets/dataset-list

/weave/reference/service-api/openapi.json get /v2/{entity}/{project}/datasets
List dataset objects.



# Dataset Read
Source: https://docs.wandb.ai/weave/reference/service-api/datasets/dataset-read

/weave/reference/service-api/openapi.json get /v2/{entity}/{project}/datasets/{object_id}/versions/{digest}
Get a dataset object.



# Eval Results Query
Source: https://docs.wandb.ai/weave/reference/service-api/eval-results/eval-results-query

/weave/reference/service-api/openapi.json post /v2/{entity}/{project}/eval_results/query
Read grouped evaluation result rows for one or more evaluations.



# Evaluation Run Create
Source: https://docs.wandb.ai/weave/reference/service-api/evaluation-runs/evaluation-run-create

/weave/reference/service-api/openapi.json post /v2/{entity}/{project}/evaluation_runs
Create an evaluation run.



# Evaluation Run Delete
Source: https://docs.wandb.ai/weave/reference/service-api/evaluation-runs/evaluation-run-delete

/weave/reference/service-api/openapi.json delete /v2/{entity}/{project}/evaluation_runs
Delete evaluation runs.



# Evaluation Run Finish
Source: https://docs.wandb.ai/weave/reference/service-api/evaluation-runs/evaluation-run-finish

/weave/reference/service-api/openapi.json post /v2/{entity}/{project}/evaluation_runs/{evaluation_run_id}/finish
Finish an evaluation run.



# Evaluation Run List
Source: https://docs.wandb.ai/weave/reference/service-api/evaluation-runs/evaluation-run-list

/weave/reference/service-api/openapi.json get /v2/{entity}/{project}/evaluation_runs
List evaluation runs.



# Evaluation Run Read
Source: https://docs.wandb.ai/weave/reference/service-api/evaluation-runs/evaluation-run-read

/weave/reference/service-api/openapi.json get /v2/{entity}/{project}/evaluation_runs/{evaluation_run_id}
Read an evaluation run.



# Image Create
Source: https://docs.wandb.ai/weave/reference/service-api/images/image-create

/weave/reference/service-api/openapi.json post /image/create



# Model Create
Source: https://docs.wandb.ai/weave/reference/service-api/models/model-create

/weave/reference/service-api/openapi.json post /v2/{entity}/{project}/models
Create a model object.



# Model Delete
Source: https://docs.wandb.ai/weave/reference/service-api/models/model-delete

/weave/reference/service-api/openapi.json delete /v2/{entity}/{project}/models/{object_id}
Delete a model object. If digests are provided, only those versions are deleted. Otherwise, all versions are deleted.



# Model List
Source: https://docs.wandb.ai/weave/reference/service-api/models/model-list

/weave/reference/service-api/openapi.json get /v2/{entity}/{project}/models
List model objects.



# Model Read
Source: https://docs.wandb.ai/weave/reference/service-api/models/model-read

/weave/reference/service-api/openapi.json get /v2/{entity}/{project}/models/{object_id}/versions/{digest}
Get a model object.



# Op Create
Source: https://docs.wandb.ai/weave/reference/service-api/ops/op-create

/weave/reference/service-api/openapi.json post /v2/{entity}/{project}/ops
Create an op object.



# Op Delete
Source: https://docs.wandb.ai/weave/reference/service-api/ops/op-delete

/weave/reference/service-api/openapi.json delete /v2/{entity}/{project}/ops/{object_id}
Delete an op object. If digests are provided, only those versions are deleted. Otherwise, all versions are deleted.



# Op List
Source: https://docs.wandb.ai/weave/reference/service-api/ops/op-list

/weave/reference/service-api/openapi.json get /v2/{entity}/{project}/ops
List op objects.



# Op Read
Source: https://docs.wandb.ai/weave/reference/service-api/ops/op-read

/weave/reference/service-api/openapi.json get /v2/{entity}/{project}/ops/{object_id}/versions/{digest}
Get an op object.



# Prediction Create
Source: https://docs.wandb.ai/weave/reference/service-api/predictions/prediction-create

/weave/reference/service-api/openapi.json post /v2/{entity}/{project}/predictions
Create a prediction.



# Prediction Delete
Source: https://docs.wandb.ai/weave/reference/service-api/predictions/prediction-delete

/weave/reference/service-api/openapi.json delete /v2/{entity}/{project}/predictions
Delete predictions.



# Prediction Finish
Source: https://docs.wandb.ai/weave/reference/service-api/predictions/prediction-finish

/weave/reference/service-api/openapi.json post /v2/{entity}/{project}/predictions/{prediction_id}/finish
Finish a prediction.



# Prediction List
Source: https://docs.wandb.ai/weave/reference/service-api/predictions/prediction-list

/weave/reference/service-api/openapi.json get /v2/{entity}/{project}/predictions
List predictions.



# Prediction Read
Source: https://docs.wandb.ai/weave/reference/service-api/predictions/prediction-read

/weave/reference/service-api/openapi.json get /v2/{entity}/{project}/predictions/{prediction_id}
Read a prediction.



# Link To Registry
Source: https://docs.wandb.ai/weave/reference/service-api/registry/link-to-registry

/weave/reference/service-api/openapi.json post /link_to_registry



# Scorer Create
Source: https://docs.wandb.ai/weave/reference/service-api/scorers/scorer-create

/weave/reference/service-api/openapi.json post /v2/{entity}/{project}/scorers
Create a scorer object.



# Scorer Delete
Source: https://docs.wandb.ai/weave/reference/service-api/scorers/scorer-delete

/weave/reference/service-api/openapi.json delete /v2/{entity}/{project}/scorers/{object_id}
Delete a scorer object.



# Scorer List
Source: https://docs.wandb.ai/weave/reference/service-api/scorers/scorer-list

/weave/reference/service-api/openapi.json get /v2/{entity}/{project}/scorers
List scorer objects.



# Scorer Read
Source: https://docs.wandb.ai/weave/reference/service-api/scorers/scorer-read

/weave/reference/service-api/openapi.json get /v2/{entity}/{project}/scorers/{object_id}/versions/{digest}
Get a scorer object.



# Score Delete
Source: https://docs.wandb.ai/weave/reference/service-api/scores/score-delete

/weave/reference/service-api/openapi.json delete /v2/{entity}/{project}/scores
Delete scores.



# Score Read
Source: https://docs.wandb.ai/weave/reference/service-api/scores/score-read

/weave/reference/service-api/openapi.json get /v2/{entity}/{project}/scores/{score_id}
Read a score.



# W&B Launch
Source: https://docs.wandb.ai/platform/launch

Scale and manage ML workloads with W&B Launch by configuring jobs, queues, and agents on your compute infrastructure.



# Add job to queue
Source: https://docs.wandb.ai/platform/launch/add-job-to-queue

Add launch jobs to a W&B Launch queue using the App UI or CLI to schedule ML workloads on target resources.

This page describes how to add launch jobs to a launch queue. Adding a job to a queue submits it to run on the queue's target resource. You can schedule ML workloads against the compute environment your team configured.

<Note>
  Ensure that you, or someone on your team, already configured a launch queue. For more information, see [Set up Launch](/platform/launch/set-up-launch/).
</Note>

## Add jobs to your queue

Add jobs to your queue interactively with the W\&B App or programmatically with the W\&B CLI.

<Tabs>
  <Tab title="W&B App">
    To add a job to your queue interactively with the W\&B App:

    1. Navigate to your W\&B project page.
    2. Select the **Jobs** icon in the project sidebar:

    <Frame>
      <img alt="Project Jobs tab" />
    </Frame>

    3. The **Jobs** page displays a list of W\&B launch jobs created from previously executed W\&B runs.

    <Frame>
      <img alt="Jobs listing" />
    </Frame>

    4. Select the **Launch** button next to the job name. A modal appears.
    5. From the **Job version** dropdown, select the version of the launch job you want to use. Launch jobs are versioned like any other [W\&B artifact](/models/artifacts/create-a-new-artifact-version/). W\&B creates different versions of the same launch job if you modify the software dependencies or source code used to run the job.
    6. Within the **Overrides** section, provide new values for any inputs configured for your launch job. Common overrides include a new entrypoint command, arguments, or values in the `wandb.Run.config` of your new W\&B run.

    <Frame>
      <img alt="Queue configuration" />
    </Frame>

    To copy and paste values from other W\&B runs that used your launch job, select the **Paste from...** button.
    7\. From the **Queue** dropdown, select the name of the launch queue you want to add your launch job to.
    8\. Use the **Job Priority** dropdown to specify the priority of your launch job. If the launch queue doesn't support prioritization, W\&B sets the priority to `Medium`.
    9\. Optional: Follow this step only if your team admin created a queue config template. Within the **Queue Configurations** field, provide values for configuration options that your team admin created. For example, the following image shows a team admin who configured AWS instance types that the team can use. In this case, team members can select either the `ml.m4.xlarge` or `ml.p3.xlarge` compute instance type to train their model.

    <Frame>
      <img alt="Config template selection" />
    </Frame>

    10. Select the **Destination project**, where the resulting run appears. This project must belong to the same entity as the queue.
    11. Select the **Launch now** button.

    Your job is now queued and runs on the queue's target resource. The resulting run appears in the destination project you selected.
  </Tab>

  <Tab title="W&B CLI">
    Use the `wandb launch` command to add jobs to a queue. For example, using the script from the [Quickstart](/platform/launch/walkthrough/) guide, create a JSON file with the following hyperparameter overrides:

    ```json title="config.json" theme={null}
    {
      "overrides": {
          "args": [],
          "run_config": {
              "learning_rate": 0,
              "epochs": 0
          },   
          "entry_point": []
      }
    }
    ```

    <Note>
      W\&B Launch uses the default parameters if you don't provide a JSON configuration file.
    </Note>

    To override the queue configuration, or if your launch queue doesn't have a configuration resource defined, specify the `resource_args` key in your `config.json` file. For example, continuing the preceding example, your `config.json` file might look similar to the following:

    ```json title="config.json" theme={null}
    {
      "overrides": {
          "args": [],
          "run_config": {
              "learning_rate": 0,
              "epochs": 0
          },
          "entry_point": []
      },
      "resource_args": {
            "[RESOURCE-TYPE]" : {
                "[KEY]": "[VALUE]"
            }
      }
    }
    ```

    Replace the bracketed placeholders with your own values.

    Provide the name of the queue for the `queue` (`-q`) flag, the name of the job for the `job` (`-j`) flag, and the path to the configuration file for the `config` (`-c`) flag:

    ```bash theme={null}
    wandb launch -j [JOB-NAME] -q [QUEUE-NAME] \
    -e [ENTITY-NAME] -c path/to/config.json
    ```

    If you work within a W\&B team, specify the `entity` flag (`-e`) to indicate which entity the queue uses.

    After the command succeeds, your job is added to the specified queue and runs on the queue's target resource.
  </Tab>
</Tabs>


# Create and deploy jobs
Source: https://docs.wandb.ai/platform/launch/create-and-deploy-jobs

Create and deploy machine learning jobs using W&B Launch to run workloads on your target compute resources.



# Create a launch job
Source: https://docs.wandb.ai/platform/launch/create-launch-job

Create W&B Launch jobs from Git repositories, code artifacts, or Docker images using the wandb launch command.

<Card title="Try in Colab" href="https://colab.research.google.com/drive/1wX0OSVxZJDHRsZaOaOEDx-lLUrO1hHgP" icon="python" />

This page shows you how to create W\&B Launch jobs so you can reproduce W\&B runs on demand. Launch jobs are blueprints for reproducing W\&B runs. Jobs are W\&B Artifacts that capture the source code, dependencies, and inputs required to execute a workload.

Use this page when you want to package a workload from a Git repository, a local code directory, or a pre-built Docker image so that you can re-run it later with different parameters or on different infrastructure.

Create and run jobs with the `wandb launch` command.

<Note>
  To create a job without submitting it for execution, use the `wandb job create` command. See the [command reference docs](/models/ref/cli/wandb-job/wandb-job-create) for more information.
</Note>

## Git jobs

Use a Git-based job when your source code lives in a remote repository and you want Launch to clone it at a specific commit, branch, or tag. W\&B Launch clones code and other tracked assets from a commit, branch, or tag in a remote Git repository. Use the `--uri` or `-u` flag to specify the URI containing the code, along with an optional `--build-context` flag to specify a subdirectory.

Run a hello world job from a Git repository with the following command:

```bash theme={null}
wandb launch --uri "https://github.com/wandb/launch-jobs.git" --build-context jobs/hello_world --dockerfile Dockerfile.wandb --project "hello-world" --job-name "hello-world" --entry-point "python job.py"
```

The command does the following:

1. Clones the [W\&B Launch jobs repository](https://github.com/wandb/launch-jobs) to a temporary directory.
2. Creates a job named `hello-world-git` in the `hello` project. The job is associated with the commit at the head of the default branch of the repository.
3. Builds a container image from the `jobs/hello_world` directory and the `Dockerfile.wandb`.
4. Starts the container and runs `python job.py`.

After the command completes, you have a reusable Git-based job in the `hello-world` project that other users or automations can re-run with `wandb launch`.

To build a job from a specific branch or commit hash, append the `-g`, `--git-hash` argument. For a full list of arguments, run `wandb launch --help`.

### Remote URL format

The Git remote associated with a Launch job can be either an HTTPS or an SSH URL. The URL type determines the protocol used to fetch job source code.

| Remote URL type | URL format                                       | Requirements for access and authentication                |
| --------------- | ------------------------------------------------ | --------------------------------------------------------- |
| HTTPS           | `https://github.com/organization/repository.git` | Username and password to authenticate with the Git remote |
| SSH             | `git@github.com:organization/repository.git`     | SSH key to authenticate with the Git remote               |

Note that the exact URL format varies by hosting provider. Jobs created with `wandb launch --uri` use the transfer protocol specified in the provided `--uri`.

## Code artifact jobs

Use a code artifact job when your source lives in a local directory rather than a remote Git repository. You can create jobs from any source code stored in a W\&B Artifact. Use a local directory with the `--uri` or `-u` argument to create a new code artifact and job.

To get started, create an empty directory and add a Python script named `main.py` with the following content:

```python theme={null}
import wandb

with wandb.init() as run:
    run.log({"metric": 0.5})
```

Add a file `requirements.txt` with the following content:

```text theme={null}
wandb>=0.17.1
```

Log the directory as a code artifact and launch a job with the following command:

```bash theme={null}
wandb launch --uri . --job-name hello-world-code --project launch-quickstart --entry-point "python main.py"
```

The preceding command does the following:

1. Logs the current directory as a code artifact named `hello-world-code`.
2. Creates a job named `hello-world-code` in the `launch-quickstart` project.
3. Builds a container image from the current directory and Launch's default Dockerfile. The default Dockerfile installs the `requirements.txt` file and sets the entry point to `python main.py`.

After the command completes, your local code is captured as a W\&B code artifact and is associated with a reusable job in the `launch-quickstart` project.

## Image jobs

Alternatively, you can build jobs from pre-made Docker images. This approach is useful when you already have a build system for your ML code, or when you don't expect to adjust the code or requirements for the job but do want to experiment with hyperparameters or different infrastructure scales.

Launch pulls the image from a Docker registry and runs it with the specified entry point, or the default entry point if none is specified. Pass a full image tag to the `--docker-image` option to create and run a job from a Docker image.

Run a job from a pre-made image with the following command:

```bash theme={null}
wandb launch --docker-image "wandb/job_hello_world:main" --project "hello-world"           
```

## Automatic job creation

In addition to jobs you create explicitly with `wandb launch`, W\&B can create jobs for you as a side effect of running tracked code. W\&B automatically creates and tracks a job for any run with tracked source code, even if that run wasn't created with Launch. Runs are considered to have tracked source code if any of the three following conditions are met:

* The run has an associated Git remote and commit hash.
* The run logged a code artifact. See [`Run.log_code`](/models/ref/python/experiments/run#log_code).
* The run was executed in a Docker container with the `WANDB_DOCKER` environment variable set to an image tag.

If W\&B automatically creates your Launch job from a W\&B run, W\&B infers the Git remote URL from the local Git repository.

### Launch job names

By default, W\&B automatically generates a job name for you. The name depends on how the job is created (GitHub, code artifact, or Docker image). Alternatively, you can define a Launch job's name with environment variables or with the W\&B Python SDK.

The following table describes the job naming convention used by default based on job source:

| Source        | Naming convention                       |
| ------------- | --------------------------------------- |
| GitHub        | `job-[GIT-REMOTE-URL]-[PATH-TO-SCRIPT]` |
| Code artifact | `job-[CODE-ARTIFACT-NAME]`              |
| Docker image  | `job-[IMAGE-NAME]`                      |

Name your job with a W\&B environment variable or with the W\&B Python SDK.

<Tabs>
  <Tab title="Environment variable">
    Set the `WANDB_JOB_NAME` environment variable to your preferred job name. For example:

    ```bash theme={null}
    WANDB_JOB_NAME=awesome-job-name
    ```
  </Tab>

  <Tab title="W&B Python SDK">
    Define the name of your job with `wandb.Settings`. Then pass this object when you initialize W\&B with `wandb.init()`. For example:

    ```python theme={null}
    settings = wandb.Settings(job_name="my-job-name")
    wandb.init(settings=settings)
    ```
  </Tab>
</Tabs>

<Note>
  For Docker image jobs, W\&B automatically adds the version alias as an alias to the job.
</Note>

## Containerization

Jobs run in a container. Image jobs use a pre-built Docker image, while Git and code artifact jobs require a container build step.

You can customize job containerization with arguments to `wandb launch` and files within the job source code. The following sections describe how to control the build context, supply a custom Dockerfile, and manage Python dependencies.

### Build context

The term build context refers to the tree of files and directories sent to the Docker daemon to build a container image. By default, Launch uses the root of the job source code as the build context. To specify a subdirectory as the build context, use the `--build-context` argument of `wandb launch` when you create and launch a job.

<Note>
  The `--build-context` argument is useful for working with Git jobs that refer to a monorepo with multiple projects. By specifying a subdirectory as the build context, you can build a container image for a specific project within the monorepo.

  See [Git jobs](#git-jobs) for a demonstration of how to use the `--build-context` argument with the official W\&B Launch jobs repository.
</Note>

### Dockerfile

The Dockerfile is a text file that contains instructions for building a Docker image. By default, Launch uses a default Dockerfile that installs the `requirements.txt` file. To use a custom Dockerfile, specify the path to the file with the `--dockerfile` argument of `wandb launch`.

Specify the Dockerfile path relative to the build context. For example, if the build context is `jobs/hello_world`, and the Dockerfile is located in the `jobs/hello_world` directory, set the `--dockerfile` argument to `Dockerfile.wandb`. See [Git jobs](#git-jobs) for a demonstration of how to use the `--dockerfile` argument with the official W\&B Launch jobs repository.

### Requirements file

If no custom Dockerfile is provided, Launch looks in the build context for Python dependencies to install. If a `requirements.txt` file is found at the root of the build context, Launch installs the dependencies listed in the file. Otherwise, if a `pyproject.toml` file is found, Launch installs dependencies from the `project.dependencies` section.


# Launch integration guides
Source: https://docs.wandb.ai/platform/launch/integration-guides

Browse integration guides for configuring W&B Launch with Docker, Kubernetes, SageMaker, and Vertex AI.



# Manage job inputs
Source: https://docs.wandb.ai/platform/launch/job-inputs

Programmatically manage and configure job inputs such as hyperparameters and file-based configs for W&B Launch jobs.

The core experience of Launch is experimenting with different job inputs such as hyperparameters and datasets, and routing these jobs to appropriate hardware. After you create a job, users beyond the original author can adjust these inputs through the W\&B UI or CLI. For information about how to set job inputs when launching from the CLI or UI, see the [Enqueue jobs](./add-job-to-queue) guide.

This guide describes how to programmatically control which inputs you can adjust for a job, so that you expose only the parameters you want end users to change. By default, W\&B jobs capture the entire `Run.config` as the inputs to a job, but the Launch SDK provides a function to control select keys in the run config or to specify JSON or YAML files as inputs.

<Note>
  Launch SDK functions require `wandb-core`. For more information, see the [`wandb-core` README](https://github.com/wandb/wandb/blob/main/core/README).
</Note>

## Reconfigure the `Run` object

By default, you can reconfigure the `Run` object returned by `wandb.init()` in a job. Use the Launch SDK to customize which parts of the `Run.config` object you can reconfigure when launching the job, so that you can hide internal settings while exposing the parameters that matter to end users.

```python theme={null}
import wandb
from wandb.sdk import launch

# Required for launch sdk use.
wandb.require("core")

config = {
    "trainer": {
        "learning_rate": 0.01,
        "batch_size": 32,
        "model": "resnet",
        "dataset": "cifar10",
        "private": {
            "key": "value",
        },
    },
    "seed": 42,
}


with wandb.init(config=config):
    launch.manage_wandb_config(
        include=["trainer"], 
        exclude=["trainer.private"],
    )
    # Etc.
```

The function `launch.manage_wandb_config()` configures the job to accept input values for the `Run.config` object. The optional `include` and `exclude` options take path prefixes within the nested config object. This is useful if, for example, a job uses a library whose options you don't want to expose to end users.

If you provide `include` prefixes, only paths within the config that match an `include` prefix accept input values. If you provide `exclude` prefixes, paths that match the `exclude` list are filtered out of the input values. If a path matches both an `include` and an `exclude` prefix, the `exclude` prefix takes precedence.

In the preceding example, the path `["trainer.private"]` filters out the `private` key from the `trainer` object, and the path `["trainer"]` filters out all keys not under the `trainer` object.

<Note>
  Use a `\`-escaped `.` to filter out keys with a `.` in their name.

  For example, `r"trainer\.private"` filters out the `trainer.private` key rather than the `private` key under the `trainer` object.

  The `r` prefix in the preceding example denotes a raw string.
</Note>

If you package and run the preceding code as a job, the input types of the job are:

```json theme={null}
{
    "trainer": {
        "learning_rate": "float",
        "batch_size": "int",
        "model": "str",
        "dataset": "str",
    },
}
```

When launching the job from the W\&B CLI or UI, you can override only the four `trainer` parameters.

### Access run config inputs

Jobs launched with run config inputs can access the input values through the `Run.config`. The `Run` returned by `wandb.init()` in the job code has the input values automatically set. To load the run config input values anywhere in the job code, use `launch.load_wandb_config()`:

```python theme={null}
from wandb.sdk import launch

run_config_overrides = launch.load_wandb_config()
```

## Reconfigure a file

The Launch SDK can also manage input values stored in config files in the job code. This is a common pattern in many deep learning and large language model use cases, such as this [torchtune](https://github.com/meta-pytorch/torchtune/blob/main/recipes/configs/llama3/8B_lora.yaml) example or this [Axolotl config](https://github.com/OpenAccess-AI-Collective/axolotl/blob/main/examples/llama-3/qlora-fsdp-70b.yaml).

<Note>
  [Sweeps on Launch](/platform/launch/sweeps-on-launch/) does not support the use of config file inputs as sweep parameters. Sweep parameters must be controlled through the `Run.config` object.
</Note>

Use the `launch.manage_config_file()` function to add a config file as an input to the Launch job, giving you access to edit values within the config file when launching the job.

By default, no run config inputs are captured if you use `launch.manage_config_file()`. Calling `launch.manage_wandb_config()` overrides this behavior.

Consider the following example:

```python theme={null}
import yaml
import wandb
from wandb.sdk import launch

# Required for launch sdk use.
wandb.require("core")

launch.manage_config_file("config.yaml")

with open("config.yaml", "r") as f:
    config = yaml.safe_load(f)

with wandb.init(config=config):
    # Etc.
    pass
```

Imagine you run the code with an adjacent file `config.yaml`:

```yaml theme={null}
learning_rate: 0.01
batch_size: 32
model: resnet
dataset: cifar10
```

The call to `launch.manage_config_file()` adds the `config.yaml` file as an input to the job, making it reconfigurable when launching from the W\&B CLI or UI.

Use the `include` and `exclude` keyword arguments to filter the acceptable input keys for the config file in the same way as `launch.manage_wandb_config()`.

### Access config file inputs

When you call `launch.manage_config_file()` in a run created by Launch, `launch` patches the contents of the config file with the input values. The patched config file is available in the job environment.

<Warning>
  Call `launch.manage_config_file()` before reading the config file in the job code to ensure input values are used.
</Warning>

## Customize a job's launch drawer UI

Beyond filtering which inputs you expose, you can define a schema for a job's inputs to create a custom UI for launching the job. This presents structured fields, validation hints, and dropdowns in the launch drawer instead of freeform text entry. To define a job's schema, include it in the call to `launch.manage_wandb_config()` or `launch.manage_config_file()`. The schema can be either a Python `dict` in the form of a [JSON Schema](https://json-schema.org/understanding-json-schema/reference) or a Pydantic model class.

<Warning>
  Job input schemas don't validate inputs. They only define the UI in the launch drawer.
</Warning>

<Tabs>
  <Tab title="JSON schema">
    The following example shows a schema with these properties:

    * `seed`, an integer.
    * `trainer`, a dictionary with some keys specified:
      * `trainer.learning_rate`, a float that must be greater than zero.
      * `trainer.batch_size`, an integer that must be either 16, 64, or 256.
      * `trainer.dataset`, a string that must be either `cifar10` or `cifar100`.

    ```python theme={null}
    schema = {
        "type": "object",
        "properties": {
            "seed": {
              "type": "integer"
            }
            "trainer": {
                "type": "object",
                "properties": {
                    "learning_rate": {
                        "type": "number",
                        "description": "Learning rate of the model",
                        "exclusiveMinimum": 0,
                    },
                    "batch_size": {
                        "type": "integer",
                        "description": "Number of samples per batch",
                        "enum": [16, 64, 256]
                    },
                    "dataset": {
                        "type": "string",
                        "description": "Name of the dataset to use",
                        "enum": ["cifar10", "cifar100"]
                    }
                }
            }
        }
    }

    launch.manage_wandb_config(
        include=["seed", "trainer"], 
        exclude=["trainer.private"],
        schema=schema,
    )
    ```

    In general, the following JSON Schema attributes are supported:

    | Attribute          | Required | Notes                                                      |
    | ------------------ | -------- | ---------------------------------------------------------- |
    | `type`             | Yes      | Must be one of `number`, `integer`, `string`, or `object`  |
    | `title`            | No       | Overrides the property's display name                      |
    | `description`      | No       | Gives the property helper text                             |
    | `enum`             | No       | Creates a dropdown select instead of a freeform text entry |
    | `minimum`          | No       | Allowed only if `type` is `number` or `integer`            |
    | `maximum`          | No       | Allowed only if `type` is `number` or `integer`            |
    | `exclusiveMinimum` | No       | Allowed only if `type` is `number` or `integer`            |
    | `exclusiveMaximum` | No       | Allowed only if `type` is `number` or `integer`            |
    | `properties`       | No       | If `type` is `object`, defines nested configurations       |
  </Tab>

  <Tab title="Pydantic model">
    The following example shows a schema with these properties:

    * `seed`, an integer.
    * `trainer`, a schema with some sub-attributes specified:
      * `trainer.learning_rate`, a float that must be greater than zero.
      * `trainer.batch_size`, an integer that must be between 1 and 256, inclusive.
      * `trainer.dataset`, a string that must be either `cifar10` or `cifar100`.

    ```python theme={null}
    class DatasetEnum(str, Enum):
        cifar10 = "cifar10"
        cifar100 = "cifar100"

    class Trainer(BaseModel):
        learning_rate: float = Field(gt=0, description="Learning rate of the model")
        batch_size: int = Field(ge=1, le=256, description="Number of samples per batch")
        dataset: DatasetEnum = Field(title="Dataset", description="Name of the dataset to use")

    class Schema(BaseModel):
        seed: int
        trainer: Trainer

    launch.manage_wandb_config(
        include=["seed", "trainer"],
        exclude=["trainer.private"],
        schema=Schema,
    )
    ```

    You can also use an instance of the class:

    ```python theme={null}
    t = Trainer(learning_rate=0.01, batch_size=32, dataset=DatasetEnum.cifar10)
    s = Schema(seed=42, trainer=t)
    launch.manage_wandb_config(
        include=["seed", "trainer"],
        exclude=["trainer.private"],
        input_schema=s,
    )
    ```
  </Tab>
</Tabs>

Adding a job input schema creates a structured form in the launch drawer for users launching the job.

<Frame>
  <img alt="Job input schema form" />
</Frame>


# Launch FAQ
Source: https://docs.wandb.ai/platform/launch/launch-faq

Find answers to frequently asked questions about W&B Launch setup, configuration, and troubleshooting.



# Are there best practices for using Launch effectively?
Source: https://docs.wandb.ai/platform/launch/launch-faq/best_practices_launch_effectively



Follow these practices to avoid common setup errors and keep your Launch workflows maintainable:

* Create the queue before you start the agent to streamline configuration. Otherwise, the agent fails with errors until you add a queue.
* Create a W\&B service account to start the agent so that it doesn't link to an individual user account.
* Use `wandb.Run.config` to manage hyperparameters, which lets you overwrite values during job re-runs. For more information about using `argparse`, see [Set the configuration with argparse](/models/track/config/#set-the-configuration-with-argparse).


# I don't want W&B to build a container for me, can I still use Launch?
Source: https://docs.wandb.ai/platform/launch/launch-faq/build_container_launch



To launch a pre-built Docker image, replace the `<>` placeholders with your values and run:

```bash theme={null}
wandb launch -d <docker-image-uri> -q <queue-name> -E <entrypoint>
```

This command creates a job and starts a run.

To create a job from an image, run:

```bash theme={null}
wandb job create image <image-name> -p <project> -e <entity>
```


# I don't like clicking, can I use Launch without going through the UI?
Source: https://docs.wandb.ai/platform/launch/launch-faq/clicking_launch_without_going_ui



Yes. The `wandb` CLI includes a `launch` subcommand for launching jobs. To see usage details, run:

```bash theme={null}
wandb launch --help
```


# How do I control who can push to a queue?
Source: https://docs.wandb.ai/platform/launch/launch-faq/control_push_queue



Queues are specific to a user team. Define the owning entity when you create the queue. To restrict access, modify team membership.


# When multiple jobs in a Docker queue download the same artifact, is any caching used, or is it re-downloaded every run?
Source: https://docs.wandb.ai/platform/launch/launch-faq/docker_queues_run_multiple_jobs_download_same_artifact_useartifact



Launch jobs share no cache. Each launch job runs independently. To share a cache, configure the queue or agent to mount one with Docker arguments in the queue configuration.

For specific use cases, you can also mount the W\&B artifacts cache as a persistent volume.


# Can I specify a Dockerfile and let W&B build a Docker image for me?
Source: https://docs.wandb.ai/platform/launch/launch-faq/dockerfile_let_wb_build_docker_image_me



Yes. You can provide your own Dockerfile and have W\&B Launch build the image for your run. This lets you control the build environment while still using Launch to queue and run jobs. This approach works well for projects with stable requirements but changing codebases.

<Warning>
  Format your Dockerfile to use mounts. For details, see the [Docker mounts documentation](https://docs.docker.com/build/guide/mounts/).
</Warning>

After you configure the Dockerfile, specify it to W\&B in one of the following ways:

* Use `Dockerfile.wandb`
* Use W\&B CLI
* Use W\&B App

<Tabs>
  <Tab title="Dockerfile.wandb">
    Include a `Dockerfile.wandb` file in the same directory as the W\&B run's entrypoint. W\&B uses this file instead of the built-in Dockerfile.
  </Tab>

  <Tab title="W&B CLI">
    To queue a job, use the `--dockerfile` flag with the `wandb launch` command:

    ```bash theme={null}
    wandb launch --dockerfile path/to/Dockerfile
    ```
  </Tab>

  <Tab title="W&B App">
    When you add a job to a queue in the W\&B App, provide the Dockerfile path in the **Overrides** section. Enter it as a key-value pair with `"dockerfile"` as the key and the path as the value.

    The following JSON shows how to include a Dockerfile from a local directory:

    ```json title="Launch job W&B App" theme={null}
    {
      "args": [],
      "run_config": {
        "lr": 0,
        "batch_size": 0,
        "epochs": 0
      },
      "entrypoint": [],
      "dockerfile": "./Dockerfile"
    }
    ```
  </Tab>
</Tabs>


# Can Launch automatically provision (and spin down) compute resources for me in the target environment?
Source: https://docs.wandb.ai/platform/launch/launch-faq/launch_automatically_provision_spin_compute_resources_target_environment



Whether Launch automatically provisions and scales compute resources depends on the target environment:

* **Amazon SageMaker and Vertex AI**: These platforms provision resources automatically.
* **Kubernetes**: Autoscalers adjust resources based on demand. W\&B solution architects can help you configure your Kubernetes infrastructure to enable retries, autoscaling, and spot instance node pools.

For support, contact [W\&B support](mailto:support@wandb.com) or use your shared Slack channel.


# How W&B Launch builds images
Source: https://docs.wandb.ai/platform/launch/launch-faq/launch_build_images



This page explains how W\&B Launch builds container images for your jobs, so you can predict which build actions Launch performs and configure your queue or job accordingly.

The steps for building an image depend on the job source and the specified accelerator base image in the resource configuration.

<Note>
  When configuring a queue or submitting a job, include a base accelerator image in the queue or job resource configuration:

  ```json theme={null}
  {
      "builder": {
          "accelerator": {
              "base_image": "[IMAGE-NAME]"
          }
      }
  }
  ```
</Note>

The build process includes the following actions based on the job type and provided accelerator base image:

\| | Install Python using apt | Install Python packages | Create a user and workdir | Copy code into image | Set entrypoint | |


# Is `wandb launch -d` or `wandb job create image` uploading a whole Docker artifact and not pulling from a registry?
Source: https://docs.wandb.ai/platform/launch/launch-faq/launch_d_wandb_job_create_image_uploading_whole_docker



No, the `wandb launch -d` command doesn't upload images to a registry. Upload images to a registry separately. Follow these steps:

1. Build an image.
2. Push the image to a registry.

The workflow is as follows. Replace `[REPO-URL]` with your registry URL and `[TAG]` with the image tag:

```bash theme={null}
docker build -t [REPO-URL]:[TAG] .
docker push [REPO-URL]:[TAG]
wandb launch -d [REPO-URL]:[TAG]
```

The launch agent then starts a job pointing to the specified container. For examples of configuring agent access to pull images from a container registry, see [Advanced agent setup](/platform/launch/setup-agent-advanced/#agent-configuration).

For Kubernetes, ensure that the Pods in the cluster have access to the registry where the image is pushed.


# Does Launch support parallelization? How can I limit the resources consumed by a job?
Source: https://docs.wandb.ai/platform/launch/launch-faq/launch_support_parallelization_limit_resources_consumed_job



Launch supports scaling jobs across multiple GPUs and nodes. For details, see the [Volcano integration guide](/models/integrations).

Each launch agent has a `max_jobs` parameter that sets the maximum number of simultaneous jobs the agent can run. Multiple agents can point to a single queue, as long as they connect to a launching infrastructure that supports them.

You can set limits on CPU, GPU, memory, and other resources at the queue or job run level in the resource configuration. For information about setting up queues with resource limits on Kubernetes, see the [Kubernetes setup guide](/platform/launch/setup-launch-kubernetes/).

For sweeps, include the following block in the queue configuration to limit the number of concurrent runs:

```yaml title="queue config" theme={null}
  scheduler:
    num_workers: 4
```


# How to make W&B Launch work with TensorFlow on GPU
Source: https://docs.wandb.ai/platform/launch/launch-faq/launch_tensorflow_gpu



To run TensorFlow jobs on GPUs with W\&B Launch, specify a custom base image for the container build. This lets the resulting container access the GPU. Add an image tag under the `builder.accelerator.base_image` key in the resource configuration. For example:

```json theme={null}
{
    "gpus": "all",
    "builder": {
        "accelerator": {
            "base_image": "tensorflow/tensorflow:latest-gpu"
        }
    }
}
```

<Note>
  In versions before W\&B 0.15.6, use `cuda` instead of `accelerator` as the parent key for `base_image`.
</Note>


# How do I fix a "permission denied" error in Launch?
Source: https://docs.wandb.ai/platform/launch/launch-faq/launcherror_permission_denied



If you see the error `Launch Error: Permission denied`, you don't have permission to log to the project. Review the following causes and resolutions:

* You aren't logged in on this machine. Run [`wandb login`](/models/ref/cli/wandb-login) in the command line.
* The specified entity doesn't exist. The entity must be your username or the name of an existing team. If necessary, create a team on the [Subscriptions page](https://app.wandb.ai/billing).
* You lack project permissions. Ask the project creator to change the privacy setting to **Open** so you can log runs to the project.


# What permissions does the agent require in Kubernetes?
Source: https://docs.wandb.ai/platform/launch/launch-faq/permissions_agent_require_kubernetes



This page describes the Kubernetes permissions that the W\&B Launch agent requires to run jobs in your cluster. Use this information to configure role-based access control (RBAC) for the agent before you deploy it.

The following Kubernetes manifest creates a Role named `wandb-launch-agent` in the `wandb` namespace. This Role lets the agent create Pods, ConfigMaps, and Secrets, and access Pod logs in the `wandb` namespace. The `wandb-cluster-role` lets the agent create Pods, access Pod logs, create Secrets and Jobs, and check Job status across any specified namespace.


# What requirements does the accelerator base image have?
Source: https://docs.wandb.ai/platform/launch/launch-faq/requirements_accelerator_base_image



For W\&B Launch jobs that use an accelerator, provide a base image that includes the necessary accelerator components. This ensures that your job has the required GPU drivers, runtime libraries, and other hardware-specific dependencies to execute. The accelerator image must meet the following requirements:

* Compatibility with Debian (the Launch Dockerfile uses `apt-get` to install Python).
* Supported CPU and GPU hardware instruction set (confirm the CUDA version compatibility with the intended GPU).
* Compatibility between the supplied accelerator version and the packages in the machine learning algorithm.
* Installation of packages that require additional steps for hardware compatibility.


# How can admins restrict which users have modify access?
Source: https://docs.wandb.ai/platform/launch/launch-faq/restrict_access_modify_example



Control access to certain queue fields for users who aren't team administrators through [queue config templates](/platform/launch/setup-queue-advanced/). Team administrators define which fields non-admin users can view, and set the editing limits. Only team administrators can create or edit queues.


# Can you specify secrets for jobs or automations? For instance, an API key which you do not wish to be directly visible to users?
Source: https://docs.wandb.ai/platform/launch/launch-faq/secrets_jobsautomations_instance_api_key_wish_directly_visible



Yes. Follow these steps:

1. Create a Kubernetes Secret in the designated namespace for the runs:\
   `kubectl create secret -n [NAMESPACE] generic [SECRET-NAME] [SECRET-VALUE]`

2. After you create the secret, configure the queue to inject the secret when runs start. Only cluster administrators can view the secret. End users cannot see it.


# launch api
Source: https://docs.wandb.ai/platform/launch/launch-library/launch



<Card title="View the source code" href="https://www.github.com/wandb/wandb/tree/v0.20.1/wandb/sdk/launch/_launch.py#L249-L331" icon="github" />

Launch a W\&B launch experiment.

```python theme={null}
launch(
    api: Api,
    job: Optional[str] = None,
    entry_point: Optional[List[str]] = None,
    version: Optional[str] = None,
    name: Optional[str] = None,
    resource: Optional[str] = None,
    resource_args: Optional[Dict[str, Any]] = None,
    project: Optional[str] = None,
    entity: Optional[str] = None,
    docker_image: Optional[str] = None,
    config: Optional[Dict[str, Any]] = None,
    synchronous: Optional[bool] = (True),
    run_id: Optional[str] = None,
    repository: Optional[str] = None
) -> AbstractRun
```

| Arguments       | Description                                                                                                                                                                                                                                                                                                                                                                                                                                               |
| :-------------- | :-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `job`           | string reference to a wandb.Job eg: wandb/test/my-job:latest                                                                                                                                                                                                                                                                                                                                                                                              |
| `api`           | An instance of a wandb Api from wandb.apis.internal.                                                                                                                                                                                                                                                                                                                                                                                                      |
| `entry_point`   | Entry point to run within the project. Defaults to using the entry point used in the original run for wandb URIs, or main.py for git repository URIs.                                                                                                                                                                                                                                                                                                     |
| `version`       | For Git-based projects, either a commit hash or a branch name.                                                                                                                                                                                                                                                                                                                                                                                            |
| `name`          | Name run under which to launch the run.                                                                                                                                                                                                                                                                                                                                                                                                                   |
| `resource`      | Execution backend for the run.                                                                                                                                                                                                                                                                                                                                                                                                                            |
| `resource_args` | Resource related arguments for launching runs onto a remote backend. Will be stored on the constructed launch config under `resource_args`.                                                                                                                                                                                                                                                                                                               |
| `project`       | Target project to send launched run to                                                                                                                                                                                                                                                                                                                                                                                                                    |
| `entity`        | Target entity to send launched run to                                                                                                                                                                                                                                                                                                                                                                                                                     |
| `config`        | A dictionary containing the configuration for the run. May also contain resource specific arguments under the key "resource\_args".                                                                                                                                                                                                                                                                                                                       |
| `synchronous`   | Whether to block while waiting for a run to complete. Defaults to True. Note that if `synchronous` is False and `backend` is "local-container", this method will return, but the current process will block when exiting until the local run completes. If the current process is interrupted, any asynchronous runs launched via this method will be terminated. If `synchronous` is True and the run fails, the current process will error out as well. |
| `run_id`        | ID for the run (To ultimately replace the :name: field)                                                                                                                                                                                                                                                                                                                                                                                                   |
| `repository`    | string name of repository path for remote registry                                                                                                                                                                                                                                                                                                                                                                                                        |

#### Example:

```python theme={null}
from wandb.sdk.launch import launch

job = "wandb/jobs/Hello World:latest"
params = {"epochs": 5}
# Run W&B project and create a reproducible docker environment
# on a local host
api = wandb.apis.internal.Api()
launch(api, job, parameters=params)
```

| Returns                                                                                              | Description |
| :--------------------------------------------------------------------------------------------------- | :---------- |
| an instance of`wandb.launch.SubmittedRun` exposing information (e.g. run ID) about the launched run. |             |

| Raises                                                                                | Description |
| :------------------------------------------------------------------------------------ | :---------- |
| `wandb.exceptions.ExecutionError` If a run launched in blocking mode is unsuccessful. |             |


# launch_add
Source: https://docs.wandb.ai/platform/launch/launch-library/launch_add



<Card title="View the source code" href="https://www.github.com/wandb/wandb/tree/v0.20.1/wandb/sdk/launch/_launch_add.py#L34-L131" icon="github" />

Enqueue a W\&B launch experiment. With either a source uri, job or docker\_image.

```python theme={null}
launch_add(
    uri: Optional[str] = None,
    job: Optional[str] = None,
    config: Optional[Dict[str, Any]] = None,
    template_variables: Optional[Dict[str, Union[float, int, str]]] = None,
    project: Optional[str] = None,
    entity: Optional[str] = None,
    queue_name: Optional[str] = None,
    resource: Optional[str] = None,
    entry_point: Optional[List[str]] = None,
    name: Optional[str] = None,
    version: Optional[str] = None,
    docker_image: Optional[str] = None,
    project_queue: Optional[str] = None,
    resource_args: Optional[Dict[str, Any]] = None,
    run_id: Optional[str] = None,
    build: Optional[bool] = (False),
    repository: Optional[str] = None,
    sweep_id: Optional[str] = None,
    author: Optional[str] = None,
    priority: Optional[int] = None
) -> "public.QueuedRun"
```

| Arguments            | Description                                                                                                                                                         |
| :------------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `uri`                | URI of experiment to run. A wandb run uri or a Git repository URI.                                                                                                  |
| `job`                | string reference to a wandb.Job eg: wandb/test/my-job:latest                                                                                                        |
| `config`             | A dictionary containing the configuration for the run. May also contain resource specific arguments under the key "resource\_args"                                  |
| `template_variables` | A dictionary containing values of template variables for a run queue. Expected format of `{"VAR_NAME": VAR_VALUE}`                                                  |
| `project`            | Target project to send launched run to                                                                                                                              |
| `entity`             | Target entity to send launched run to                                                                                                                               |
| `queue`              | the name of the queue to enqueue the run to                                                                                                                         |
| `priority`           | the priority level of the job, where 1 is the highest priority                                                                                                      |
| `resource`           | Execution backend for the run: W\&B provides built-in support for "local-container" backend                                                                         |
| `entry_point`        | Entry point to run within the project. Defaults to using the entry point used in the original run for wandb URIs, or main.py for git repository URIs.               |
| `name`               | Name run under which to launch the run.                                                                                                                             |
| `version`            | For Git-based projects, either a commit hash or a branch name.                                                                                                      |
| `docker_image`       | The name of the docker image to use for the run.                                                                                                                    |
| `resource_args`      | Resource related arguments for launching runs onto a remote backend. Will be stored on the constructed launch config under `resource_args`.                         |
| `run_id`             | optional string indicating the id of the launched run                                                                                                               |
| `build`              | optional flag defaulting to false, requires queue to be set if build, an image is created, creates a job artifact, pushes a reference to that job artifact to queue |
| `repository`         | optional string to control the name of the remote repository, used when pushing images to a registry                                                                |
| `project_queue`      | optional string to control the name of the project for the queue. Primarily used for back compatibility with project scoped queues                                  |

#### Example:

```python theme={null}
from wandb.sdk.launch import launch_add

project_uri = "https://github.com/wandb/examples"
params = {"alpha": 0.5, "l1_ratio": 0.01}
# Run W&B project and create a reproducible docker environment
# on a local host
api = wandb.apis.internal.Api()
launch_add(uri=project_uri, parameters=params)
```

| Returns                                                                                                                                                                                                  | Description |
| :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :---------- |
| an instance of`wandb.api.public.QueuedRun` which gives information about the queued run, or if `wait_until_started` or `wait_until_finished` are called, gives access to the underlying Run information. |             |

| Raises                                         | Description |
| :--------------------------------------------- | :---------- |
| `wandb.exceptions.LaunchError` if unsuccessful |             |


# LaunchAgent
Source: https://docs.wandb.ai/platform/launch/launch-library/launchagent



<Card title="View the source code" href="https://www.github.com/wandb/wandb/tree/v0.20.1/wandb/sdk/launch/agent/agent.py#L164-L924" icon="github" />

Launch agent class which polls run given run queues and launches runs for wandb launch.

```python theme={null}
LaunchAgent(
    api: Api,
    config: Dict[str, Any]
)
```

| Arguments | Description                                           |
| :-------- | :---------------------------------------------------- |
| `api`     | Api object to use for making requests to the backend. |
| `config`  | Config dictionary for the agent.                      |

| Attributes               | Description                                              |
| :----------------------- | :------------------------------------------------------- |
| `num_running_jobs`       | Return the number of jobs not including schedulers.      |
| `num_running_schedulers` | Return just the number of schedulers.                    |
| `thread_ids`             | Returns a list of keys running thread ids for the agent. |

## Methods

### `check_sweep_state`

[View source](https://www.github.com/wandb/wandb/tree/v0.20.1/wandb/sdk/launch/agent/agent.py#L786-L803)

```python theme={null}
check_sweep_state(
    launch_spec, api
)
```

Check the state of a sweep before launching a run for the sweep.

### `fail_run_queue_item`

[View source](https://www.github.com/wandb/wandb/tree/v0.20.1/wandb/sdk/launch/agent/agent.py#L295-L304)

```python theme={null}
fail_run_queue_item(
    run_queue_item_id, message, phase, files=None
)
```

### `finish_thread_id`

[View source](https://www.github.com/wandb/wandb/tree/v0.20.1/wandb/sdk/launch/agent/agent.py#L416-L509)

```python theme={null}
finish_thread_id(
    thread_id, exception=None
)
```

Removes the job from our list for now.

### `get_job_and_queue`

[View source](https://www.github.com/wandb/wandb/tree/v0.20.1/wandb/sdk/launch/agent/agent.py#L908-L915)

```python theme={null}
get_job_and_queue()
```

### `initialized`

[View source](https://www.github.com/wandb/wandb/tree/v0.20.1/wandb/sdk/launch/agent/agent.py#L190-L193)

```python theme={null}
@classmethod
initialized() -> bool
```

Return whether the agent is initialized.

### `loop`

[View source](https://www.github.com/wandb/wandb/tree/v0.20.1/wandb/sdk/launch/agent/agent.py#L572-L653)

```python theme={null}
loop()
```

Loop infinitely to poll for jobs and run them.

| Raises              | Description                        |
| :------------------ | :--------------------------------- |
| `KeyboardInterrupt` | if the agent is requested to stop. |

### `name`

[View source](https://www.github.com/wandb/wandb/tree/v0.20.1/wandb/sdk/launch/agent/agent.py#L180-L188)

```python theme={null}
@classmethod
name() -> str
```

Return the name of the agent.

### `pop_from_queue`

[View source](https://www.github.com/wandb/wandb/tree/v0.20.1/wandb/sdk/launch/agent/agent.py#L340-L363)

```python theme={null}
pop_from_queue(
    queue
)
```

Pops an item off the runqueue to run as a job.

| Arguments | Description        |
| :-------- | :----------------- |
| `queue`   | Queue to pop from. |

| Returns                    | Description |
| :------------------------- | :---------- |
| Item popped off the queue. |             |

| Raises      | Description                                  |
| :---------- | :------------------------------------------- |
| `Exception` | if there is an error popping from the queue. |

### `print_status`

[View source](https://www.github.com/wandb/wandb/tree/v0.20.1/wandb/sdk/launch/agent/agent.py#L365-L381)

```python theme={null}
print_status() -> None
```

Prints the current status of the agent.

### `run_job`

[View source](https://www.github.com/wandb/wandb/tree/v0.20.1/wandb/sdk/launch/agent/agent.py#L511-L541)

```python theme={null}
run_job(
    job, queue, file_saver
)
```

Set up project and run the job.

| Arguments | Description |
| :-------- | :---------- |
| `job`     | Job to run. |

### `task_run_job`

[View source](https://www.github.com/wandb/wandb/tree/v0.20.1/wandb/sdk/launch/agent/agent.py#L656-L688)

```python theme={null}
task_run_job(
    launch_spec, job, default_config, api, job_tracker
)
```

### `update_status`

[View source](https://www.github.com/wandb/wandb/tree/v0.20.1/wandb/sdk/launch/agent/agent.py#L383-L394)

```python theme={null}
update_status(
    status
)
```

Update the status of the agent.

| Arguments | Description                    |
| :-------- | :----------------------------- |
| `status`  | Status to update the agent to. |


# Monitor launch queue
Source: https://docs.wandb.ai/platform/launch/launch-queue-observability

Monitor launch queue workloads with an interactive dashboard to spot inefficient jobs and analyze resource usage.

Use the interactive **Queue monitoring dashboard** to view when a launch queue is in heavy use or idle, visualize workloads that are running, and spot inefficient jobs. The launch queue dashboard helps you decide whether you're effectively using your compute hardware or cloud resources.

For deeper analysis, the page links to the W\&B experiment tracking workspace and to external infrastructure monitoring providers like Datadog, NVIDIA Base Command, or cloud consoles.

<Note>
  Queue monitoring dashboards are available only in the W\&B Multi-tenant Cloud deployment option.
</Note>

## Dashboard and plots

Use the **Monitor** tab to view the activity of a queue that occurred during the last seven days. Use the left panel to control time ranges, grouping, and filters.

The dashboard contains several plots that answer common questions about performance and efficiency. The following sections describe UI elements of queue dashboards.

### Job status

The **Job status** plot shows how many jobs are running, pending, queued, or completed in each time interval. Use the **Job status** plot to identify periods of idleness in the queue.

<Frame>
  <img alt="Job status timeline" />
</Frame>

For example, suppose you have a fixed resource (such as DGX BasePod). If you observe an idle queue with the fixed resource, this might suggest an opportunity to run lower-priority preemptible launch jobs such as sweeps.

On the other hand, suppose you use a cloud resource and you see periodic bursts of activity. Periodic bursts of activity might suggest an opportunity to save money by reserving resources for particular times.

To the right of the plot, a key shows which colors represent the [status of a launch job](./launch-view-jobs#check-the-status-of-a-job).

<Note>
  `Queued` items might indicate opportunities to shift workloads to other queues. A spike in failures can identify users who might need help with their launch job setup.
</Note>

### Queued time

The **Queued time** plot shows the amount of time (in seconds) that a launch job was on a queue for a given date or time range.

<Frame>
  <img alt="Queued time metrics" />
</Frame>

The x-axis shows a time that you specify, and the y-axis shows the time (in seconds) a launch job was on a launch queue. For example, suppose on a given day 10 launch jobs are queued. The **Queue time** plot shows 600 seconds if those 10 launch jobs wait an average of 60 seconds each.

<Note>
  Use the **Queued time** plot to identify users affected by long queue times.
</Note>

Customize the color of each job with the **Grouping** control in the left bar, which helps you identify which users and jobs are affected by scarce queue capacity.

### Job runs

The **Job runs** plot shows the start and end of every job executed in a time period, with distinct colors for each run. This lets you see at a glance which workloads the queue was processing at a given time.

<Frame>
  <img alt="Job runs timeline" />
</Frame>

Use the **Select** tool in the bottom right of the panel to brush over jobs and populate details in the table below.

### CPU and GPU usage

Use the **GPU use by a job**, **CPU use by a job**, **GPU memory by job**, and **System memory by job** plots to view the efficiency of your launch jobs.

<Frame>
  <img alt="GPU usage metrics" />
</Frame>

For example, you can use the **GPU memory by job** plot to view whether a W\&B run took a long time to complete and whether it used a low percentage of its CPU cores.

The x-axis of each plot shows the duration of a W\&B run (created by a launch job) in seconds. Hover your mouse over a data point to view information about a W\&B run such as the run ID, the project the run belongs to, the launch job that created the W\&B run, and more.

### Errors

The **Errors** panel shows errors that occurred on a given launch queue. More specifically, the Errors panel shows a timestamp of when the error occurred, the name of the launch job where the error comes from, and the error message that was created. By default, errors appear in order from latest to oldest.

<Frame>
  <img alt="Error logs panel" />
</Frame>

Use the **Errors** panel to identify and unblock users.

## External links

The queue observability dashboard's view is consistent across all queue types, but it's often useful to jump directly into environment-specific monitors. To do so, add a link from the console directly from the queue observability dashboard.

To add an external link, click **Manage Links** to open a panel. Add the full URL of the page you want. Next, add a label. Links that you add appear in the **External Links** section.


# Launch terms and concepts
Source: https://docs.wandb.ai/platform/launch/launch-terminology

Learn key W&B Launch concepts including jobs, queues, target resources, agents, and agent environments.

This page defines the core terms and concepts used throughout the W\&B Launch documentation. Use it as a reference when you encounter unfamiliar terms in setup guides, tutorials, or the W\&B App.

With W\&B Launch, you enqueue [jobs](#launch-job) onto [queues](#launch-queue) to create runs. Jobs are Python scripts instrumented with W\&B. Queues hold a list of jobs to execute on a [target resource](#target-resources). [Agents](#launch-agent) pull jobs from queues and execute them on target resources. W\&B tracks Launch jobs similarly to how it tracks [runs](/models/runs/).

The following sections describe each of these concepts in more detail.

## Launch job

A Launch job is a specific type of [W\&B Artifact](/models/artifacts/) that represents a task to complete. For example, common Launch jobs include training a model or triggering a model evaluation. Job definitions include:

* Python code and other file assets, including at least one runnable entrypoint.
* Information about the input (config parameter) and output (metrics logged).
* Information about the environment (for example, `requirements.txt`, base `Dockerfile`).

The three main kinds of job definitions are:

| Job types                           | Definition                                                                                  | How to run this job type                                                                                  |
| ----------------------------------- | ------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------- |
| Artifact-based (or code-based) jobs | Code and other assets are saved as a W\&B artifact.                                         | To run artifact-based jobs, you must configure the Launch agent with a builder.                           |
| Git-based jobs                      | Code and other assets are cloned from a certain commit, branch, or tag in a git repository. | To run git-based jobs, you must configure the Launch agent with a builder and git repository credentials. |
| Image-based jobs                    | Code and other assets are baked into a Docker image.                                        | To run image-based jobs, you might need to configure the Launch agent with image repository credentials.  |

<Note>
  Although Launch jobs can perform activities not related to model training (for example, deploying a model to a Triton inference server), all jobs must call `wandb.init()` to complete successfully. This creates a run for tracking purposes in a W\&B workspace.
</Note>

Find jobs you created in the W\&B App under the **Jobs** tab of your project workspace. From there, you can configure jobs and send them to a [Launch queue](#launch-queue) to execute on different [target resources](#target-resources).

## Launch queue

Launch *queues* are ordered lists of jobs to execute on a specific target resource. Launch queues are first-in, first-out (FIFO). No practical limit applies to the number of queues you can have, but a good guideline is one queue per target resource. You can enqueue jobs with the W\&B App UI, W\&B CLI, or Python SDK. You can then configure one or more Launch agents to pull items from the queue and execute them on the queue's target resource.

## Target resources

The compute environment that a Launch queue executes jobs on is called the *target resource*.

W\&B Launch supports the following target resources:

* [Docker](/platform/launch/setup-launch-docker/)
* [Kubernetes](/platform/launch/setup-launch-kubernetes/)
* [AWS SageMaker](/platform/launch/setup-launch-sagemaker/)
* [Google Cloud Vertex AI](/platform/launch/setup-vertex/)

Each target resource accepts a different set of configuration parameters called *resource configurations*. Resource configurations take on default values defined by each Launch queue, but each job can override them independently. See the documentation for each target resource for more details.

## Launch agent

Launch agents are lightweight, persistent programs that periodically check Launch queues for jobs to execute. When a Launch agent receives a job, it first builds or pulls the image from the job definition, then runs it on the target resource.

One agent can poll multiple queues. However, you must configure the agent properly to support all the backing target resources for each queue it polls.

## Launch agent environment

The agent environment is the environment where a Launch agent is running, polling for jobs.

<Note>
  The agent's runtime environment is independent of a queue's target resource. In other words, you can deploy agents anywhere as long as you configure them sufficiently to access the required target resources.
</Note>


# View launch jobs
Source: https://docs.wandb.ai/platform/launch/launch-view-jobs

View launch job details, list available jobs, and check job statuses in the W&B App and CLI.

This page describes how to view information about launch jobs in queues. You can inspect job details in the W\&B App, list jobs from the CLI, and interpret the status of a queued run.

## View jobs

To view jobs in a queue, use the W\&B App:

1. Navigate to the W\&B App at [https://wandb.ai/home](https://wandb.ai/home).
2. Select **Launch** in the **Applications** section of the left sidebar.
3. Select the **All entities** dropdown and select the entity the launch job belongs to.
4. Expand the collapsible UI from the Launch Application page to view a list of jobs added to that specific queue.

<Note>
  The launch agent creates a run when it executes a launch job. In other words, each run listed corresponds to a specific job in that queue.
</Note>

For example, the following image shows two runs created from a job called `job-source-launch_demo-canonical`. The job is in a queue called `Start queue`. The first run in the queue is called `resilient-snowball` and the second run is called `earthy-energy-165`.

<Frame>
  <img alt="Launch jobs status view" />
</Frame>

In the W\&B App UI, you can find additional information about runs created from launch jobs, such as the:

* **Run**: The name of the W\&B run assigned to that job.
* **Job ID**: The name of the job.
* **Project**: The name of the project the run belongs to.
* **Status**: The status of the queued run.
* **Author**: The W\&B entity that created the run.
* **Creation date**: The timestamp when the queue was created.
* **Start time**: The timestamp when the job started.
* **Duration**: Time, in seconds, it took to complete the job's run.

## List jobs

View a list of jobs that exist in a project with the W\&B CLI. Use the `wandb job list` command and provide the name of the project and entity the launch job belongs to with the `--project` and `--entity` flags, respectively.

Replace `[ENTITY]` with your W\&B entity, and replace `[PROJECT-NAME]` with the name of the project the launch job belongs to.

```bash theme={null}
wandb job list --entity [ENTITY] --project [PROJECT-NAME]
```

## Check the status of a job

Each queued run has a status that reflects where it is in the launch lifecycle. The following table defines the statuses a queued run can have:

| Status       | Description                                                                                                                 |
| ------------ | --------------------------------------------------------------------------------------------------------------------------- |
| **Idle**     | The run is in a queue with no active agents.                                                                                |
| **Queued**   | The run is in a queue waiting for an agent to process it.                                                                   |
| **Pending**  | An agent has picked up the run, but it hasn't started yet. This could be due to resources being unavailable on the cluster. |
| **Running**  | The run is currently executing.                                                                                             |
| **Killed**   | The user killed the job.                                                                                                    |
| **Crashed**  | The run stopped sending data or didn't successfully start.                                                                  |
| **Failed**   | The run ended with a non-zero exit code, or the run failed to start.                                                        |
| **Finished** | The job completed successfully.                                                                                             |


# Set up Launch
Source: https://docs.wandb.ai/platform/launch/set-up-launch

Set up W&B Launch by configuring a queue and deploying an agent to submit jobs to your compute infrastructure.

This page describes the high-level steps required to set up W\&B Launch so that you can submit machine learning jobs to your own compute infrastructure (such as Kubernetes, Amazon SageMaker, or Vertex AI) directly from W\&B. Setting up Launch enables your team to share compute resources, queue jobs, and reproduce runs across environments.

To set up Launch, you must complete the following:

1. **Set up a queue**: Queues are first-in, first-out (FIFO) and possess a queue configuration. A queue's configuration controls where and how jobs run on a target resource.
2. **Set up an agent**: Agents run on your machine or infrastructure and poll one or more queues for launch jobs. When the agent pulls a job, it ensures that the image is built and available. The agent then submits the job to the target resource.

After you complete both steps, your team can submit launch jobs to the queue and run them automatically on the target resource.

## Set up a queue

Configure launch queues to point to a specific target resource along with any additional configuration specific to that resource. For example, a launch queue that points to a Kubernetes cluster might include environment variables or set a custom namespace in its launch queue configuration. When you create a queue, you specify both the target resource you want to use and the configuration for that resource to use.

When an agent receives a job from a queue, it also receives the queue configuration. When the agent submits the job to the target resource, it includes the queue configuration along with any overrides from the job itself. For example, you can use a job configuration to specify the Amazon SageMaker instance type for that job instance only. In this case, it's common to use [queue config templates](/platform/launch/setup-queue-advanced/#configure-queue-template) as the end user interface.

### Create a queue

Create a queue to define where launch jobs run and how they're configured. To create a queue:

1. Navigate to Launch App at [wandb.ai/launch](https://wandb.ai/launch).
2. Click the **create queue** button on the top right of the screen.
3. From the **Entity** dropdown menu, select the entity the queue belongs to.
4. Provide a name for your queue in the **Queue** field.
5. From the **Resource** dropdown, select the compute resource you want jobs added to this queue to use.
6. Choose whether to allow **Prioritization** for this queue. If prioritization is enabled, a user on your team can define a priority for their launch job when they enqueue it. Higher priority jobs run before lower priority jobs.
7. Provide a resource configuration in either JSON or YAML format in the **Configuration** field. The structure and semantics of your configuration document depend on the resource type that the queue points to. For more details, see the dedicated set up page for your target resource.

<Frame>
  <img alt="Creating a Launch queue" />
</Frame>

You now have a queue that can receive launch jobs and route them to your chosen target resource. Next, set up an agent to pull jobs from this queue.

## Set up a launch agent

Launch agents are long-running processes that poll one or more launch queues for jobs. Launch agents dequeue jobs in FIFO order or in priority order depending on the queues they pull from. When an agent dequeues a job from a queue, it optionally builds an image for that job. The agent then submits the job to the target resource along with configuration options specified in the queue configuration.

<Note>
  Agents are flexible and can be configured to support many use cases. The required configuration for your agent depends on your specific use case. See the dedicated page for [Docker](/platform/launch/setup-launch-docker/), [Amazon SageMaker](/platform/launch/setup-launch-sagemaker/), [Kubernetes](/platform/launch/setup-launch-kubernetes/), or [Vertex AI](/platform/launch/setup-vertex/).
</Note>

<Note>
  W\&B recommends you start agents with a service account's API key, rather than a specific user's API key. Using a service account's API key has two benefits:

  * The agent isn't dependent on an individual user.
  * Launch views the author associated with a run created through Launch as the user who submitted the launch job, rather than the user associated with the agent.
</Note>

### Agent configuration

Configure the launch agent so it knows which queues to poll, which entity to operate under, and how many jobs to run in parallel. Configure the launch agent with a YAML file named `launch-config.yaml`. By default, W\&B checks for the config file in `~/.config/wandb/launch-config.yaml`. You can optionally specify a different directory when you activate the launch agent.

The contents of your launch agent's configuration file depend on your launch agent's environment, the launch queue's target resource, Docker builder requirements, cloud registry requirements, and so forth.

Independent of your use case, the launch agent has the following core configurable options:

* `max_jobs`: Maximum number of jobs the agent can execute in parallel.
* `entity`: The entity that the queue belongs to.
* `queues`: The name of one or more queues for the agent to watch.

<Note>
  You can use the W\&B CLI to specify universal configurable options for the launch agent (instead of the config YAML file): maximum number of jobs, W\&B entity, and launch queues. See the [`wandb launch-agent`](/models/ref/cli/wandb-launch-agent) command for more information.
</Note>

The following YAML snippet shows how to specify core launch agent config keys. Replace `[ENTITY-NAME]` with your W\&B entity, and replace `[QUEUE-NAME]` with the name of a queue for the agent to poll.

```yaml title="launch-config.yaml" theme={null}
# Max number of concurrent runs to perform. -1 = no limit
max_jobs: -1

entity: [ENTITY-NAME]

# List of queues to poll.
queues:
  - [QUEUE-NAME]
```

### Configure a container builder

You can configure the launch agent to build images. You must configure the agent to use a container builder if you intend to use launch jobs created from git repositories or code artifacts. See [Create a launch job](/platform/launch/create-launch-job/) for more information on how to create a launch job.

W\&B Launch supports three builder options:

* Docker: The Docker builder uses a local Docker daemon to build images.
* [Kaniko](https://github.com/GoogleContainerTools/kaniko): Kaniko is a Google project that enables image building in environments where a Docker daemon is unavailable.
* Noop: The agent doesn't try to build jobs, and instead only pulls pre-built images.

<Note>
  Use the Kaniko builder if your agent is polling in an environment where a Docker daemon is unavailable (for example, a Kubernetes cluster).

  See [Set up Kubernetes](/platform/launch/setup-launch-kubernetes/) for details about the Kaniko builder.
</Note>

To specify an image builder, include the builder key in your agent configuration. For example, the following code snippet shows a portion of the launch config (`launch-config.yaml`) that specifies to use Docker or Kaniko:

```yaml title="launch-config.yaml" theme={null}
builder:
  type: docker | kaniko | noop
```

### Configure a container registry

Sometimes, you might want to connect a launch agent to a cloud registry so it can pull pre-built images or push images it builds. Common scenarios for connecting a launch agent to a cloud registry include:

* You want to run a job in an environment other than where you built it, such as a powerful workstation or cluster.
* You want to use the agent to build images and run these images on Amazon SageMaker or Vertex AI.
* You want the launch agent to provide credentials to pull from an image repository.

To learn more about how to configure the agent to interact with a container registry, see the [Advanced agent setup page](/platform/launch/setup-agent-advanced/).

## Activate the launch agent

After you configure the agent, activate it so that it starts polling the queues you specified and dispatching jobs to your target resource. Activate the launch agent with the `launch-agent` W\&B CLI command. Replace `[QUEUE-1]` and `[QUEUE-2]` with the names of queues for the agent to poll:

```bash theme={null}
wandb launch-agent -q [QUEUE-1] -q [QUEUE-2] --max-jobs 5
```

Once the agent is running, the agent pulls any jobs submitted to the specified queues and forwards them to the target resource.

Sometimes, you might want to have a launch agent poll queues from within a Kubernetes cluster. See the [Advanced queue set up page](/platform/launch/setup-queue-advanced/) for more information.


# Set up launch agent
Source: https://docs.wandb.ai/platform/launch/setup-agent-advanced

Configure advanced agent options for W&B Launch including Docker and Kaniko builders and container registry settings.

# Advanced agent setup

This guide explains how to configure the W\&B Launch agent to build container images in different environments, push those images to cloud container registries, and customize the build process. Use it when you need to run launch jobs that require image builds and want to control where and how the agent produces those images. This page is for administrators and operators who deploy and manage the Launch agent.

<Note>
  Build is only required for git and code artifact jobs. Image jobs do not require build.

  See [Create a launch job](/platform/launch/create-launch-job/) for more information on job types.
</Note>

## Builders

The Launch agent supports two builders for producing container images. Choose the builder that matches the environment where the agent runs.

The Launch agent can build images using [Docker](https://docs.docker.com/) or [Kaniko](https://github.com/GoogleContainerTools/kaniko).

* Kaniko: Builds a container image in Kubernetes without running the build as a privileged container.
* Docker: Builds a container image by executing a `docker build` command locally.

Control the builder type with the `builder.type` key in the Launch agent config. Set it to `docker`, `kaniko`, or `noop` to turn off build. By default, the agent Helm chart sets the `builder.type` to `noop`. The agent uses additional keys in the `builder` section to configure the build process.

If you don't specify a builder in the agent config and a working `docker` CLI is found, the agent defaults to Docker. If Docker isn't available, the agent defaults to `noop`.

<Note>
  Use Kaniko for building images in a Kubernetes cluster. Use Docker for all other cases.
</Note>

## Push to a container registry

To run built images on your compute target, the agent must push them to a container registry that the target can pull from. The following sections explain how the agent tags and uploads images.

The Launch agent tags all images it builds with a unique source hash. The agent pushes the image to the registry specified in the `builder.destination` key.

For example, if you set the `builder.destination` key to `my-registry.example.com/my-repository`, the agent tags and pushes the image to `my-registry.example.com/my-repository:[SOURCE-HASH]`. If the image exists in the registry, the agent skips the build.

### Agent configuration

The agent reads its configuration from a YAML file. Where you provide that file depends on how you run the agent.

If you deploy the agent with the Helm chart, provide the agent config in the `agentConfig` key in the `values.yaml` file.

If you invoke the agent yourself with `wandb launch-agent`, provide the agent config as a path to a YAML file with the `--config` flag. By default, the agent loads the config from `~/.config/wandb/launch-config.yaml`.

Within your Launch agent config (`launch-config.yaml`), provide the name of the target resource environment and the container registry for the `environment` and `registry` keys, respectively.

The following tabs demonstrate how to configure the Launch agent based on your environment and registry.

<Tabs>
  <Tab title="AWS">
    The AWS environment configuration requires the region key. Set the region to the AWS region that the agent runs in.

    ```yaml title="launch-config.yaml" theme={null}
    environment:
      type: aws
      region: [AWS-REGION]
    builder:
      type: [BUILDER-TYPE]
      # URI of the ECR repository where the agent stores images.
      # Make sure the region matches what you configured in your
      # environment.
      destination: [ACCOUNT-ID].ecr.[AWS-REGION].amazonaws.com/[REPOSITORY-NAME]
      # If you use Kaniko, specify the S3 bucket where the agent stores the
      # build context.
      build-context-store: s3://[BUCKET-NAME]/[PATH]
    ```

    The agent uses `boto3` to load the default AWS credentials. See the [boto3 documentation](https://boto3.amazonaws.com/v1/documentation/api/latest/index.html) for more information on how to configure default AWS credentials.
  </Tab>

  <Tab title="Google Cloud">
    The Google Cloud environment requires region and project keys. Set `region` to the region that the agent runs in. Set `project` to the Google Cloud project that the agent runs in. The agent uses `google.auth.default()` in Python to load the default credentials.

    ```yaml title="launch-config.yaml" theme={null}
    environment:
      type: gcp
      region: [GCP-REGION]
      project: [GCP-PROJECT-ID]
    builder:
      type: [BUILDER-TYPE]
      # URI of the Artifact Registry repository and image name where the agent
      # stores images. Make sure the region and project match what you
      # configured in your environment.
      uri: [REGION]-docker.pkg.dev/[PROJECT-ID]/[REPOSITORY-NAME]/[IMAGE-NAME]
      # If you use Kaniko, specify the GCS bucket where the agent stores the
      # build context.
      build-context-store: gs://[BUCKET-NAME]/[PATH]
    ```

    See the [`google-auth` documentation](https://google-auth.readthedocs.io/en/latest/reference/google.auth.html#google.auth.default) for more information on how to configure default Google Cloud credentials so they are available to the agent.
  </Tab>

  <Tab title="Azure">
    The Azure environment doesn't require any additional keys. When the agent starts, it uses `azure.identity.DefaultAzureCredential()` to load the default Azure credentials.

    ```yaml title="launch-config.yaml" theme={null}
    environment:
      type: azure
    builder:
      type: [BUILDER-TYPE]
      # URI of the Azure Container Registry repository where the agent stores images.
      destination: https://[REGISTRY-NAME].azurecr.io/[REPOSITORY-NAME]
      # If you use Kaniko, specify the Azure Blob Storage container where the agent
      # stores the build context.
      build-context-store: https://[STORAGE-ACCOUNT-NAME].blob.core.windows.net/[CONTAINER-NAME]
    ```

    See the [`azure-identity` documentation](https://learn.microsoft.com/python/api/azure-identity/azure.identity.defaultazurecredential?view=azure-python) for more information on how to configure default Azure credentials.
  </Tab>
</Tabs>

## Agent permissions

The agent must have permission to push images to your container registry and, if you use Kaniko, to read and write build context in cloud storage. The agent permissions required vary by use case.

### Cloud registry permissions

The agent needs registry permissions so it can create repositories, upload image layers, and push tagged images. The following permissions are required for Launch agents to interact with cloud registries.

<Tabs>
  <Tab title="AWS">
    ```yaml theme={null}
    {
      'Version': '2012-10-17',
      'Statement':
        [
          {
            'Effect': 'Allow',
            'Action':
              [
                'ecr:CreateRepository',
                'ecr:UploadLayerPart',
                'ecr:PutImage',
                'ecr:CompleteLayerUpload',
                'ecr:InitiateLayerUpload',
                'ecr:DescribeRepositories',
                'ecr:DescribeImages',
                'ecr:BatchCheckLayerAvailability',
                'ecr:BatchDeleteImage',
              ],
            'Resource': 'arn:aws:ecr:[REGION]:[ACCOUNT-ID]:repository/[REPOSITORY]',
          },
          {
            'Effect': 'Allow',
            'Action': 'ecr:GetAuthorizationToken',
            'Resource': '*',
          },
        ],
    }
    ```
  </Tab>

  <Tab title="Google Cloud">
    ```js theme={null}
    artifactregistry.dockerimages.list;
    artifactregistry.repositories.downloadArtifacts;
    artifactregistry.repositories.list;
    artifactregistry.repositories.uploadArtifacts;
    ```
  </Tab>

  <Tab title="Azure">
    Add the [`AcrPush` role](https://learn.microsoft.com/azure/role-based-access-control/built-in-roles/containers#acrpush) if you use the Kaniko builder.
  </Tab>
</Tabs>

### Storage permissions for Kaniko

The Launch agent requires permission to push to cloud storage if the agent uses the Kaniko builder. Kaniko uses a context store outside of the pod that runs the build job.

<Tabs>
  <Tab title="AWS">
    Use Amazon S3 as the context store for the Kaniko builder on AWS. Use the following policy to give the agent access to an S3 bucket:

    ```json theme={null}
    {
      "Version": "2012-10-17",
      "Statement": [
        {
          "Sid": "ListObjectsInBucket",
          "Effect": "Allow",
          "Action": ["s3:ListBucket"],
          "Resource": ["arn:aws:s3:::[BUCKET-NAME]"]
        },
        {
          "Sid": "AllObjectActions",
          "Effect": "Allow",
          "Action": "s3:*Object",
          "Resource": ["arn:aws:s3:::[BUCKET-NAME]/*"]
        }
      ]
    }
    ```
  </Tab>

  <Tab title="Google Cloud">
    On Google Cloud, the agent requires the following IAM permissions to upload build contexts to GCS:

    ```js theme={null}
    storage.buckets.get;
    storage.objects.create;
    storage.objects.delete;
    storage.objects.get;
    ```
  </Tab>

  <Tab title="Azure">
    The agent requires the [Storage Blob Data Contributor](https://learn.microsoft.com/azure/role-based-access-control/built-in-roles#storage-blob-data-contributor) role to upload build contexts to Azure Blob Storage.
  </Tab>
</Tabs>

## Customize the Kaniko build

To override defaults such as caching behavior or environment variables for the build pod, customize the Kubernetes Job that Kaniko runs. Specify the Kubernetes Job spec that the Kaniko job uses in the `builder.kaniko-config` key of the agent configuration. For example:

```yaml title="launch-config.yaml" theme={null}
builder:
  type: kaniko
  build-context-store: [MY-BUILD-CONTEXT-STORE]
  destination: [MY-IMAGE-DESTINATION]
  build-job-name: wandb-image-build
  kaniko-config:
    spec:
      template:
        spec:
          containers:
          - args:
            - "--cache=false" # Args must be in the format "key=value"
            env:
            - name: "MY_ENV_VAR"
              value: "my-env-var-value"
```

## Deploy Launch agent into CoreWeave

If your workloads benefit from GPU-accelerated infrastructure, you can deploy the Launch agent to CoreWeave Cloud. CoreWeave is a cloud infrastructure built for GPU-accelerated workloads.

For information on how to deploy the Launch agent to CoreWeave, see the [CoreWeave documentation](https://docs.coreweave.com/partners/weights-and-biases#integration).

<Note>
  You need to create a [CoreWeave account](https://cloud.coreweave.com/login) to deploy the Launch agent into a CoreWeave infrastructure.
</Note>


# Tutorial: Set up W&B Launch with Docker
Source: https://docs.wandb.ai/platform/launch/setup-launch-docker

Set up W&B Launch to run ML jobs using Docker on a local machine as both the agent environment and compute target.

This tutorial describes how to configure W\&B Launch to use Docker on a local machine for both the launch agent environment and for the queue's target resource. By the end, you have a working Docker-based launch queue and a local launch agent ready to run ML jobs.

Using Docker to run jobs and as the launch agent's environment on the same local machine is useful if your compute is installed on a machine that doesn't have a cluster management system (such as Kubernetes). You can also use Docker queues to run workloads on workstations.

<Note>
  This setup is common for users who run experiments on their local machine, or who SSH into a remote machine to submit launch jobs.
</Note>

When you use Docker with Launch, W\&B first builds an image, and then builds and runs a container from that image. W\&B builds the image with the Docker `docker run [IMAGE-URI]` command. W\&B interprets the queue configuration as additional arguments passed to the `docker run` command.

## Configure a Docker queue

A queue configuration for a Docker target resource defines how Launch translates queue options into a `docker run` command.

The launch queue configuration (for a Docker target resource) accepts the same options defined in the [`docker run`](/models/ref/cli/wandb-docker-run) CLI command.

The agent receives options defined in the queue configuration. The agent then merges the received options with any overrides from the launch job's configuration to produce a final `docker run` command that runs on the target resource (in this case, a local machine).

Two syntax transformations take place:

* Define repeated options in the queue configuration as a list.
* Define flag options in the queue configuration as a boolean with the value `true`.

For example, the following queue configuration:

```json theme={null}
{
  "env": ["MY_ENV_VAR=value", "MY_EXISTING_ENV_VAR"],
  "volume": "/mnt/datasets:/mnt/datasets",
  "rm": true,
  "gpus": "all"
}
```

Results in the following `docker run` command:

```bash theme={null}
docker run \
  --env MY_ENV_VAR=value \
  --env MY_EXISTING_ENV_VAR \
  --volume "/mnt/datasets:/mnt/datasets" \
  --rm [IMAGE-URI] \
  --gpus all
```

Specify volumes either as a list of strings, or as a single string. Use a list if you specify multiple volumes.

Docker passes environment variables that aren't assigned a value from the launch agent environment. If the launch agent has an environment variable `MY_EXISTING_ENV_VAR`, that variable is available in the container. This is useful if you want to use other config keys without publishing them in the queue configuration.

The `--gpus` flag of the `docker run` command lets you specify GPUs that are available to a Docker container. For more information about the `--gpus` flag, see the [Docker documentation](https://docs.docker.com/config/containers/resource_constraints/#gpu).

<Note>
  * Install the [NVIDIA Container Toolkit](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/install-guide.html#docker) to use GPUs within a Docker container.
  * If you build images from a code or artifact-sourced job, you can override the base image used by the [agent](#configure-a-launch-agent-on-a-local-machine) to include the NVIDIA Container Toolkit.
    For example, within your launch queue, you can override the base image to `tensorflow/tensorflow:latest-gpu`:

    ```json theme={null}
    {
      "builder": {
        "accelerator": {
          "base_image": "tensorflow/tensorflow:latest-gpu"
        }
      }
    }
    ```
</Note>

## Create a queue

To create a queue that uses Docker as the compute resource, follow these steps:

1. Navigate to the [Launch page](https://wandb.ai/launch).
2. Click the **Create Queue** button.
3. Select the **Entity** you want to create the queue in.
4. Enter a name for your queue in the **Name** field.
5. Select **Docker** as the **Resource**.
6. Define your Docker queue configuration in the **Configuration** field.
7. Click the **Create Queue** button.

You now have a Docker-based launch queue ready to receive jobs. Next, configure a launch agent on your local machine to pull jobs from this queue.

## Configure a launch agent on a local machine

Configure the launch agent with a YAML config file named `launch-config.yaml`. By default, W\&B checks for the config file in `~/.config/wandb/launch-config.yaml`. You can optionally specify a different directory when you activate the launch agent.

<Note>
  You can use the W\&B CLI to specify core configurable options for the launch agent (instead of the config YAML file): maximum number of jobs, W\&B entity, and launch queues. See the [`wandb launch-agent`](/models/ref/cli/wandb-launch-agent) command for more information.
</Note>

### Core agent config options

The following tabs demonstrate how to specify the core config agent options with the W\&B CLI and with a YAML config file:

<Tabs>
  <Tab title="W&B CLI">
    ```bash theme={null}
    wandb launch-agent -q [QUEUE-NAME] --max-jobs [N]
    ```
  </Tab>

  <Tab title="Config file">
    ```yaml title="launch-config.yaml" theme={null}
    max_jobs: [N-CONCURRENT-JOBS]
    queues:
    	- [QUEUE-NAME]
    ```
  </Tab>
</Tabs>

### Docker image builders

You can configure the launch agent on your machine to build Docker images. By default, your machine's local image repository stores these images. To enable your launch agent to build Docker images, set the `builder` key in the launch agent config to `docker`:

```yaml title="launch-config.yaml" theme={null}
builder:
	type: docker
```

If you don't want the agent to build Docker images, and instead use prebuilt images from a registry, set the `builder` key in the launch agent config to `noop`:

```yaml title="launch-config.yaml" theme={null}
builder:
  type: noop
```

### Container registries

Launch uses external container registries such as Docker Hub, Google Container Registry, Azure Container Registry, and Amazon ECR. If you want to run a job on a different environment from where you built it, configure your agent to pull from a container registry, so the agent can retrieve images that weren't built locally.

To learn more about how to connect the launch agent with a cloud registry, see the [Advanced agent setup](./setup-agent-advanced#agent-configuration) page.


# Tutorial: Set up W&B Launch on Kubernetes
Source: https://docs.wandb.ai/platform/launch/setup-launch-kubernetes

Set up W&B Launch on a Kubernetes cluster using Helm charts, Kaniko image building, and Kubernetes job specs.

This tutorial walks cluster administrators through setting up W\&B Launch on a Kubernetes cluster so ML engineers can submit and manage training workloads directly from W\&B. You can use W\&B Launch to push ML workloads to a Kubernetes cluster, giving ML engineers an interface right in W\&B to use the resources you already manage with Kubernetes.

W\&B maintains an [official Launch agent image](https://hub.docker.com/r/wandb/launch-agent) that you can deploy to your cluster with a [Helm chart](https://github.com/wandb/helm-charts/tree/main/charts/launch-agent) that W\&B maintains.

W\&B uses the [Kaniko](https://github.com/GoogleContainerTools/kaniko) builder to let the Launch agent build Docker images in a Kubernetes cluster. To learn more about how to set up Kaniko for the Launch agent, or how to turn off job building and only use prebuilt Docker images, see [Advanced agent setup](./setup-agent-advanced).

<Note>
  To install Helm and apply or upgrade the W\&B Launch agent Helm chart, you must have `kubectl` access to the cluster with sufficient permissions to create, update, and delete Kubernetes resources. Typically, this requires a user with `cluster-admin` or a custom role with equivalent permissions.
</Note>

## Configure a queue for Kubernetes

A Launch queue defines the Kubernetes workload spec that the agent uses to run each job. The Launch queue configuration for a Kubernetes target resource resembles either a [Kubernetes job spec](https://kubernetes.io/docs/concepts/workloads/controllers/job/) or a [Kubernetes custom resource spec](https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/).

You can control any aspect of the Kubernetes workload resource spec when you create a Launch queue.

<Tabs>
  <Tab title="Kubernetes job spec">
    ```yaml theme={null}
    spec:
      template:
        spec:
          containers:
            - env:
                - name: MY_ENV_VAR
                  value: some-value
              resources:
                requests:
                  cpu: 1000m
                  memory: 1Gi
    metadata:
      labels:
        queue: k8s-test
    namespace: wandb
    ```
  </Tab>

  <Tab title="Custom resource spec">
    In some use cases, you might want to use `CustomResource` definitions. For example, `CustomResource` definitions are useful when you want to perform multi-node distributed training. See the tutorial for using Launch with multi-node jobs using Volcano for an example application. Another use case is when you want to use Launch with Kubeflow.

    The following YAML snippet shows a sample Launch queue config that uses Kubeflow:

    ```yaml theme={null}
    kubernetes:
      kind: PyTorchJob
      spec:
        pytorchReplicaSpecs:
          Master:
            replicas: 1
            template:
              spec:
                containers:
                  - name: pytorch
                    image: '${image_uri}'
                    imagePullPolicy: Always
            restartPolicy: Never
          Worker:
            replicas: 2
            template:
              spec:
                containers:
                  - name: pytorch
                    image: '${image_uri}'
                    imagePullPolicy: Always
            restartPolicy: Never
        ttlSecondsAfterFinished: 600
      metadata:
        name: '${run_id}-pytorch-job'
      apiVersion: kubeflow.org/v1
    ```
  </Tab>
</Tabs>

For security reasons, W\&B injects the following resources into your Launch queue if you don't specify them:

* `securityContext`
* `backOffLimit`
* `ttlSecondsAfterFinished`

The following YAML snippet shows how these values appear in your Launch queue:

```yaml title="example-spec.yaml" theme={null}
spec:
  template:
    backOffLimit: 0
    ttlSecondsAfterFinished: 60
    securityContext:
      allowPrivilegeEscalation: false
      capabilities:
        drop:
          - ALL
      seccompProfile:
        type: "RuntimeDefault"
```

## Create a queue

Create a queue in the W\&B App that uses Kubernetes as its compute resource:

1. Navigate to the [Launch page](https://wandb.ai/launch).
2. Click the **Create Queue** button.
3. Select the **Entity** in which you want to create the queue.
4. Provide a name for your queue in the **Name** field.
5. Select **Kubernetes** as the **Resource**.
6. Within the **Configuration** field, provide the Kubernetes job workflow spec or custom resource spec you configured in [Configure a queue for Kubernetes](#configure-a-queue-for-kubernetes).

## Configure a Launch agent with Helm

With a queue in place, you next deploy the Launch agent that pulls jobs from the queue and runs them on your cluster. Use the [Helm chart](https://github.com/wandb/helm-charts/tree/main/charts/launch-agent) provided by W\&B to deploy the Launch agent into your Kubernetes cluster. Control the behavior of the Launch agent with the `values.yaml` [file](https://github.com/wandb/helm-charts/blob/main/charts/launch-agent/values.yaml).

Within the `launchConfig` key in the `values.yaml` file, specify the contents that you would normally define in your Launch agent config file (`~/.config/wandb/launch-config.yaml`).

For example, suppose you have a Launch agent config that lets you run a Launch agent in EKS that uses the Kaniko Docker image builder. Replace `[QUEUE-NAME]`, `[MAX-CONCURRENT-JOBS]`, `[MY-REGISTRY-URI]`, and `[S3-BUCKET-URI]` with your own values:

```yaml title="launch-config.yaml" theme={null}
queues:
  - [QUEUE-NAME]
max_jobs: [MAX-CONCURRENT-JOBS]
environment:
  type: aws
  region: us-east-1
registry:
  type: ecr
  uri: [MY-REGISTRY-URI]
builder:
  type: kaniko
  build-context-store: [S3-BUCKET-URI]
```

Within your `values.yaml` file, this might look like the following. Replace `[QUEUE-NAME]`, `[MAX-CONCURRENT-JOBS]`, `[AWS-REGION]`, `[MY-REGISTRY-URI]`, and `[S3-BUCKET-URI]` with your own values:

```yaml title="values.yaml" theme={null}
agent:
  labels: {}
  # W&B API key.
  apiKey: ''
  # Container image to use for the agent.
  image: wandb/launch-agent:latest
  # Image pull policy for agent image.
  imagePullPolicy: Always
  # Resources block for the agent spec.
  resources:
    limits:
      cpu: 1000m
      memory: 1Gi

# Namespace to deploy launch agent into
namespace: wandb

# W&B api url (Set yours here)
baseUrl: https://api.wandb.ai

# Additional target namespaces that the launch agent can deploy into
additionalTargetNamespaces:
  - default
  - wandb

# This should be set to the literal contents of your launch agent config.
launchConfig: |
  queues:
    - [QUEUE-NAME]
  max_jobs: [MAX-CONCURRENT-JOBS]
  environment:
    type: aws
    region: [AWS-REGION]
  registry:
    type: ecr
    uri: [MY-REGISTRY-URI]
  builder:
    type: kaniko
    build-context-store: [S3-BUCKET-URI]

# The contents of a git credentials file. This will be stored in a k8s secret
# and mounted into the agent container. Set this if you want to clone private
# repos.
gitCreds: |

# Annotations for the wandb service account. Useful when setting up workload identity on gcp.
serviceAccount:
  annotations:
    iam.gke.io/gcp-service-account:
    azure.workload.identity/client-id:

# Set to access key for azure storage if using kaniko with azure.
azureStorageAccessKey: ''
```

For more information about registries, environments, and required agent permissions, see [Advanced agent setup](./setup-agent-advanced).


# Tutorial: Set up W&B Launch on SageMaker
Source: https://docs.wandb.ai/platform/launch/setup-launch-sagemaker

Configure W&B Launch to submit training jobs to Amazon SageMaker with ECR, S3, and IAM setup instructions.

This tutorial shows ML engineers and platform administrators how to configure W\&B Launch to submit training jobs to Amazon SageMaker. By the end, you'll have the AWS resources, IAM roles, queue configuration, and Launch agent required to run SageMaker training jobs from W\&B.

You can use W\&B Launch to submit Launch jobs to Amazon SageMaker to train machine learning models using provided or custom algorithms on the SageMaker platform. SageMaker handles compute resource provisioning and release, so it can be a good choice for teams without an EKS cluster.

A W\&B Launch queue connected to Amazon SageMaker runs Launch jobs as SageMaker Training Jobs with the [CreateTrainingJob API](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateTrainingJob.html). Use the Launch queue configuration to control arguments sent to the `CreateTrainingJob` API.

Amazon SageMaker [uses Docker images to run training jobs](https://docs.aws.amazon.com/sagemaker/latest/dg/your-algorithms-training-algo-dockerfile.html). You must store images pulled by SageMaker in the Amazon Elastic Container Registry (ECR). This means you must store the image you use for training on ECR.

<Note>
  This guide shows how to run SageMaker Training Jobs. For information about how to deploy models for inference on Amazon SageMaker, see [this example Launch job](https://github.com/wandb/launch-jobs/tree/main/jobs/deploy_to_sagemaker_endpoints).
</Note>

## Prerequisites

Before you get started, you must satisfy the following prerequisites:

* [Decide if you want the Launch agent to build a Docker image for you](#decide-if-you-want-the-launch-agent-to-build-a-docker-image).
* [Set up AWS resources and gather information about S3, ECR, and SageMaker IAM roles](#set-up-aws-resources).
* [Create an IAM role for the Launch agent](#create-an-iam-role-for-the-launch-agent).

The following sections describe how to complete each prerequisite.

### Decide if you want the Launch agent to build a Docker image

Decide if you want the W\&B Launch agent to build a Docker image for you. You can choose from two options:

* Permit the Launch agent to build a Docker image, push the image to Amazon ECR, and submit [SageMaker Training](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateTrainingJob.html) jobs for you. This option can offer some simplicity to ML engineers who rapidly iterate over training code.
* Use an existing Docker image that contains your training or inference scripts. This option works well with existing CI systems. If you choose this option, you must manually upload your Docker image to your container registry on Amazon ECR.

### Set up AWS resources

You must have the following AWS resources configured in your preferred AWS region:

1. An [ECR repository](https://docs.aws.amazon.com/AmazonECR/latest/userguide/repository-create.html) to store container images.
2. One or more [S3 buckets](https://docs.aws.amazon.com/AmazonS3/latest/userguide/create-bucket-overview.html) to store inputs and outputs for your SageMaker Training jobs.
3. An IAM role for Amazon SageMaker that permits SageMaker to run training jobs and interact with Amazon ECR and Amazon S3.

Record the ARNs for these resources. You need the ARNs when you define the [Launch queue configuration](#configure-launch-queue-for-sagemaker).

### Create an IAM policy for the Launch agent

The Launch agent requires an IAM policy that grants the permissions needed to submit SageMaker training jobs and, optionally, to push images to ECR. Complete the following steps to create the policy:

1. From the IAM screen in AWS, create a new policy.
2. Toggle to the JSON policy editor, then paste the following policy based on your use case. Substitute placeholders in `[BRACKETS]` with your own values:

<Tabs>
  <Tab title="Agent submits pre-built Docker image">
    ```json theme={null}
      {
        "Version": "2012-10-17",
        "Statement": [
          {
            "Effect": "Allow",
            "Action": [
              "logs:DescribeLogStreams",
              "SageMaker:AddTags",
              "SageMaker:CreateTrainingJob",
              "SageMaker:DescribeTrainingJob"
            ],
            "Resource": "arn:aws:sagemaker:[REGION]:[ACCOUNT-ID]:*"
          },
          {
            "Effect": "Allow",
            "Action": "iam:PassRole",
            "Resource": "arn:aws:iam::[ACCOUNT-ID]:role/[ROLE-ARN-FROM-QUEUE-CONFIG]"
          },
        {
            "Effect": "Allow",
            "Action": "kms:CreateGrant",
            "Resource": "[ARN-OF-KMS-KEY]",
            "Condition": {
              "StringEquals": {
                "kms:ViaService": "SageMaker.[REGION].amazonaws.com",
                "kms:GrantIsForAWSResource": "true"
              }
            }
          }
        ]
      }
    ```
  </Tab>

  <Tab title="Agent builds and submits Docker image">
    ```json theme={null}
      {
        "Version": "2012-10-17",
        "Statement": [
          {
            "Effect": "Allow",
            "Action": [
              "logs:DescribeLogStreams",
              "SageMaker:AddTags",
              "SageMaker:CreateTrainingJob",
              "SageMaker:DescribeTrainingJob"
            ],
            "Resource": "arn:aws:sagemaker:[REGION]:[ACCOUNT-ID]:*"
          },
          {
            "Effect": "Allow",
            "Action": "iam:PassRole",
            "Resource": "arn:aws:iam::[ACCOUNT-ID]:role/[ROLE-ARN-FROM-QUEUE-CONFIG]"
          },
           {
          "Effect": "Allow",
          "Action": [
            "ecr:CreateRepository",
            "ecr:UploadLayerPart",
            "ecr:PutImage",
            "ecr:CompleteLayerUpload",
            "ecr:InitiateLayerUpload",
            "ecr:DescribeRepositories",
            "ecr:DescribeImages",
            "ecr:BatchCheckLayerAvailability",
            "ecr:BatchDeleteImage"
          ],
          "Resource": "arn:aws:ecr:[REGION]:[ACCOUNT-ID]:repository/[REPOSITORY]"
        },
        {
          "Effect": "Allow",
          "Action": "ecr:GetAuthorizationToken",
          "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": "kms:CreateGrant",
            "Resource": "[ARN-OF-KMS-KEY]",
            "Condition": {
              "StringEquals": {
                "kms:ViaService": "SageMaker.[REGION].amazonaws.com",
                "kms:GrantIsForAWSResource": "true"
              }
            }
          }
        ]
      }
    ```
  </Tab>
</Tabs>

3. Click **Next**.
4. Give the policy a name and description.
5. Click **Create policy**.

You now have an IAM policy that you can attach to the Launch agent role in the next section.

### Create an IAM role for the Launch agent

The Launch agent requires permission to create Amazon SageMaker training jobs. Attaching the policy you created in the preceding section to a dedicated role lets the agent assume those permissions at runtime. Follow the procedure to create an IAM role:

1. From the IAM screen in AWS, create a new role.
2. For **Trusted Entity**, select **AWS Account** (or another option that suits your organization's policies).
3. Scroll through the permissions screen and select the policy name you created in the preceding section.
4. Give the role a name and description.
5. Select **Create role**.
6. Record the ARN of the role. You specify the ARN when you set up the Launch agent.

To create IAM roles, see the [AWS Identity and Access Management Documentation](https://docs.aws.amazon.com/IAM/latest/UserGuide/introduction.html).

<Note>
  * If you want the Launch agent to build images, see the [Advanced agent set up](./setup-agent-advanced) for additional permissions required.
  * The `kms:CreateGrant` permission for SageMaker queues is required only if the associated `ResourceConfig` has a specified `VolumeKmsKeyId` and the associated role doesn't have a policy that permits this action.
</Note>

## Configure the Launch queue for SageMaker

With the AWS prerequisites in place, you can create the W\&B Launch queue that routes jobs to SageMaker. Create a queue in the W\&B App that uses SageMaker as its compute resource:

1. Navigate to the [Launch App](https://wandb.ai/launch).
2. Click **Create Queue**.
3. Select the **Entity** in which you want to create the queue.
4. Provide a name for your queue in the **Name** field.
5. Select **SageMaker** as the **Resource**.
6. Within the **Configuration** field, provide information about your SageMaker job. By default, W\&B populates a YAML and JSON `CreateTrainingJob` request body:
   ```json theme={null}
   {
     "RoleArn": "[REQUIRED]", 
     "ResourceConfig": {
         "InstanceType": "ml.m4.xlarge",
         "InstanceCount": 1,
         "VolumeSizeInGB": 2
     },
     "OutputDataConfig": {
         "S3OutputPath": "[REQUIRED]"
     },
     "StoppingCondition": {
         "MaxRuntimeInSeconds": 3600
     }
   }
   ```

You must at minimum specify:

* `RoleArn`: ARN of the SageMaker execution IAM role (see [prerequisites](#prerequisites)). Don't confuse this with the Launch **agent** IAM role.
* `OutputDataConfig.S3OutputPath`: An Amazon S3 URI that specifies where SageMaker stores outputs.
* `ResourceConfig`: Required specification of a resource config. For resource config options, see the [AWS `ResourceConfig` documentation](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_ResourceConfig.html).
* `StoppingCondition`: Required specification of the stopping conditions for the training job. For options, see the [AWS `StoppingCondition` documentation](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_StoppingCondition.html).

7. Click **Create Queue**.

Your queue is created and ready to receive jobs after you configure a Launch agent to poll it.

## Set up the Launch agent

The following sections describe where you can deploy your agent and how to configure your agent based on where you deploy it.

You have [several options for how to deploy the Launch agent for an Amazon SageMaker](#decide-where-to-run-the-launch-agent) queue: on a local machine, on an EC2 instance, or in an EKS cluster. [Configure your Launch agent](#configure-a-launch-agent) based on where you deploy your agent.

### Decide where to run the Launch agent

For production workloads and for customers who already have an EKS cluster, W\&B recommends that you deploy the Launch agent to the EKS cluster using this Helm chart.

For production workloads without a current EKS cluster, an EC2 instance is a good option. Although the Launch agent instance keeps running all the time, the agent doesn't need more than a `t2.micro` sized EC2 instance, which is affordable.

For experimental or solo use cases, you can run the Launch agent on your local machine as a fast way to get started.

Based on your use case, follow the instructions in the following tabs to configure your Launch agent:

<Tabs>
  <Tab title="EKS">
    W\&B recommends that you use the [W\&B managed Helm chart](https://github.com/wandb/helm-charts/tree/main/charts/launch-agent) to install the agent in an EKS cluster.
  </Tab>

  <Tab title="EC2">
    Navigate to the Amazon EC2 Dashboard and complete the following steps:

    1. Click **Launch instance**.
    2. Provide a name for the **Name** field. Optionally add a tag.
    3. From the **Instance type**, select an instance type for your EC2 container. You don't need more than 1 vCPU and 1 GiB of memory (for example, a `t2.micro`).
    4. Create a key pair for your organization within the **Key pair (login)** field. You use this key pair to [connect to your EC2 instance](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/connect.html) with SSH client at a later step.
    5. Within **Network settings**, select a security group for your organization.
    6. Expand **Advanced details**. For **IAM instance profile**, select the Launch agent IAM role you created in the preceding section.
    7. Review the **Summary** field. If correct, select **Launch instance**.

    Navigate to **Instances** within the left panel of the EC2 Dashboard on AWS. Ensure that the EC2 instance you created is running (see the **Instance state** column). After you confirm your EC2 instance is running, navigate to your local machine's terminal and complete the following:

    1. Select **Connect**.
    2. Select the **SSH client** tab and follow the instructions to connect to your EC2 instance.
    3. Within your EC2 instance, install the following packages:
       ```bash theme={null}
       sudo yum install python311 -y && python3 -m ensurepip --upgrade && pip3 install wandb && pip3 install wandb[launch]
       ```
    4. Next, install and start Docker within your EC2 instance:
       ```bash theme={null}
       sudo yum update -y && \
       sudo yum install -y docker python3 && \
       sudo systemctl start docker && \
       sudo systemctl enable docker && \
       sudo usermod -a -G docker ec2-user

       newgrp docker
       ```

    You can now set up the Launch agent config.
  </Tab>

  <Tab title="Local machine">
    Use the AWS config files located at `~/.aws/config` and `~/.aws/credentials` to associate a role with an agent that polls on a local machine. Provide the IAM role ARN that you created for the Launch agent in the preceding step.

    ```yaml title="~/.aws/config" theme={null}
    [profile SageMaker-agent]
    role_arn = arn:aws:iam::[ACCOUNT-ID]:role/[AGENT-ROLE-NAME]
    source_profile = default                                                                   
    ```

    ```yaml title="~/.aws/credentials" theme={null}
    [default]
    aws_access_key_id=[ACCESS-KEY-ID]
    aws_secret_access_key=[SECRET-ACCESS-KEY]
    aws_session_token=[SESSION-TOKEN]
    ```

    Session tokens have a [max length](https://docs.aws.amazon.com/cli/latest/reference/sts/get-session-token.html#description) of 1 hour or 3 days, depending on the associated principal.
  </Tab>
</Tabs>

### Configure a Launch agent

After you decide where to run the agent, configure it so that it can poll your SageMaker queue and authenticate to AWS. Configure the Launch agent with a YAML config file named `launch-config.yaml`.

By default, W\&B checks for the config file in `~/.config/wandb/launch-config.yaml`. You can optionally specify a different directory when you activate the Launch agent with the `-c` flag.

The following YAML snippet demonstrates how to specify the core config agent options:

```yaml title="launch-config.yaml" theme={null}
max_jobs: -1
queues:
  - [QUEUE-NAME]
environment:
  type: aws
  region: [YOUR-REGION]
registry:
  type: ecr
  uri: [ECR-REPO-ARN]
builder: 
  type: docker

```

Now start the agent with `wandb launch-agent`.

Your Launch agent is now running and polls the SageMaker queue for jobs.

## Optional: Push your Launch job Docker image to Amazon ECR

<Note>
  This section applies only if your Launch agent uses existing Docker images that contain your training or inference logic. [Your Launch agent supports two behavior options.](#decide-if-you-want-the-launch-agent-to-build-a-docker-image)
</Note>

Upload your Docker image that contains your Launch job to your Amazon ECR repo. Your Docker image must be in your ECR registry before you submit new Launch jobs if you use image-based jobs.


# Configure launch queue
Source: https://docs.wandb.ai/platform/launch/setup-queue-advanced

Configure advanced launch queue options including queue config templates with defaults, minimums, and maximums.

This page describes how to configure advanced launch queue options, including queue config templates, dynamic macros, and accelerator base images. These options help administrators enforce guardrails and tailor queues to specific compute environments.

## Set up queue config templates

Queue config templates let administrators manage guardrails on compute consumption. Set defaults, minimums, and maximum values for fields such as memory consumption, GPU, and runtime duration.

After you configure a queue with config templates, members of your team can alter the fields you defined only within the range you specified.

### Configure queue template

You can configure a queue template on an existing queue or on a new queue. The following procedure adds template fields to a queue's config so that team members can only set values within the limits you define.

1. Navigate to the [W\&B Launch App](https://wandb.ai/launch).
2. Select **View queue** next to the name of the queue you want to add a template to.
3. Select the **Config** tab. This shows information about your queue such as when the queue was created, the queue config, and existing launch-time overrides.
4. Navigate to the **Queue config** section.
5. Identify the config key-values you want to create a template for.
6. Replace the value in the config with a template field. Template fields take the form of `{{variable-name}}`.
7. Click the **Parse configuration** button. When you parse your configuration, W\&B automatically creates tiles below the queue config for each template you created.
8. For each tile generated, you must first specify the data type (string, integer, or float) the queue config can allow. To do this, select the data type from the **Type** dropdown menu.
9. Based on your data type, complete the fields that appear within each tile.
10. Click **Save config**.

After you save the config, the queue enforces the template constraints on subsequent launches.

For example, suppose you want to create a template that limits which AWS instances your team can use. Before you add a template field, your queue config might look something similar to:

```yaml title="launch config" theme={null}
RoleArn: arn:aws:iam:region:account-id:resource-type/resource-id
ResourceConfig:
  InstanceType: ml.m4.xlarge
  InstanceCount: 1
  VolumeSizeInGB: 2
OutputDataConfig:
  S3OutputPath: s3://bucketname
StoppingCondition:
  MaxRuntimeInSeconds: 3600
```

When you add a template field for the `InstanceType`, your config looks like:

```yaml title="launch config" theme={null}
RoleArn: arn:aws:iam:region:account-id:resource-type/resource-id
ResourceConfig:
  InstanceType: "{{aws_instance}}"
  InstanceCount: 1
  VolumeSizeInGB: 2
OutputDataConfig:
  S3OutputPath: s3://bucketname
StoppingCondition:
  MaxRuntimeInSeconds: 3600
```

Next, you click **Parse configuration**. A new tile labeled `aws-instance` appears underneath the **Queue config**.

From there, you select String as the data type from the **Type** dropdown. This populates fields where you can specify values a user can choose from. For example, in the following image the team admin configured two AWS instance types that users can choose from, `ml.m4.xlarge` and `ml.p3.xlarge`:

<Frame>
  <img alt="Launch queue template tile with AWS instance type options" />
</Frame>

## Dynamically configure launch jobs

You can dynamically configure queue configs using macros that the agent evaluates when it dequeues a job from the queue. Use macros to inject run-specific values into your queue config at launch time. You can set the following macros:

| Macro             | Description                                   |
| ----------------- | --------------------------------------------- |
| `${project_name}` | The name of the project the run launches to.  |
| `${entity_name}`  | The owner of the project the run launches to. |
| `${run_id}`       | The ID of the run that launches.              |
| `${run_name}`     | The name of the run that launches.            |
| `${image_uri}`    | The URI of the container image for this run.  |

<Note>
  Any custom macro not listed in the preceding table, for example `${MY_ENV_VAR}`, is substituted with an environment variable from the agent's environment.
</Note>

## Use the launch agent to build images that execute on accelerators or GPUs

If you use launch to build images that run in an accelerator environment, you might need to specify an accelerator base image so that the resulting image is compatible with the target hardware.

The accelerator base image must satisfy the following requirements:

* Debian compatibility. The Launch Dockerfile uses `apt-get` to fetch `python`.
* Compatible CPU and GPU hardware instruction sets. Make sure your CUDA version is supported by the GPU you intend to use.
* Compatibility between the accelerator version you provide and the packages installed in your ML algorithm.
* Any additional steps required to set up the installed packages for compatibility with the hardware.

### Use GPUs with TensorFlow

To ensure TensorFlow uses your GPU, specify a Docker image and its image tag for the `builder.accelerator.base_image` key in the queue resource configuration.

For example, the `tensorflow/tensorflow:latest-gpu` base image ensures TensorFlow uses your GPU. You can configure this using the resource configuration in the queue.

The following JSON snippet demonstrates how to specify the TensorFlow base image in your queue config:

```json title="Queue config" theme={null}
{
    "builder": {
        "accelerator": {
            "base_image": "tensorflow/tensorflow:latest-gpu"
        }
    }
}
```


# Tutorial: Set up W&B Launch on Vertex AI
Source: https://docs.wandb.ai/platform/launch/setup-vertex

Set up W&B Launch on Google Cloud Vertex AI using CustomJob specs and Artifact Registry for ML workloads.

This tutorial walks you through configuring W\&B Launch to submit jobs for execution as Vertex AI training jobs, so you can offload training workloads to Google Cloud's managed infrastructure. With Vertex AI training jobs, you can train machine learning models using either provided or custom algorithms on the Vertex AI platform. After you start a launch job, Vertex AI manages the underlying infrastructure, scaling, and orchestration. This guide is for ML engineers and platform administrators who already use W\&B Launch and want to run jobs on Google Cloud Vertex AI.

W\&B Launch works with Vertex AI through the `CustomJob` class in the `google-cloud-aiplatform` SDK. You can control the parameters of a `CustomJob` with the launch queue configuration. You can't configure Vertex AI to pull images from a private registry outside of Google Cloud. This means that you must store container images in Google Cloud or in a public registry if you want to use Vertex AI with W\&B Launch. See the Vertex AI documentation for more information about making container images accessible to Vertex jobs.

## Prerequisites

Before you configure a Launch queue, make sure the following Google Cloud resources and permissions are in place:

1. **Create or access a Google Cloud project with the Vertex AI API enabled.** See the [Google Cloud API Console docs](https://support.google.com/googleapi/answer/6158841?hl=en) for more information about enabling an API.
2. **Create a Google Cloud Artifact Registry repository** to store images you want to execute on Vertex. See the [Google Cloud Artifact Registry documentation](https://cloud.google.com/artifact-registry/docs/overview) for more information.
3. **Create a staging GCS bucket** for Vertex AI to store its metadata. This bucket must be in the same region as your Vertex AI workloads to serve as a staging bucket. You can use the same bucket for staging and build contexts.
4. **Create a service account** with the necessary permissions to spin up Vertex AI jobs. See the [Google Cloud IAM documentation](https://cloud.google.com/iam/docs/creating-managing-service-accounts) for more information about assigning permissions to service accounts.
5. **Grant your service account permission to manage Vertex jobs**, as shown in the following table:

| Permission                     | Resource Scope                 | Description                                                                            |
| ------------------------------ | ------------------------------ | -------------------------------------------------------------------------------------- |
| `aiplatform.customJobs.create` | Specified Google Cloud Project | Lets you create new machine learning jobs within the project.                          |
| `aiplatform.customJobs.list`   | Specified Google Cloud Project | Lets you list machine learning jobs within the project.                                |
| `aiplatform.customJobs.get`    | Specified Google Cloud Project | Lets you retrieve information about specific machine learning jobs within the project. |

<Note>
  If you want your Vertex AI workloads to assume the identity of a non-standard service account, refer to the Vertex AI documentation for instructions about service account creation and necessary permissions. Use the `spec.service_account` field of the launch queue configuration to select a custom service account for your W\&B runs.
</Note>

## Configure a queue for Vertex AI

With your Google Cloud prerequisites in place, the next step is to plan the queue configuration that W\&B Launch uses to submit Vertex AI jobs. The queue configuration for Vertex AI resources specifies inputs to the `CustomJob` constructor in the Vertex AI Python SDK, and the `run` method of the `CustomJob`. Resource configurations are stored under the `spec` and `run` keys:

* The `spec` key contains values for the named arguments of the [`CustomJob` constructor](https://cloud.google.com/vertex-ai/docs/pipelines/customjob-component) in the Vertex AI Python SDK.
* The `run` key contains values for the named arguments of the `run` method of the `CustomJob` class in the Vertex AI Python SDK.

Customization of the execution environment happens in the `spec.worker_pool_specs` list. A worker pool spec defines a group of workers that run your job. The worker spec in the default config asks for a single `n1-standard-4` machine with no accelerators. You can change the machine type, accelerator type, and count to suit your needs.

For more information about available machine types and accelerator types, see the [Vertex AI documentation](https://cloud.google.com/vertex-ai/docs/reference/rest/v1/MachineSpec).

## Create a queue

Now that you have planned your queue configuration, create a queue in the W\&B App that uses Vertex AI as its compute resource:

1. Navigate to the [Launch page](https://wandb.ai/launch).

2. Click the **Create Queue** button.

3. Select the **Entity** you want to create the queue in.

4. Provide a name for your queue in the **Name** field.

5. Select **Google Cloud Vertex AI** as the **Resource**.

6. Within the **Configuration** field, provide information about your Vertex AI `CustomJob` you defined in [Configure a queue for Vertex AI](#configure-a-queue-for-vertex-ai). By default, W\&B populates a YAML and JSON request body similar to the following:

   ```yaml theme={null}
   spec:
     worker_pool_specs:
       - machine_spec:
           machine_type: n1-standard-4
           accelerator_type: ACCELERATOR_TYPE_UNSPECIFIED
           accelerator_count: 0
         replica_count: 1
         container_spec:
           image_uri: ${image_uri}
     staging_bucket: [STAGING-BUCKET]
   run:
     restart_job_on_worker_restart: false
   ```

7. After you configure your queue, click the **Create Queue** button.

At minimum, you must specify the following fields:

* `spec.worker_pool_specs`: non-empty list of worker pool specifications.
* `spec.staging_bucket`: GCS bucket for staging Vertex AI assets and metadata.

<Warning>
  Some of the Vertex AI docs show worker pool specifications with all keys in camel case, for example, `workerPoolSpecs`. The Vertex AI Python SDK uses snake case for these keys, for example, `worker_pool_specs`.

  Every key in the launch queue configuration should use snake case.
</Warning>

## Configure a launch agent

With the queue created, configure a launch agent to poll the queue and dispatch jobs to Vertex AI. The launch agent is configurable through a config file that is, by default, located at `~/.config/wandb/launch-config.yaml`.

```yaml theme={null}
max_jobs: [N-CONCURRENT-JOBS]
queues:
  - [QUEUE-NAME]
```

If you want the launch agent to build images for you that are executed in Vertex AI, see [Advanced agent set up](./setup-agent-advanced).

## Set up agent permissions

Finally, give the launch agent the credentials it needs to act as the service account you created in the prerequisites. Multiple methods exist to authenticate as this service account. You can authenticate through Workload Identity, a downloaded service account JSON, environment variables, the Google Cloud Platform command-line tool, or a combination of these methods.


# Create sweeps with W&B Launch
Source: https://docs.wandb.ai/platform/launch/sweeps-on-launch

Discover how to automate hyperparamter sweeps on launch.

<Card title="Try in Colab" href="https://colab.research.google.com/drive/1WxLKaJlltThgZyhc7dcZhDQ6cjVQDfil#scrollTo=AFEzIxA6foC7" icon="python" />

This page describes how to create a hyperparameter tuning job ([sweeps](/models/sweeps/)) with W\&B Launch, so you can automate hyperparameter exploration on the same infrastructure that runs your other launch jobs. With sweeps on launch, a sweep scheduler is pushed to a Launch Queue with the specified hyperparameters to sweep over. The sweep scheduler starts as it is picked up by the agent, launching sweep runs onto the same queue with chosen hyperparameters. This continues until the sweep finishes or is stopped.

You can use the default W\&B Sweep scheduling engine or implement your own custom scheduler:

1. Standard sweep scheduler: Use the default W\&B Sweep scheduling engine that controls [W\&B Sweeps](/models/sweeps/). The familiar `bayes`, `grid`, and `random` methods are available.
2. Custom sweep scheduler: Configure the sweep scheduler to run as a job. This option enables full customization. The following section shows an example of how to extend the standard sweep scheduler to include more logging.

<Note>
  This guide assumes that W\&B Launch has been previously configured. If W\&B Launch isn't configured, see the [how to get started](./#how-to-get-started) section of the launch documentation.
</Note>

<Note>
  We recommend you create a sweep on launch using the `basic` method if you're a first-time user of sweeps on launch. Use a custom sweeps on launch scheduler when the standard W\&B scheduling engine doesn't meet your needs.
</Note>

## Create a sweep with a W\&B standard scheduler

This section describes how to create W\&B Sweeps with Launch using the standard scheduling engine. You can create a sweep interactively with the W\&B App or programmatically with the W\&B CLI. For advanced configurations of Launch sweeps, including the ability to customize the scheduler, use the CLI.

<Note>
  Before you create a sweep with W\&B Launch, ensure that you first create a job to sweep over. See the [Create a Job](/platform/launch/create-launch-job/) page for more information.
</Note>

<Tabs>
  <Tab title="W&B app">
    Create a sweep interactively with the W\&B App.

    1. Navigate to your W\&B project on the W\&B App.
    2. Select the sweeps icon in the project sidebar (broom image).
    3. Select the **Create Sweep** button.
    4. Click the **Configure Launch** button.
    5. From the **Job** dropdown menu, select the name of your job and the job version you want to create a sweep from.
    6. From the **Queue** dropdown menu, select a queue to run the sweep on.
    7. From the **Job Priority** dropdown, specify the priority of your launch job. A launch job's priority is set to "Medium" if the launch queue doesn't support prioritization.
    8. Optional: Configure override args for the run or sweep scheduler. For example, using the scheduler overrides, configure the number of concurrent runs the scheduler manages using `num_workers`.
    9. Optional: From the **Destination Project** dropdown menu, select a project to save the sweep to.
    10. Click **Save**.
    11. Select **Launch Sweep**.

    <Frame>
      <img alt="Launch sweep configuration" />
    </Frame>
  </Tab>

  <Tab title="CLI">
    Programmatically create a W\&B Sweep with Launch with the W\&B CLI.

    1. Create a Sweep configuration.
    2. Specify the full job name within your sweep configuration.
    3. Initialize a sweep agent.

    <Note>
      Steps 1 and 3 are the same steps you normally take when you create a W\&B Sweep.
    </Note>

    For example, the following code snippet specifies `'wandb/jobs/Hello World 2:latest'` for the job value:

    ```yaml theme={null}
    # launch-sweep-config.yaml

    job: 'wandb/jobs/Hello World 2:latest'
    description: sweep examples using launch jobs

    method: bayes
    metric:
      goal: minimize
      name: loss_metric
    parameters:
      learning_rate:
        max: 0.02
        min: 0
        distribution: uniform
      epochs:
        max: 20
        min: 0
        distribution: int_uniform

    # Optional scheduler parameters:

    # scheduler:
    #   num_workers: 1  # concurrent sweep runs
    #   docker_image: [BASE-IMAGE-FOR-THE-SCHEDULER]
    #   resource: [RESOURCE-TYPE-FOR-EXAMPLE-LOCAL-CONTAINER]
    #   resource_args:  # resource arguments passed to runs
    #     env: 
    #         - WANDB_API_KEY

    # Optional Launch Params
    # launch: 
    #    registry: [REGISTRY-FOR-IMAGE-PULLING]
    ```

    For information about how to create a sweep configuration, see the [Define sweep configuration](/platform/launch/sweeps-on-launch/) page.

    4. Initialize a sweep. Provide the path to your config file, the name of your job queue, your W\&B entity, and the name of the project.

    ```bash theme={null}
    wandb launch-sweep [PATH-TO-YAML-FILE] --queue [QUEUE-NAME] --entity [YOUR-ENTITY] --project [PROJECT-NAME]
    ```

    For more information about W\&B Sweeps, see the [Tune Hyperparameters](/models/sweeps/) chapter.
  </Tab>
</Tabs>

## Create a custom sweep scheduler

This section describes how to create a custom sweep scheduler when the standard scheduling engine doesn't meet your needs. You can create a custom sweep scheduler either with the W\&B scheduler or a custom scheduler.

<Note>
  Using scheduler jobs requires `wandb` CLI version >= `0.15.4`
</Note>

<Tabs>
  <Tab title="W&B scheduler">
    Create a launch sweep using the W\&B sweep scheduling logic as a job.

    1. Identify the W\&B scheduler job in the public wandb/sweep-jobs project, or use the job name:
       `'wandb/sweep-jobs/job-wandb-sweep-scheduler:latest'`.
    2. Construct a configuration yaml with an additional `scheduler` block that includes a `job` key pointing to this name, as shown in the following example.
    3. Use the `wandb launch-sweep` command with the new config.

    Example config:

    ```yaml theme={null}
    # launch-sweep-config.yaml  
    description: Launch sweep config using a scheduler job
    scheduler:
      job: wandb/sweep-jobs/job-wandb-sweep-scheduler:latest
      num_workers: 8  # allows 8 concurrent sweep runs

    # training/tuning job that the sweep runs will execute
    job: wandb/sweep-jobs/job-fashion-MNIST-train:latest
    method: grid
    parameters:
      learning_rate:
        min: 0.0001
        max: 0.1
    ```
  </Tab>

  <Tab title="Custom scheduler">
    Create custom schedulers by creating a scheduler-job. For the purposes of this guide, modify the `WandbScheduler` to provide more logging.

    1. Clone the `wandb/launch-jobs` repo (specifically: `wandb/launch-jobs/jobs/sweep_schedulers`)
    2. Modify the `wandb_scheduler.py` to achieve your desired increased logging. Example: Add logging to the function `_poll`. This is called once every polling cycle (configurable timing), before new sweep runs launch.
    3. Run the modified file to create a job, with: `python wandb_scheduler.py --project [PROJECT] --entity [ENTITY] --name CustomWandbScheduler`
    4. Identify the name of the job created, either in the UI or in the output of the previous call, which is a code-artifact job (unless otherwise specified).
    5. Create a sweep configuration where the scheduler points to your new job.

    ```yaml theme={null}
    ...
    scheduler:
      job: '[ENTITY]/[PROJECT]/job-CustomWandbScheduler:latest'
    ...
    ```
  </Tab>

  <Tab title="Optuna scheduler">
    Optuna is a hyperparameter optimization framework that uses a variety of algorithms to find the best hyperparameters for a given model (similar to W\&B). In addition to the [sampling algorithms](https://optuna.readthedocs.io/en/stable/reference/samplers/index.html), Optuna also provides a variety of [pruning algorithms](https://optuna.readthedocs.io/en/stable/reference/pruners.html) that you can use to terminate poorly performing runs early. This is useful when you run many runs, as it can save time and resources. The classes are configurable. Pass in the expected parameters in the `scheduler.settings.pruner/sampler.args` block of the config file.

    Create a launch sweep using Optuna's scheduling logic with a job.

    1. First, create your own job or use a pre-built Optuna scheduler image job.
       * See the [`wandb/launch-jobs`](https://github.com/wandb/launch-jobs/blob/main/jobs/sweep_schedulers) repo for examples on how to create your own job.
       * To use a pre-built Optuna image, you can either navigate to `job-optuna-sweep-scheduler` in the `wandb/sweep-jobs` project or use the job name: `wandb/sweep-jobs/job-optuna-sweep-scheduler:latest`.

    2. After you create a job, you can create a sweep. Construct a sweep config that includes a `scheduler` block with a `job` key pointing to the Optuna scheduler job, as shown in the following example.

    ```yaml theme={null}
      # optuna_config_basic.yaml
      description: A basic Optuna scheduler
      job: wandb/sweep-jobs/job-fashion-MNIST-train:latest
      run_cap: 5
      metric:
        name: epoch/val_loss
        goal: minimize

      scheduler:
        job: wandb/sweep-jobs/job-optuna-sweep-scheduler:latest
        resource: local-container  # required for scheduler jobs sourced from images
        num_workers: 2

        # optuna specific settings
        settings:
          pruner:
            type: PercentilePruner
            args:
              percentile: 25.0  # kill 75% of runs
              n_warmup_steps: 10  # pruning turned off for first x steps

      parameters:
        learning_rate:
          min: 0.0001
          max: 0.1
    ```

    3. Finally, launch the sweep to an active queue with the `launch-sweep` command:

    ```bash theme={null}
    wandb launch-sweep [CONFIG-YAML] -q [QUEUE] -p [PROJECT] -e [ENTITY]
    ```

    For the exact implementation of the Optuna sweep scheduler job, see [wandb/launch-jobs](https://github.com/wandb/launch-jobs/blob/main/jobs/sweep_schedulers/optuna_scheduler/optuna_scheduler.py). For more examples of what's possible with the Optuna scheduler, check out [wandb/examples](https://github.com/wandb/examples/tree/master/examples/launch/launch-sweeps/optuna-scheduler).
  </Tab>
</Tabs>

Examples of custom sweep scheduler jobs are available in the [wandb/launch-jobs](https://github.com/wandb/launch-jobs) repo under `jobs/sweep_schedulers`. This guide shows how to use the publicly available **W\&B Scheduler Job**, and demonstrates a process for creating custom sweep scheduler jobs.

## Resume sweeps on launch

This section describes how to resume a launch-sweep from a previously launched sweep, which is useful when you want to continue exploring with updated scheduler settings or on a different queue. Although you can't change hyperparameters and the training job, you can change scheduler-specific parameters and the queue it is pushed to.

<Note>
  If the initial sweep used a training job with an alias like `latest`, resuming can lead to different results if the latest job version has changed since the last run.
</Note>

1. Identify the sweep name/ID for a previously run launch sweep. The sweep ID is an eight-character string (for example, `hhd16935`) that you can find in your project on the W\&B App.
2. If you change the scheduler parameters, construct an updated config file.
3. In your terminal, execute the following command. Replace content wrapped in `[` and `]` with your information:

```bash theme={null}
wandb launch-sweep [OPTIONAL-CONFIG-YAML] --resume_id [SWEEP-ID] --queue [QUEUE-NAME]
```


# Tutorial: W&B Launch basics
Source: https://docs.wandb.ai/platform/launch/walkthrough

Follow this getting started tutorial for W&B Launch covering jobs, queues, agents, and running ML workloads.

## What is Launch

<Card title="Try in Colab" href="https://colab.research.google.com/drive/1wX0OSVxZJDHRsZaOaOEDx-lLUrO1hHgP" icon="python" />

This tutorial introduces ML practitioners and platform teams to W\&B Launch, and walks you through creating a launch job, setting up a queue, connecting an agent, and submitting jobs to run. By the end, you've run a containerized ML workload through W\&B Launch and have a working queue and agent that your team can reuse.

Scale training [runs](/models/runs/) from your desktop to a compute resource like Amazon SageMaker, Kubernetes, and more with W\&B Launch. After you configure Launch, you can run training scripts, model evaluation suites, prepare models for production inference, and more with a few clicks and commands.

## How it works

Launch is composed of three fundamental components: **launch jobs**, **queues**, and **agents**.

A [*launch job*](./launch-terminology#launch-job) is a blueprint for configuring and running tasks in your ML workflow. Once you have a launch job, you can add it to a [*launch queue*](./launch-terminology#launch-queue). A launch queue is a first-in, first-out (FIFO) queue where you can configure and submit your jobs to a particular compute target resource, such as Amazon SageMaker or a Kubernetes cluster.

As jobs are added to the queue, [*launch agents*](./launch-terminology#launch-agent) poll that queue and run the job on the system targeted by the queue.

<Frame>
  <img alt="W&B Launch overview diagram" />
</Frame>

Based on your use case, you (or someone on your team) configure the launch queue according to your chosen [compute resource target](./launch-terminology#target-resources) (for example, Amazon SageMaker) and deploy a launch agent on your own infrastructure.

See the [Terms and concepts](./launch-terminology) page for more information about Launch.

## Get started

Depending on your use case, explore the following resources to get started with W\&B Launch:

* If this is your first time using W\&B Launch, start with the [Launch walkthrough](#walkthrough) guide.
* Learn how to set up [W\&B Launch](/platform/launch/set-up-launch/).
* Create a [launch job](/platform/launch/launch-terminology/#launch-job).
* Check out the W\&B Launch [public jobs GitHub repository](https://github.com/wandb/launch-jobs) for templates of common tasks like [deploying to Triton](https://github.com/wandb/launch-jobs/tree/main/jobs/deploy_to_nvidia_triton), [evaluating an LLM](https://github.com/wandb/launch-jobs/tree/main/jobs/openai_evals), or more.
  * View launch jobs created from this repository in this public [`wandb/jobs` project](https://wandb.ai/wandb/jobs/jobs) W\&B project.

## Walkthrough

This page walks through the basics of the W\&B Launch workflow.

<Note>
  W\&B Launch runs machine learning workloads in containers. Familiarity with containers isn't required but may be helpful for this walkthrough. See the [Docker documentation](https://docs.docker.com/guides/docker-concepts/the-basics/what-is-a-container/) for a primer on containers.
</Note>

## Prerequisites

Before you get started, ensure you have satisfied the following prerequisites. These cover the account, tooling, and authentication you need to run the commands later in this walkthrough.

1. Sign up for an account at [https://wandb.ai/site](https://wandb.ai/site) and then log in to your W\&B account.
2. This walkthrough requires terminal access to a machine with a working Docker CLI and engine. See the [Docker installation guide](https://docs.docker.com/engine/install/) for more information.
3. Install W\&B Python SDK version `0.17.1` or higher:
   ```bash theme={null}
   pip install wandb>=0.17.1
   ```
4. In your terminal, run `wandb login` or set the `WANDB_API_KEY` environment variable to authenticate with W\&B.

<Tabs>
  <Tab title="Log in to W&B">
    In your terminal, run:

    ```bash theme={null}
    wandb login
    ```
  </Tab>

  <Tab title="Environment variable">
    ```bash theme={null}
        WANDB_API_KEY=[API-KEY]
    ```

    Replace `[API-KEY]` with your W\&B API key.
  </Tab>
</Tabs>

## Create a launch job

A launch job is the blueprint for the workload you want Launch to run, so you need one before you can submit work to a queue. Create a [launch job](./launch-terminology#launch-job) in one of three ways: with a Docker image, from a git repository, or from local source code:

<Tabs>
  <Tab title="With a Docker image">
    To run a pre-made container that logs a message to W\&B, open a terminal and run the following command:

    ```bash theme={null}
    wandb launch --docker-image wandb/job_hello_world:main --project launch-quickstart
    ```

    The preceding command downloads and runs the container image `wandb/job_hello_world:main`.

    Launch configures the container to report everything logged with `wandb` to the `launch-quickstart` project. The container logs a message to W\&B and displays a link to the run in W\&B. Click the link to view the run in the W\&B UI.
  </Tab>

  <Tab title="From a git repository">
    To launch the same hello-world job from its [source code in the W\&B Launch jobs repository](https://github.com/wandb/launch-jobs), run the following command:

    ```bash theme={null}
    wandb launch --uri https://github.com/wandb/launch-jobs.git \\
    --job-name hello-world-git --project launch-quickstart \\ 
    --build-context jobs/hello_world --dockerfile Dockerfile.wandb \\ 
    --entry-point "python job.py"
    ```

    The command does the following:

    1. Clones the [W\&B Launch jobs repository](https://github.com/wandb/launch-jobs) to a temporary directory.
    2. Creates a job named **hello-world-git** in the **hello** project. This job tracks the exact source code and configuration used to run the code.
    3. Builds a container image from the `jobs/hello_world` directory and the `Dockerfile.wandb`.
    4. Starts the container and runs the `job.py` python script.

    The console output shows the image build and run. The output of the container should be nearly identical to the previous example.
  </Tab>

  <Tab title="From local source code">
    You can launch code that isn't versioned in a git repository by specifying a local directory path to the `--uri` argument.

    Create an empty directory and add a Python script named `train.py` with the following content:

    ```python theme={null}
    import wandb

    with wandb.init() as run:
        run.log({"hello": "world"})
    ```

    Add a file `requirements.txt` with the following content:

    ```text theme={null}
    wandb>=0.17.1
    ```

    From within the directory, run the following command:

    ```bash theme={null}
    wandb launch --uri . --job-name hello-world-code --project launch-quickstart --entry-point "python train.py"
    ```

    The command does the following:

    1. Logs the contents of the current directory to W\&B as a Code Artifact.
    2. Creates a job named **hello-world-code** in the **launch-quickstart** project.
    3. Builds a container image by copying `train.py` and `requirements.txt` into a base image and `pip install` the requirements.
    4. Starts the container and runs `python train.py`.
  </Tab>
</Tabs>

## Create a queue

With a launch job in place, the next step is to create a queue that defines where and how jobs run. Launch is designed to help teams build workflows around shared compute. In the examples so far, the `wandb launch` command has run a container synchronously on the local machine. Launch queues and agents enable asynchronous runs of jobs on shared resources and advanced features like prioritization and hyperparameter optimization. To create a basic queue, follow these steps:

1. Navigate to [wandb.ai/launch](https://wandb.ai/launch) and click the **Create a queue** button.
2. Select an **Entity** to associate the queue with.
3. Enter a **Queue name**.
4. Select **Docker** as the **Resource**.
5. Leave **Configuration** blank.
6. Click **Create queue**.

After you click the button, the browser redirects to the **Agents** tab of the queue view. The queue remains in the **Not active** state until an agent starts polling.

<Frame>
  <img alt="Docker queue creation" />
</Frame>

For advanced queue configuration options, see the [advanced queue setup page](/platform/launch/setup-queue-advanced/).

## Connect an agent to the queue

A queue stays idle until an agent polls it, so you must connect an agent before any jobs can run. The queue view displays an **Add an agent** button in a red banner at the top of the screen if the queue has no polling agents. Click the button to view and copy the command to run an agent. The command should look like the following:

```bash theme={null}
wandb launch-agent --queue [QUEUE-NAME] --entity [ENTITY-NAME]
```

Replace `[QUEUE-NAME]` with the name of your queue and `[ENTITY-NAME]` with your W\&B entity. Run the command in a terminal to start the agent. The agent polls the specified queue for jobs to run. After it receives a job, the agent downloads or builds and then runs a container image for the job, as if you had run the `wandb launch` command locally.

Navigate back to [the Launch page](https://wandb.ai/launch) and verify that the queue now shows as **Active**.

## Submit a job to the queue

With a queue and a polling agent in place, you can now submit one of the launch jobs you created earlier and watch it run on the shared compute. Navigate to your **launch-quickstart** project in your W\&B account and open the jobs tab from the navigation on the left side of the screen.

The **Jobs** page displays a list of jobs created from previous runs. Click your launch job to view source code, dependencies, and any runs created from the job. After you complete this walkthrough, the list contains three jobs.

Pick one of the jobs and follow these instructions to submit it to the queue:

1. Click the **Launch** button to submit the job to a queue. The **Launch** drawer appears.
2. Select the **Queue** you created earlier and click **Launch**.

This submits the job to the queue. The agent polling this queue picks up and runs the job. You can monitor the progress of the job from the W\&B UI or by inspecting the output of the agent in the terminal.

The `wandb launch` command can push jobs to the queue directly by specifying the `--queue` argument. For example, to submit the hello-world container job to the queue, run the following command:

```bash theme={null}
wandb launch --docker-image wandb/job_hello_world:main --project launch-quickstart --queue [QUEUE-NAME]
```

Replace `[QUEUE-NAME]` with the name of your queue.

You now have a complete W\&B Launch workflow in place: a launch job, a queue, a polling agent, and a job running asynchronously on shared compute. You can reuse this pattern for your own training, evaluation, and inference workloads.


# Authentication & Access
Source: https://docs.wandb.ai/support/inference/tags/authentication-access



<Card title="API error code 401: Authentication failed" href="/support/inference/articles/api-error-code-401-authentication-failed">
  A 401 error with the message "Authentication failed" means your authentication credentials are incorrect, or your W\&B pr ...
</Card>

<Card title="API error code 403 - Country, region, or territory not supported" href="/support/inference/articles/api-error-code-403-country-region-or-ter">
  A 403 error with the message "Country, region, or territory not supported" means you're accessing Serverless Inference f ...
</Card>

<Card title="API error code 403: The inference gateway is not enabled for your organization" href="/support/inference/articles/api-error-code-403-the-inference-gateway">
  A 403 error with the message The inference gateway is not enabled for your organization means your organization doesn't  ...
</Card>


# Billing
Source: https://docs.wandb.ai/support/inference/tags/billing



<Card title="API error code 402 - You exceeded your current quota" href="/support/inference/articles/api-error-code-402-you-exceeded-your-cur">
  A 402 error with the message You exceeded your current quota, please check your plan and billing details means you've ru ...
</Card>


# Quotas & Rate Limits
Source: https://docs.wandb.ai/support/inference/tags/quotas-rate-limits



<Card title="API error code 402 - You exceeded your current quota" href="/support/inference/articles/api-error-code-402-you-exceeded-your-cur">
  A 402 error with the message You exceeded your current quota, please check your plan and billing details means you've ru ...
</Card>

<Card title="API error code 429 - Concurrency limit reached for requests" href="/support/inference/articles/api-error-code-429-concurrency-limit-rea">
  A 429 error with the message "Concurrency limit reached for requests" means you're sending too many concurrent requests  ...
</Card>


# Server Errors
Source: https://docs.wandb.ai/support/inference/tags/server-errors



<Card title="API error code 500 - The server had an error while processing your request" href="/support/inference/articles/api-error-code-500-the-server-had-an-err">
  A 500 error with the message The server had an error while processing your request indicates an internal server error in ...
</Card>

<Card title="API error code 503: The engine is currently overloaded" href="/support/inference/articles/api-error-code-503-the-engine-is-current">
  A 503 error with the message "The engine is currently overloaded, please try again later" means the Serverless Inference ...
</Card>