W&B Models

• 1: Experiments
• 1.1: Create an experiment
• 1.2: Configure experiments
• 1.3: Projects
• 1.4: View experiments results
• 1.5: What are runs?
• 1.5.1: Add labels to runs with tags
• 1.5.2: Filter and search runs
• 1.5.3: Fork a run
• 1.5.4: Group runs into experiments
• 1.5.5: Move runs
• 1.5.6: Resume a run
• 1.5.7: Rewind a run
• 1.5.8: Send an alert
• 1.6: Log objects and media
• 1.6.1: Create and track plots from experiments
• 1.6.2: Customize log axes
• 1.6.3: Log distributed training experiments
• 1.6.4: Log media and objects
• 1.6.5: Log models
• 1.6.6: Log summary metrics
• 1.6.7: Log tables
• 1.6.8: Track CSV files with experiments
• 1.7: Track Jupyter notebooks
• 1.8: Experiments limits and performance
• 1.9: Reproduce experiments
• 1.10: Import and export data
• 1.11: Environment variables
• 2: Sweeps
• 2.1: Tutorial: Define, initialize, and run a sweep
• 2.2: Add W&B (wandb) to your code
• 2.3: Define a sweep configuration
• 2.3.1: Sweep configuration options
• 2.4: Initialize a sweep
• 2.5: Start or stop a sweep agent
• 2.6: Parallelize agents
• 2.7: Visualize sweep results
• 2.8: Manage sweeps with the CLI
• 2.9: Learn more about sweeps
• 2.10: Manage algorithms locally
• 2.11: Sweeps troubleshooting
• 2.12: Sweeps UI
• 2.13: Tutorial: Create sweep job from project
• 3: Tables
• 3.1: Tutorial: Log tables, visualize and query data
• 3.2: Visualize and analyze tables
• 3.3: Example tables
• 3.4: Export table data
• 4: W&B App UI Reference
• 4.1: Panels
• 4.1.1: Line plots
• 4.1.1.1: Line plot reference
• 4.1.1.2: Point aggregation
• 4.1.1.3: Smooth line plots
• 4.1.2: Bar plots
• 4.1.3: Parallel coordinates
• 4.1.4: Scatter plots
• 4.1.5: Save and diff code
• 4.1.6: Parameter importance
• 4.1.7: Compare run metrics
• 4.1.8: Query panels
• 4.1.8.1: Embed objects
• 4.2: Custom charts
• 4.2.1: Tutorial: Use custom charts
• 4.3: Manage workspace, section, and panel settings
• 4.4: Settings
• 4.4.1: Manage user settings
• 4.4.2: Manage billing settings
• 4.4.3: Manage team settings
• 4.4.4: Manage email settings
• 4.4.5: Manage teams
• 4.4.6: Manage storage
• 4.4.7: System metrics
• 4.4.8: Anonymous mode

W&B Models is the system of record for ML Practitioners who want to organize their models, boost productivity and collaboration, and deliver production ML at scale.

With W&B Models, you can:

• Track and visualize all ML experiments.
• Optimize and fine-tune models at scale with hyperparameter sweeps.
• Maintain a centralized hub of all models, with a seamless handoff point to devops and deployment teams.
• Configure custom automations that trigger key workflows for model CI/CD.

Machine learning practitioners rely on W&B Models as their ML system of record to track and visualize experiments, manage model versions and lineage, and optimize hyperparameters.

1 - Experiments

Track machine learning experiments with a few lines of code. You can then review the results in an interactive dashboard or export your data to Python for programmatic access using our Public API.

Utilize W&B Integrations if you use popular frameworks such as PyTorch, Keras, or Scikit. See our Integration guides for a for a full list of integrations and information on how to add W&B to your code.

The image above shows an example dashboard where you can view and compare metrics across multiple runs.

How it works

Track a machine learning experiment with a few lines of code:

1. Create a W&B run.
2. Store a dictionary of hyperparameters, such as learning rate or model type, into your configuration (run.config).
3. Log metrics (run.log()) over time in a training loop, such as accuracy and loss.
4. Save outputs of a run, like the model weights or a table of predictions.

The following code demonstrates a common W&B experiment tracking workflow:

# Start a run.
#
# When this block exits, it waits for logged data to finish uploading.
# If an exception is raised, the run is marked failed.
with wandb.init(entity="", project="my-project-name") as run:
  # Save mode inputs and hyperparameters.
  run.config.learning_rate = 0.01

  # Run your experiment code.
  for epoch in range(num_epochs):
    # Do some training...

    # Log metrics over time to visualize model performance.
    run.log({"loss": loss})

  # Upload model outputs as artifacts.
  run.log_artifact(model)

Get started

Depending on your use case, explore the following resources to get started with W&B Experiments:

• Read the W&B Quickstart for a step-by-step outline of the W&B Python SDK commands you could use to create, track, and use a dataset artifact.
• Explore this chapter to learn how to:
  • Create an experiment
  • Configure experiments
  • Log data from experiments
  • View results from experiments
• Explore the W&B Python Library within the W&B API Reference Guide.

Best practices and tips

For best practices and tips for experiments and logging, see Best Practices: Experiments and Logging.

1.1 - Create an experiment

Use the W&B Python SDK to track machine learning experiments. You can then review the results in an interactive dashboard or export your data to Python for programmatic access with the W&B Public API.

This guide describes how to use W&B building blocks to create a W&B Experiment.

How to create a W&B Experiment

Create a W&B Experiment in four steps:

1. Initialize a W&B Run
2. Capture a dictionary of hyperparameters
3. Log metrics inside your training loop
4. Log an artifact to W&B

Initialize a W&B run

Use wandb.init() to create a W&B Run.

The following snippet creates a run in a W&B project named “cat-classification” with the description “My first experiment” to help identify this run. Tags “baseline” and “paper1” are included to remind us that this run is a baseline experiment intended for a future paper publication.

import wandb

with wandb.init(
    project="cat-classification",
    notes="My first experiment",
    tags=["baseline", "paper1"],
) as run:
    ...

wandb.init() returns a Run object.

Capture a dictionary of hyperparameters

Save a dictionary of hyperparameters such as learning rate or model type. The model settings you capture in config are useful later to organize and query your results.

with wandb.init(
    ...,
    config={"epochs": 100, "learning_rate": 0.001, "batch_size": 128},
) as run:
    ...

For more information on how to configure an experiment, see Configure Experiments.

Log metrics inside your training loop

Call run.log() to log metrics about each training step such as accuracy and loss.

model, dataloader = get_model(), get_data()

for epoch in range(run.config.epochs):
    for batch in dataloader:
        loss, accuracy = model.training_step()
        run.log({"accuracy": accuracy, "loss": loss})

For more information on different data types you can log with W&B, see Log Data During Experiments.

Log an artifact to W&B

Optionally log a W&B Artifact. Artifacts make it easy to version datasets and models.

# You can save any file or even a directory. In this example, we pretend
# the model has a save() method that outputs an ONNX file.
model.save("path_to_model.onnx")
run.log_artifact("path_to_model.onnx", name="trained-model", type="model")

For more information about Artifacts, see the Artifacts Chapter. For more information about versioning models, see Model Management.

Putting it all together

The full script with the preceding code snippets is found below:

import wandb

with wandb.init(
    project="cat-classification",
    notes="",
    tags=["baseline", "paper1"],
    # Record the run's hyperparameters.
    config={"epochs": 100, "learning_rate": 0.001, "batch_size": 128},
) as run:
    # Set up model and data.
    model, dataloader = get_model(), get_data()

    # Run your training while logging metrics to visualize model performance.
    for epoch in range(run.config["epochs"]):
        for batch in dataloader:
            loss, accuracy = model.training_step()
            run.log({"accuracy": accuracy, "loss": loss})

    # Upload the trained model as an artifact.
    model.save("path_to_model.onnx")
    run.log_artifact("path_to_model.onnx", name="trained-model", type="model")

Next steps: Visualize your experiment

Use the W&B Dashboard as a central place to organize and visualize results from your machine learning models. With just a few clicks, construct rich, interactive charts like parallel coordinates plots, parameter importance analyzes, and more.

For more information on how to view experiments and specific runs, see Visualize results from experiments.

Best practices

The following are some suggested guidelines to consider when you create experiments:

1. Finish your runs: Use wandb.init() in a with statement to automatically mark the run as finished when the code completes or raises an exception.

   • In Jupyter notebooks, it may be more convenient to manage the Run object yourself. In this case, you can explicitly call finish() on the Run object to mark it complete:

     # In a notebook cell:
     run = wandb.init()
     
     # In a different cell:
     run.finish()
     

2. Config: Track hyperparameters, architecture, dataset, and anything else you’d like to use to reproduce your model. These will show up in columns— use config columns to group, sort, and filter runs dynamically in the app.
3. Project: A project is a set of experiments you can compare together. Each project gets a dedicated dashboard page, and you can easily turn on and off different groups of runs to compare different model versions.
4. Notes: Set a quick commit message directly from your script. Edit and access notes in the Overview section of a run in the W&B App.
5. Tags: Identify baseline runs and favorite runs. You can filter runs using tags. You can edit tags at a later time on the Overview section of your project’s dashboard on the W&B App.
6. Create multiple run sets to compare experiments: When comparing experiments, create multiple run sets to make metrics easy to compare. You can toggle run sets on or off on the same chart or group of charts.

The following code snippet demonstrates how to define a W&B Experiment using the best practices listed above:

import wandb

config = {
    "learning_rate": 0.01,
    "momentum": 0.2,
    "architecture": "CNN",
    "dataset_id": "cats-0192",
}

with wandb.init(
    project="detect-cats",
    notes="tweak baseline",
    tags=["baseline", "paper1"],
    config=config,
) as run:
    ...

For more information about available parameters when defining a W&B Experiment, see the wandb.init API docs in the API Reference Guide.

1.2 - Configure experiments

Use the config property of a run to save your training configuration:

• hyperparameter
• input settings such as the dataset name or model type
• any other independent variables for your experiments.

The run.config property makes it easy to analyze your experiments and reproduce your work in the future. You can group by configuration values in the W&B App, compare the configurations of different W&B runs, and evaluate how each training configuration affects the output. The config property is a dictionary-like object that can be composed from multiple dictionary-like objects.

Set up an experiment configuration

Configurations are typically defined in the beginning of a training script. Machine learning workflows may vary, however, so you are not required to define a configuration at the beginning of your training script.

Use dashes (-) or underscores (_) instead of periods (.) in your config variable names.

Use the dictionary access syntax ["key"]["value"] instead of the attribute access syntax config.key.value if your script accesses run.config keys below the root.

The following sections outline different common scenarios of how to define your experiments configuration.

Set the configuration at initialization

Pass a dictionary at the beginning of your script when you call the wandb.init() API to generate a background process to sync and log data as a W&B Run.

The proceeding code snippet demonstrates how to define a Python dictionary with configuration values and how to pass that dictionary as an argument when you initialize a W&B Run.

import wandb

# Define a config dictionary object
config = {
    "hidden_layer_sizes": [32, 64],
    "kernel_sizes": [3],
    "activation": "ReLU",
    "pool_sizes": [2],
    "dropout": 0.5,
    "num_classes": 10,
}

# Pass the config dictionary when you initialize W&B
with wandb.init(project="config_example", config=config) as run:
    ...

If you pass a nested dictionary as the config, W&B flattens the names using dots.

Access the values from the dictionary similarly to how you access other dictionaries in Python:

# Access values with the key as the index value
hidden_layer_sizes = run.config["hidden_layer_sizes"]
kernel_sizes = run.config["kernel_sizes"]
activation = run.config["activation"]

# Python dictionary get() method
hidden_layer_sizes = run.config.get("hidden_layer_sizes")
kernel_sizes = run.config.get("kernel_sizes")
activation = run.config.get("activation")

Set the configuration with argparse

You can set your configuration with an argparse object. argparse, short for argument parser, is a standard library module in Python 3.2 and above that makes it easy to write scripts that take advantage of all the flexibility and power of command line arguments.

This is useful for tracking results from scripts that are launched from the command line.

The proceeding Python script demonstrates how to define a parser object to define and set your experiment config. The functions train_one_epoch and evaluate_one_epoch are provided to simulate a training loop for the purpose of this demonstration:

# config_experiment.py
import argparse
import random

import numpy as np
import wandb


# Training and evaluation demo code
def train_one_epoch(epoch, lr, bs):
    acc = 0.25 + ((epoch / 30) + (random.random() / 10))
    loss = 0.2 + (1 - ((epoch - 1) / 10 + random.random() / 5))
    return acc, loss


def evaluate_one_epoch(epoch):
    acc = 0.1 + ((epoch / 20) + (random.random() / 10))
    loss = 0.25 + (1 - ((epoch - 1) / 10 + random.random() / 6))
    return acc, loss


def main(args):
    # Start a W&B Run
    with wandb.init(project="config_example", config=args) as run:
        # Access values from config dictionary and store them
        # into variables for readability
        lr = run.config["learning_rate"]
        bs = run.config["batch_size"]
        epochs = run.config["epochs"]

        # Simulate training and logging values to W&B
        for epoch in np.arange(1, epochs):
            train_acc, train_loss = train_one_epoch(epoch, lr, bs)
            val_acc, val_loss = evaluate_one_epoch(epoch)

            run.log(
                {
                    "epoch": epoch,
                    "train_acc": train_acc,
                    "train_loss": train_loss,
                    "val_acc": val_acc,
                    "val_loss": val_loss,
                }
            )


if __name__ == "__main__":
    parser = argparse.ArgumentParser(
        formatter_class=argparse.ArgumentDefaultsHelpFormatter
    )

    parser.add_argument("-b", "--batch_size", type=int, default=32, help="Batch size")
    parser.add_argument(
        "-e", "--epochs", type=int, default=50, help="Number of training epochs"
    )
    parser.add_argument(
        "-lr", "--learning_rate", type=int, default=0.001, help="Learning rate"
    )

    args = parser.parse_args()
    main(args)

Set the configuration throughout your script

You can add more parameters to your config object throughout your script. The proceeding code snippet demonstrates how to add new key-value pairs to your config object:

import wandb

# Define a config dictionary object
config = {
    "hidden_layer_sizes": [32, 64],
    "kernel_sizes": [3],
    "activation": "ReLU",
    "pool_sizes": [2],
    "dropout": 0.5,
    "num_classes": 10,
}

# Pass the config dictionary when you initialize W&B
with wandb.init(project="config_example", config=config) as run:
    # Update config after you initialize W&B
    run.config["dropout"] = 0.2
    run.config.epochs = 4
    run.config["batch_size"] = 32

You can update multiple values at a time:

run.config.update({"lr": 0.1, "channels": 16})

Set the configuration after your Run has finished

Use the W&B Public API to update a completed run’s config.

You must provide the API with your entity, project name and the run’s ID. You can find these details in the Run object or in the W&B App UI:

with wandb.init() as run:
    ...

# Find the following values from the Run object if it was initiated from the
# current script or notebook, or you can copy them from the W&B App UI.
username = run.entity
project = run.project
run_id = run.id

# Note that api.run() returns a different type of object than wandb.init().
api = wandb.Api()
api_run = api.run(f"{username}/{project}/{run_id}")
api_run.config["bar"] = 32
api_run.update()

absl.FLAGS

You can also pass in absl flags.

flags.DEFINE_string("model", None, "model to run")  # name, default, help

run.config.update(flags.FLAGS)  # adds absl flags to config

File-Based Configs

If you place a file named config-defaults.yaml in the same directory as your run script, the run automatically picks up the key-value pairs defined in the file and passes them to run.config.

The following code snippet shows a sample config-defaults.yaml YAML file:

batch_size:
  desc: Size of each mini-batch
  value: 32

You can override the default values automatically loaded from config-defaults.yaml by setting updated values in the config argument of wandb.init. For example:

import wandb

# Override config-defaults.yaml by passing custom values
with wandb.init(config={"epochs": 200, "batch_size": 64}) as run:
    ...

To load a configuration file other than config-defaults.yaml, use the --configs command-line argument and specify the path to the file:

python train.py --configs other-config.yaml

Example use case for file-based configs

Suppose you have a YAML file with some metadata for the run, and then a dictionary of hyperparameters in your Python script. You can save both in the nested config object:

hyperparameter_defaults = dict(
    dropout=0.5,
    batch_size=100,
    learning_rate=0.001,
)

config_dictionary = dict(
    yaml=my_yaml_file,
    params=hyperparameter_defaults,
)

with wandb.init(config=config_dictionary) as run:
    ...

TensorFlow v1 flags

You can pass TensorFlow flags into the wandb.config object directly.

with wandb.init() as run:
    run.config.epochs = 4

    flags = tf.app.flags
    flags.DEFINE_string("data_dir", "/tmp/data")
    flags.DEFINE_integer("batch_size", 128, "Batch size.")
    run.config.update(flags.FLAGS)  # add tensorflow flags as config

1.3 - Projects

A project is a central location where you visualize results, compare experiments, view and download artifacts, create an automation, and more.

Each project contains the proceeding which you can access from the sidebar:

• Overview: snapshot of your project
• Workspace: personal visualization sandbox
• Runs: A table that lists all the runs in your project
• Automations: Automations configured in your project
• Sweeps: automated exploration and optimization
• Reports: saved snapshots of notes, runs, and graphs
• Artifacts: Contains all runs and the artifacts associated with that run

Overview tab

• Project name: The name of the project. W&B creates a project for you when you initialize a run with the name you provide for the project field. You can change the name of the project at any time by selecting the Edit button in the upper right corner.
• Description: A description of the project.
• Project visibility: The visibility of the project. The visibility setting that determines who can access it. See Project visibility for more information.
• Last active: Timestamp of the last time data is logged to this project
• Owner: The entity that owns this project
• Contributors: The number of users that contribute to this project
• Total runs: The total number of runs in this project
• Total compute: we add up all the run times in your project to get this total
• Undelete runs: Click the dropdown menu and click “Undelete all runs” to recover deleted runs in your project.
• Delete project: click the dot menu in the right corner to delete a project

View a live example

Workspace tab

A project’s workspace gives you a personal sandbox to compare experiments. Use projects to organize models that can be compared, working on the same problem with different architectures, hyperparameters, datasets, preprocessing etc.

Runs Sidebar: list of all the runs in your project.

• Dot menu: hover over a row in the sidebar to see the menu appear on the left side. Use this menu to rename a run, delete a run, or stop and active run.
• Visibility icon: click the eye to turn on and off runs on graphs
• Color: change the run color to another one of our presets or a custom color
• Search: search runs by name. This also filters visible runs in the plots.
• Filter: use the sidebar filter to narrow down the set of runs visible
• Group: select a config column to dynamically group your runs, for example by architecture. Grouping makes plots show up with a line along the mean value, and a shaded region for the variance of points on the graph.
• Sort: pick a value to sort your runs by, for example runs with the lowest loss or highest accuracy. Sorting will affect which runs show up on the graphs.
• Expand button: expand the sidebar into the full table
• Run count: the number in parentheses at the top is the total number of runs in the project. The number (N visualized) is the number of runs that have the eye turned on and are available to be visualized in each plot. In the example below, the graphs are only showing the first 10 of 183 runs. Edit a graph to increase the max number of runs visible.

If you pin, hide, or change the order of columns in the Runs tab, the Runs sidebar reflects these customizations.

Panels layout: use this scratch space to explore results, add and remove charts, and compare versions of your models based on different metrics

View a live example

Add a section of panels

Click the section dropdown menu and click “Add section” to create a new section for panels. You can rename sections, drag them to reorganize them, and expand and collapse sections.

Each section has options in the upper right corner:

• Switch to custom layout: The custom layout allows you to resize panels individually.
• Switch to standard layout: The standard layout lets you resize all panels in the section at once, and gives you pagination.
• Add section: Add a section above or below from the dropdown menu, or click the button at the bottom of the page to add a new section.
• Rename section: Change the title for your section.
• Export section to report: Save this section of panels to a new report.
• Delete section: Remove the whole section and all the charts. This can be undone with the undo button at the bottom of the page in the workspace bar.
• Add panel: Click the plus button to add a panel to the section.

Move panels between sections

Drag and drop panels to reorder and organize into sections. You can also click the “Move” button in the upper right corner of a panel to select a section to move the panel to.

Resize panels

• Standard layout: All panels maintain the same size, and there are pages of panels. You can resize the panels by clicking and dragging the lower right corner. Resize the section by clicking and dragging the lower right corner of the section.
• Custom layout: All panels are sized individually, and there are no pages.

Search for metrics

Use the search box in the workspace to filter down the panels. This search matches the panel titles, which are by default the name of the metrics visualized.

Runs tab

Use the Runs tab to filter, group, and sort your runs.

The proceeding tabs demonstrate some common actions you can take in the Runs tab.

• Customize columns
• Sort
• Filter
• Group

The Runs tab shows details about runs in the project. It shows a large number of columns by default.

• To view all visible columns, scroll the page horizontally.
• To change the order of the columns, drag a column to the left or right.
• To pin a column, hover over the column name, click the action menu .... that appears, then click Pin column. Pinned columns appear near the left of the page, after the Name column. To unpin a pinned column, choose Unpin column
• To hide a column, hover over the column name, click the action menu .... that appears, then click Hide column. To view all columns that are currently hidden, click Columns.
• To show, hide, pin, and unpin multiple columns at once, click Columns.
  • Click the name of a hidden column to unhide it.
  • Click the name of a visible column to hide it.
  • Click the pin icon next to a visible column to pin it.

When you customize the Runs tab, the customization is also reflected in the Runs selector of the Workspace tab.

Sort all rows in a Table by the value in a given column.

1. Hover your mouse over the column title. A kebob menu will appear (three vertical docs).
2. Select on the kebob menu (three vertical dots).
3. Choose Sort Asc or Sort Desc to sort the rows in ascending or descending order, respectively.

The preceding image demonstrates how to view sorting options for a Table column called val_acc.

Filter all rows by an expression with the Filter button on the top left of the dashboard.

Select Add filter to add one or more filters to your rows. Three dropdown menus will appear. From left to right the filter types are based on: Column name, Operator , and Values

	Column name	Binary relation	Value
Accepted values	String	=, ≠, ≤, ≥, IN, NOT IN,	Integer, float, string, timestamp, null

The expression editor shows a list of options for each term using autocomplete on column names and logical predicate structure. You can connect multiple logical predicates into one expression using “and” or “or” (and sometimes parentheses).

The preceding image shows a filter that is based on the `val_loss` column. The filter shows runs with a validation loss less than or equal to 1.

Group all rows by the value in a particular column with the Group by button in a column header.

By default, this turns other numeric columns into histograms showing the distribution of values for that column across the group. Grouping is helpful for understanding higher-level patterns in your data.

Reports tab

See all the snapshots of results in one place, and share findings with your team.

Sweeps tab

Start a new sweep from your project.

Artifacts tab

View all the artifacts associated with a project, from training datasets and fine-tuned models to tables of metrics and media.

Overview panel

On the overview panel, you’ll find a variety of high-level information about the artifact, including its name and version, the hash digest used to detect changes and prevent duplication, the creation date, and any aliases. You can add or remove aliases here, take notes on both the version as well as the artifact as a whole.

Metadata panel

The metadata panel provides access to the artifact’s metadata, which is provided when the artifact is constructed. This metadata might include configuration arguments required to reconstruct the artifact, URLs where more information can be found, or metrics produced during the run which logged the artifact. Additionally, you can see the configuration for the run which produced the artifact as well as the history metrics at the time of logging the artifact.

Usage panel

The Usage panel provides a code snippet for downloading the artifact for use outside of the web app, for example on a local machine. This section also indicates and links to the run which output the artifact and any runs which use the artifact as an input.

Files panel

The files panel lists the files and folders associated with the artifact. W&B uploads certain files for a run automatically. For example, requirements.txt shows the versions of each library the run used, and wandb-metadata.json, and wandb-summary.json include information about the run. Other files may be uploaded, such as artifacts or media, depending on the run’s configuration. You can navigate through this file tree and view the contents directly in the W&B web app.

Tables associated with artifacts are particularly rich and interactive in this context. Learn more about using Tables with Artifacts here.

Lineage panel

The lineage panel provides a view of all of the artifacts associated with a project and the runs that connect them to each other. It shows run types as blocks and artifacts as circles, with arrows to indicate when a run of a given type consumes or produces an artifact of a given type. The type of the particular artifact selected in the left-hand column is highlighted.

Click the Explode toggle to view all of the individual artifact versions and the specific runs that connect them.

Action History Audit tab

The action history audit tab shows all of the alias actions and membership changes for a Collection so you can audit the entire evolution of the resource.

Versions tab

The versions tab shows all versions of the artifact as well as columns for each of the numeric values of the Run History at the time of logging the version. This allows you to compare performance and quickly identify versions of interest.

Star a project

Add a star to a project to mark that project as important. Projects that you and your team mark as important with stars appear at the top of your organization’s home page.

For example, the proceeding image shows two projects that are marked as important, the zoo_experiment and registry_demo. Both projects appear within the top of the organization’s home page within the Starred projects section.

There are two ways to mark a project as important: within a project’s overview tab or within your team’s profile page.

• Project overview
• Team profile

1. Navigate to your W&B project on the W&B App at https://wandb.ai/<team>/<project-name>.
2. Select the Overview tab from the project sidebar.
3. Choose the star icon in the upper right corner next to the Edit button.

1. Navigate to your team’s profile page at https://wandb.ai/<team>/projects.
2. Select the Projects tab.
3. Hover your mouse next to the project you want to star. Click on star icon that appears.

For example, the proceeding image shows the star icon next to the “Compare_Zoo_Models” project.

Confirm that your project appears on the landing page of your organization by clicking on the organization name in the top left corner of the app.

Delete a project

You can delete your project by clicking the three dots on the right of the overview tab.

If the project is empty, you can delete it by clicking the dropdown menu in the top-right and selecting Delete project.

Add notes to a project

Add notes to your project either as a description overview or as a markdown panel within your workspace.

Add description overview to a project

Descriptions you add to your page appear in the Overview tab of your profile.

1. Navigate to your W&B project
2. Select the Overview tab from the project sidebar
3. Choose Edit in the upper right hand corner
4. Add your notes in the Description field
5. Select the Save button

Add notes to run workspace

1. Navigate to your W&B project
2. Select the Workspace tab from the project sidebar
3. Choose the Add panels button from the top right corner
4. Select the TEXT AND CODE dropdown from the modal that appears
5. Select Markdown
6. Add your notes in the markdown panel that appears in your workspace

1.4 - View experiments results

W&B workspace is your personal sandbox to customize charts and explore model results. A W&B workspace consists of Tables and Panel sections:

• Tables: All runs logged to your project are listed in the project’s table. Turn on and off runs, change colors, and expand the table to see notes, config, and summary metrics for each run.
• Panel sections: A section that contains one or more panels. Create new panels, organize them, and export to reports to save snapshots of your workspace.

Workspace types

There are two main workspace categories: Personal workspaces and Saved views.

• Personal workspaces: A customizable workspace for in-depth analysis of models and data visualizations. Only the owner of the workspace can edit and save changes. Teammates can view a personal workspace but teammates can not make changes to someone else’s personal workspace.
• Saved views: Saved views are collaborative snapshots of a workspace. Anyone on your team can view, edit, and save changes to saved workspace views. Use saved workspace views for reviewing and discussing experiments, runs, and more.

The proceeding image shows multiple personal workspaces created by Cécile-parker’s teammates. In this project, there are no saved views:

Saved workspace views

Improve team collaboration with tailored workspace views. Create Saved Views to organize your preferred setup of charts and data.

Create a new saved workspace view

1. Navigate to a personal workspace or a saved view.
2. Make edits to the workspace.
3. Click on the meatball menu (three horizontal dots) at the top right corner of your workspace. Click on Save as a new view.

New saved views appear in the workspace navigation menu.

Update a saved workspace view

Saved changes overwrite the previous state of the saved view. Unsaved changes are not retained. To update a saved workspace view in W&B:

1. Navigate to a saved view.
2. Make the desired changes to your charts and data within the workspace.
3. Click the Save button to confirm your changes.

Delete a saved workspace view

Remove saved views that are no longer needed.

1. Navigate to the saved view you want to remove.
2. Select the three horizontal lines (…) at the top right of the view.
3. Choose Delete view.
4. Confirm the deletion to remove the view from your workspace menu.

Share a workspace view

Share your customized workspace with your team by sharing the workspace URL directly. All users with access to the workspace project can see the saved Views of that workspace.

Programmatically creating workspaces

wandb-workspaces is a Python library for programmatically working with W&B workspaces and reports.

Define a workspace programmatically with wandb-workspaces. wandb-workspaces is a Python library for programmatically working with W&B workspaces and reports.

You can define the workspace’s properties, such as:

• Set panel layouts, colors, and section orders.
• Configure workspace settings like default x-axis, section order, and collapse states.
• Add and customize panels within sections to organize workspace views.
• Load and modify existing workspaces using a URL.
• Save changes to existing workspaces or save as new views.
• Filter, group, and sort runs programmatically using simple expressions.
• Customize run appearance with settings like colors and visibility.
• Copy views from one workspace to another for integration and reuse.

Install Workspace API

In addition to wandb, ensure that you install wandb-workspaces:

pip install wandb wandb-workspaces

Define and save a workspace view programmatically

import wandb_workspaces.reports.v2 as wr

workspace = ws.Workspace(entity="your-entity", project="your-project", views=[...])
workspace.save()

Edit an existing view

existing_workspace = ws.Workspace.from_url("workspace-url")
existing_workspace.views[0] = ws.View(name="my-new-view", sections=[...])
existing_workspace.save()

Copy a workspace saved view to another workspace

old_workspace = ws.Workspace.from_url("old-workspace-url")
old_workspace_view = old_workspace.views[0]
new_workspace = ws.Workspace(entity="new-entity", project="new-project", views=[old_workspace_view])

new_workspace.save()

See wandb-workspace examples for comprehensive workspace API examples. For an end to end tutorial, see Programmatic Workspaces tutorial.

1.5 - What are runs?

A run is a single unit of computation logged by W&B. You can think of a W&B run as an atomic element of your whole project. In other words, each run is a record of a specific computation, such as training a model and logging the results, hyperparameter sweeps, and so forth.

Common patterns for initiating a run include, but are not limited to:

• Training a model
• Changing a hyperparameter and conducting a new experiment
• Conducting a new machine learning experiment with a different model
• Logging data or a model as a W&B Artifact
• Downloading a W&B Artifact

W&B stores runs that you create into projects. You can view runs and their properties within the run’s project workspace on the W&B App UI. You can also programmatically access run properties with the wandb.Api.Run object.

Anything you log with run.log is recorded in that run. Consider the proceeding code snippet.

import wandb

run = wandb.init(entity="nico", project="awesome-project")
run.log({"accuracy": 0.9, "loss": 0.1})

The first line imports the W&B Python SDK. The second line initializes a run in the project awesome-project under the entity nico. The third line logs the accuracy and loss of the model to that run.

Within the terminal, W&B returns:

wandb: Syncing run earnest-sunset-1
wandb: ⭐️ View project at https://wandb.ai/nico/awesome-project
wandb: 🚀 View run at https://wandb.ai/nico/awesome-project/runs/1jx1ud12
wandb:                                                                                
wandb: 
wandb: Run history:
wandb: accuracy ▁
wandb:     loss ▁
wandb: 
wandb: Run summary:
wandb: accuracy 0.9
wandb:     loss 0.5
wandb: 
wandb: 🚀 View run earnest-sunset-1 at: https://wandb.ai/nico/awesome-project/runs/1jx1ud12
wandb: ⭐️ View project at: https://wandb.ai/nico/awesome-project
wandb: Synced 6 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)
wandb: Find logs at: ./wandb/run-20241105_111006-1jx1ud12/logs

The URL W&B returns in the terminal to redirects you to the run’s workspace in the W&B App UI. Note that the panels generated in the workspace corresponds to the single point.

Logging a metrics at a single point of time might not be that useful. A more realistic example in the case of training discriminative models is to log metrics at regular intervals. For example, consider the proceeding code snippet:

epochs = 10
lr = 0.01

run = wandb.init(
    entity="nico",
    project="awesome-project",
    config={
        "learning_rate": lr,
        "epochs": epochs,
    },
)

offset = random.random() / 5

# simulating a training run
for epoch in range(epochs):
    acc = 1 - 2**-epoch - random.random() / (epoch + 1) - offset
    loss = 2**-epoch + random.random() / (epoch + 1) + offset
    print(f"epoch={epoch}, accuracy={acc}, loss={loss}")
    run.log({"accuracy": acc, "loss": loss})

This returns the following output:

wandb: Syncing run jolly-haze-4
wandb: ⭐️ View project at https://wandb.ai/nico/awesome-project
wandb: 🚀 View run at https://wandb.ai/nico/awesome-project/runs/pdo5110r
lr: 0.01
epoch=0, accuracy=-0.10070974957523078, loss=1.985328507123956
epoch=1, accuracy=0.2884687745057535, loss=0.7374362314407752
epoch=2, accuracy=0.7347387967382066, loss=0.4402409835486663
epoch=3, accuracy=0.7667969248039795, loss=0.26176963846423457
epoch=4, accuracy=0.7446848791003173, loss=0.24808611724405083
epoch=5, accuracy=0.8035095836268268, loss=0.16169791827329466
epoch=6, accuracy=0.861349032371624, loss=0.03432578493587426
epoch=7, accuracy=0.8794926436276016, loss=0.10331872172219471
epoch=8, accuracy=0.9424839917077272, loss=0.07767793473500445
epoch=9, accuracy=0.9584880427028566, loss=0.10531971149250456
wandb: 🚀 View run jolly-haze-4 at: https://wandb.ai/nico/awesome-project/runs/pdo5110r
wandb: Find logs at: wandb/run-20241105_111816-pdo5110r/logs

The training script calls run.log 10 times. Each time the script calls run.log, W&B logs the accuracy and loss for that epoch. Selecting the URL that W&B prints from the preceding output, directs you to the run’s workspace in the W&B App UI.

Note that W&B captures the simulated training loop within a single run called jolly-haze-4. This is because the script calls wandb.init method only once.

As another example, during a sweep, W&B explores a hyperparameter search space that you specify. W&B implements each new hyperparameter combination that the sweep creates as a unique run.

Initialize a run

Initialize a W&B run with wandb.init(). The proceeding code snippet shows how to import the W&B Python SDK and initialize a run.

Ensure to replace values enclosed in angle brackets (< >) with your own values:

import wandb

run = wandb.init(entity="<entity>", project="<project>")

When you initialize a run, W&B logs your run to the project you specify for the project field (wandb.init(project="<project>"). W&B creates a new project if the project does not already exist. If the project already exists, W&B stores the run in that project.

Each run in W&B has a unique identifier known as a run ID. You can specify a unique ID or let W&B randomly generate one for you.

Each run also has a human-readable, non-unique identifier known as a run name. You can specify a name for your run or let W&B randomly generate one for you.

For example, consider the proceeding code snippet:

import wandb

run = wandb.init(entity="wandbee", project="awesome-project")

The code snippet produces the proceeding output:

🚀 View run exalted-darkness-6 at: 
https://wandb.ai/nico/awesome-project/runs/pgbn9y21
Find logs at: wandb/run-20241106_090747-pgbn9y21/logs

Since the preceding code did not specify an argument for the id parameter, W&B creates a unique run ID. Where nico is the entity that logged the run, awesome-project is the name of the project the run is logged to, exalted-darkness-6 is the name of the run, and pgbn9y21 is the run ID.

import wandb

run = wandb.init(entity="<entity>", project="<project>")
# Training code, logging, and so forth
run.finish()

Each run has a state that describes the current status of the run. See Run states for a full list of possible run states.

Run states

The proceeding table describes the possible states a run can be in:

Unique run identifiers

Run IDs are unique identifiers for runs. By default, W&B generates a random and unique run ID for you when you initialize a new run. You can also specify your own unique run ID when you initialize a run.

Autogenerated run IDs

If you do not specify a run ID when you initialize a run, W&B generates a random run ID for you. You can find the unique ID of a run in the W&B App UI.

1. Navigate to the W&B App UI at https://wandb.ai/home.
2. Navigate to the W&B project you specified when you initialized the run.
3. Within your project’s workspace, select the Runs tab.
4. Select the Overview tab.

W&B displays the unique run ID in the Run path field. The run path consists of the name of your team, the name of the project, and the run ID. The unique ID is the last part of the run path.

For example, in the proceeding image, the unique run ID is 9mxi1arc:

Custom run IDs

You can specify your own run ID by passing the id parameter to the wandb.init method.

import wandb

run = wandb.init(entity="<project>", project="<project>", id="<run-id>")

You can use a run’s unique ID to directly navigate to the run’s overview page in the W&B App UI. The proceeding cell shows the URL path for a specific run:

https://wandb.ai/<entity>/<project>/<run-id>

Where values enclosed in angle brackets (< >) are placeholders for the actual values of the entity, project, and run ID.

Name your run

The name of a run is a human-readable, non-unique identifier.

By default, W&B generates a random run name when you initialize a new run. The name of a run appears within your project’s workspace and at the top of the run’s overview page.

You can specify a name for your run by passing the name parameter to the wandb.init method.

import wandb

run = wandb.init(entity="<project>", project="<project>", name="<run-name>")

Add a note to a run

Notes that you add to a specific run appear on the run page in the Overview tab and in the table of runs on the project page.

1. Navigate to your W&B project
2. Select the Workspace tab from the project sidebar
3. Select the run you want to add a note to from the run selector
4. Choose the Overview tab
5. Select the pencil icon next to the Description field and add your notes

Stop a run

Stop a run from the W&B App or programmatically.

• Programmatically
• W&B App

1. Navigate to the terminal or code editor where you initialized the run.
2. Press Ctrl+D to stop the run.

For example, following the preceding instructions, your terminal might looks similar to the following:

KeyboardInterrupt
wandb: 🚀 View run legendary-meadow-2 at: https://wandb.ai/nico/history-blaster-4/runs/o8sdbztv
wandb: Synced 5 W&B file(s), 0 media file(s), 0 artifact file(s) and 1 other file(s)
wandb: Find logs at: ./wandb/run-20241106_095857-o8sdbztv/logs

Navigate to the W&B App UI to confirm the run is no longer active:

1. Navigate to the project that your run was logging to.
2. Select the name of the run.

You can find the name of the run that you stop from the output of your terminal or code editor. For example, in the preceding example, the name of the run is legendary-meadow-2.

3. Choose the **Overview** tab from the project sidebar.

Next to the State field, the run’s state changes from running to Killed.

1. Navigate to the project that your run is logging to.
2. Select the run you want to stop within the run selector.
3. Choose the Overview tab from the project sidebar.
4. Select the top button next to the State field.

Next to the State field, the run’s state changes from running to Killed.

See State fields for a full list of possible run states.

View logged runs

View a information about a specific run such as the state of the run, artifacts logged to the run, log files recorded during the run, and more.

To view a specific run:

1. Navigate to the W&B App UI at https://wandb.ai/home.

2. Navigate to the W&B project you specified when you initialized the run.

3. Within the project sidebar, select the Workspace tab.

4. Within the run selector, click the run you want to view, or enter a partial run name to filter for matching runs.

   By default, long run names are truncated in the middle for readability. To truncate run names at the beginning or end instead, click the action ... menu at the top of the list of runs, then set Run name cropping to crop the end, middle, or beginning.

Note that the URL path of a specific run has the proceeding format:

https://wandb.ai/<team-name>/<project-name>/runs/<run-id>

Where values enclosed in angle brackets (< >) are placeholders for the actual values of the team name, project name, and run ID.

Overview tab

Use the Overview tab to learn about specific run information in a project, such as:

• Author: The W&B entity that creates the run.
• Command: The command that initializes the run.
• Description: A description of the run that you provided. This field is empty if you do not specify a description when you create the run. You can add a description to a run with the W&B App UI or programmatically with the Python SDK.
• Duration: The amount of time the run is actively computing or logging data, excluding any pauses or waiting.
• Git repository: The git repository associated with the run. You must enable git to view this field.
• Host name: Where W&B computes the run. W&B displays the name of your machine if you initialize the run locally on your machine.
• Name: The name of the run.
• OS: Operating system that initializes the run.
• Python executable: The command that starts the run.
• Python version: Specifies the Python version that creates the run.
• Run path: Identifies the unique run identifier in the form entity/project/run-ID.
• Runtime: Measures the total time from the start to the end of the run. It’s the wall-clock time for the run. Runtime includes any time where the run is paused or waiting for resources, while duration does not.
• Start time: The timestamp when you initialize the run.
• State: The state of the run.
• System hardware: The hardware W&B uses to compute the run.
• Tags: A list of strings. Tags are useful for organizing related runs together or applying temporary labels like baseline or production.
• W&B CLI version: The W&B CLI version installed on the machine that hosted the run command.

W&B stores the proceeding information below the overview section:

• Artifact Outputs: Artifact outputs produced by the run.
• Config: List of config parameters saved with wandb.config.
• Summary: List of summary parameters saved with wandb.log(). By default, W&B sets this value to the last value logged.

View an example project overview here.

Workspace tab

Use the Workspace tab to view, search, group, and arrange visualizations such as autogenerated and custom plots, system metrics, and more.

View an example project workspace here

Runs tab

Use the Runs tab to filter, group, and sort your runs.

The proceeding tabs demonstrate some common actions you can take in the Runs tab.

• Customize columns
• Sort
• Filter
• Group

The Runs tab shows details about runs in the project. It shows a large number of columns by default.

• To view all visible columns, scroll the page horizontally.
• To change the order of the columns, drag a column to the left or right.
• To pin a column, hover over the column name, click the action menu .... that appears, then click Pin column. Pinned columns appear near the left of the page, after the Name column. To unpin a pinned column, choose Unpin column
• To hide a column, hover over the column name, click the action menu .... that appears, then click Hide column. To view all columns that are currently hidden, click Columns.
• To show, hide, pin, and unpin multiple columns at once, click Columns.
  • Click the name of a hidden column to unhide it.
  • Click the name of a visible column to hide it.
  • Click the pin icon next to a visible column to pin it.

When you customize the Runs tab, the customization is also reflected in the Runs selector of the Workspace tab.

Sort all rows in a Table by the value in a given column.

1. Hover your mouse over the column title. A kebob menu will appear (three vertical docs).
2. Select on the kebob menu (three vertical dots).
3. Choose Sort Asc or Sort Desc to sort the rows in ascending or descending order, respectively.

The preceding image demonstrates how to view sorting options for a Table column called val_acc.

Filter all rows by an expression with the Filter button above the dashboard.

Select Add filter to add one or more filters to your rows. Three dropdown menus will appear. From left to right the filter types are based on: Column name, Operator , and Values

	Column name	Binary relation	Value
Accepted values	String	=, ≠, ≤, ≥, IN, NOT IN,	Integer, float, string, timestamp, null

The expression editor shows a list of options for each term using autocomplete on column names and logical predicate structure. You can connect multiple logical predicates into one expression using “and” or “or” (and sometimes parentheses).

The preceding image shows a filter that is based on the `val_loss` column. The filter shows runs with a validation loss less than or equal to 1.

Group all rows by the value in a particular column with the Group by button above the dashboard.

By default, this turns other numeric columns into histograms showing the distribution of values for that column across the group. Grouping is helpful for understanding higher-level patterns in your data.

System tab

The System tab shows system metrics tracked for a specific run such as CPU utilization, system memory, disk I/O, network traffic, GPU utilization and more.

For a full list of system metrics W&B tracks, see System metrics.

View an example system tab here.

Logs tab

The Log tab shows output printed on the command line such as the standard output (stdout) and standard error (stderr).

Choose the Download button in the upper right hand corner to download the log file.

View an example logs tab here.

Files tab

Use the Files tab to view files associated with a specific run such as model checkpoints, validation set examples, and more

View an example files tab here.

Artifacts tab

The Artifacts tab lists the input and output artifacts for the specified run.

View an example artifacts tab here.

Delete runs

Delete one or more runs from a project with the W&B App.

1. Navigate to the project that contains the runs you want to delete.
2. Select the Runs tab from the project sidebar.
3. Select the checkbox next to the runs you want to delete.
4. Choose the Delete button (trash can icon) above the table.
5. From the modal that appears, choose Delete.

Organize runs

This section provides instructions on how to organize runs using groups and job types. By assigning runs to groups (for example, experiment names) and specifying job types (for example, preprocessing, training, evaluation, debugging), you can streamline your workflow and improve model comparison.

Assign a group or job type to a run

Each run in W&B can be categorized by group and a job type:

• Group: a broad category for the experiment, used to organize and filter runs.
• Job type: the function of the run, such as preprocessing, training, or evaluation.

The proceeding example workspace, trains a baseline model using increasing amounts of data from the Fashion-MNIST dataset. The workspace uses colorts to represent the amount of data used:

• Yellow to dark green indicate increasing amounts of data for the baseline model.
• Light blue to violet to magenta indicate amounts of data for a more complex “double” model with additional parameters.

Use W&B’s filtering options and search bar to compare runs based on specific conditions, such as:

• Training on the same dataset.
• Evaluating on the same test set.

When you apply filters, the Table view is updated automatically. This allows you to identify performance differences between models, such as determining which classes are significantly more challenging for one model compared to another.

1.5.1 - Add labels to runs with tags

Add tags to label runs with particular features that might not be obvious from the logged metrics or artifact data.

For example, you can add a tag to a run to indicated that run’s model is in_production, that run is preemptible, this run represents the baseline, and so forth.

Add tags to one or more runs

Programmatically or interactively add tags to your runs.

Based on your use case, select the tab below that best fits your needs:

• W&B Python SDK
• Public API
• Project page
• Run page

import wandb

run = wandb.init(
  entity="entity",
  project="<project-name>",
  tags=["tag1", "tag2"]
)

import wandb

run = wandb.init(
  entity="entity", 
  project="capsules", 
  tags=["debug"]
  )

# python logic to train model

if current_loss < threshold:
    run.tags = run.tags + ("release_candidate",)

run = wandb.Api().run("{entity}/{project}/{run-id}")
run.tags.append("tag1")  # you can choose tags based on run data here
run.update()

Remove tags from one or more runs

Tags can also be removed from runs with the W&B App UI.

• Project page
• Run page

1.5.2 - Filter and search runs

Use your project page to gain insights from runs logged to W&B. You can filter and search runs from both the Workspace page and the Runs page.

Filter runs

Filter runs based on their status, tags, or other properties with the filter button.

Filter runs with tags

Filter runs based on their tags with the filter button.

Filter runs with regex

If regex doesn’t provide you the desired results, you can make use of tags to filter out the runs in Runs Table. Tags can be added either on run creation or after they’re finished. Once the tags are added to a run, you can add a tag filter as shown in the gif below.

Search runs

Use regex to find runs with the regex you specify. When you type a query in the search box, that will filter down the visible runs in the graphs on the workspace as well as filtering the rows of the table.

Group runs

To group runs by one or more columns (including hidden columns):

1. Below the search box, click the Group button, which looks like a lined sheet of paper.
2. Select one or more columns to group results by.
3. Each set of grouped runs is collapsed by default. To expand it, click the arrow next to the group name.

Sort runs by minimum and maximum values

Sort the runs table by the minimum or maximum value of a logged metric. This is particularly useful if you want to view the best (or worst) recorded value.

The following steps describe how to sort the run table by a specific metric based on the minimum or maximum recorded value:

1. Hover your mouse over the column with the metric you want to sort with.
2. Select the kebob menu (three vertical lines).
3. From the dropdown, select either Show min or Show max.
4. From the same dropdown, select Sort by asc or Sort by desc to sort in ascending or descending order, respectively.

Search End Time for runs

We provide a column named End Time that logs that last heartbeat from the client process. The field is hidden by default.

Export runs table to CSV

Export the table of all your runs, hyperparameters, and summary metrics to a CSV with the download button.

1.5.3 - Fork a run

Use fork_from when you initialize a run with wandb.init() to “fork” from an existing W&B run. When you fork from a run, W&B creates a new run using the run ID and step of the source run.

Forking a run enables you to explore different parameters or models from a specific point in an experiment without impacting the original run.

Start a forked run

To fork a run, use the fork_from argument in wandb.init() and specify the source run ID and the step from the source run to fork from:

import wandb

# Initialize a run to be forked later
original_run = wandb.init(project="your_project_name", entity="your_entity_name")
# ... perform training or logging ...
original_run.finish()

# Fork the run from a specific step
forked_run = wandb.init(
    project="your_project_name",
    entity="your_entity_name",
    fork_from=f"{original_run.id}?_step=200",
)

Using an immutable run ID

Use an immutable run ID to ensure you have a consistent and unchanging reference to a specific run. Follow these steps to obtain the immutable run ID from the user interface:

1. Access the Overview Tab: Navigate to the Overview tab on the source run’s page.

2. Copy the Immutable Run ID: Click on the ... menu (three dots) located in the top-right corner of the Overview tab. Select the Copy Immutable Run ID option from the dropdown menu.

By following these steps, you will have a stable and unchanging reference to the run, which can be used for forking a run.

Continue from a forked run

After initializing a forked run, you can continue logging to the new run. You can log the same metrics for continuity and introduce new metrics.

For example, the following code example shows how to first fork a run and then how to log metrics to the forked run starting from a training step of 200:

import wandb
import math

# Initialize the first run and log some metrics
run1 = wandb.init("your_project_name", entity="your_entity_name")
for i in range(300):
    run1.log({"metric": i})
run1.finish()

# Fork from the first run at a specific step and log the metric starting from step 200
run2 = wandb.init(
    "your_project_name", entity="your_entity_name", fork_from=f"{run1.id}?_step=200"
)

# Continue logging in the new run
# For the first few steps, log the metric as is from run1
# After step 250, start logging the spikey pattern
for i in range(200, 300):
    if i < 250:
        run2.log({"metric": i})  # Continue logging from run1 without spikes
    else:
        # Introduce the spikey behavior starting from step 250
        subtle_spike = i + (2 * math.sin(i / 3.0))  # Apply a subtle spikey pattern
        run2.log({"metric": subtle_spike})
    # Additionally log the new metric at all steps
    run2.log({"additional_metric": i * 1.1})
run2.finish()

1.5.4 - Group runs into experiments

Group individual jobs into experiments by passing a unique group name to wandb.init().

Use cases

1. Distributed training: Use grouping if your experiments are split up into different pieces with separate training and evaluation scripts that should be viewed as parts of a larger whole.
2. Multiple processes: Group multiple smaller processes together into an experiment.
3. K-fold cross-validation: Group together runs with different random seeds to see a larger experiment. Here’s an example of k-fold cross-validation with sweeps and grouping.

There are three ways to set grouping:

1. Set group in your script

Pass an optional group and job_type to wandb.init(). This gives you a dedicated group page for each experiment, which contains the individual runs. For example:wandb.init(group="experiment_1", job_type="eval")

2. Set a group environment variable

Use WANDB_RUN_GROUP to specify a group for your runs as an environment variable. For more on this, check our docs for Environment Variables. Group should be unique within your project and shared by all runs in the group. You can use wandb.util.generate_id() to generate a unique 8 character string to use in all your processes— for example, os.environ["WANDB_RUN_GROUP"] = "experiment-" + wandb.util.generate_id()

3. Toggle grouping in the UI

You can dynamically group by any config column. For example, if you use wandb.config to log batch size or learning rate, you can then group by those hyperparameters dynamically in the web app.

Distributed training with grouping

Suppose you set grouping in wandb.init(), we will group runs by default in the UI. You can toggle this on and off by clicking the Group button at the top of the table. Here’s an example project generated from sample code where we set grouping. You can click on each “Group” row in the sidebar to get to a dedicated group page for that experiment.

From the project page above, you can click a Group in the left sidebar to get to a dedicated page like this one:

Grouping dynamically in the UI

You can group runs by any column, for example by hyperparameter. Here’s an example of what that looks like:

• Sidebar: Runs are grouped by the number of epochs.
• Graphs: Each line represents the group’s mean, and the shading indicates the variance. This behavior can be changed in the graph settings.

Turn off grouping

Click the grouping button and clear group fields at any time, which returns the table and graphs to their ungrouped state.

Grouping graph settings

Click the edit button in the upper right corner of a graph and select the Advanced tab to change the line and shading. You can select the mean, minimum, or maximum value for the line in each group. For the shading, you can turn off shading, and show the min and max, the standard deviation, and the standard error.

1.5.5 - Move runs

This page shows how to move a run from one project to another, into or out of a team, or from one team to another. You must have access to the run at its current and new locations.

To customize the Runs tab, refer to Project page.

Move runs between your projects

To move runs from one project to another:

1. Navigate to the project that contains the runs you want to move.
2. Select the Runs tab from the project sidebar.
3. Select the checkbox next to the runs you want to move.
4. Choose the Move button above the table.
5. Select the destination project from the dropdown.

Move runs to a team

Move runs to a team you are a member of:

1. Navigate to the project that contains the runs you want to move.
2. Select the Runs tab from the project sidebar.
3. Select the checkbox next to the runs you want to move.
4. Choose the Move button above the table.
5. Select the destination team and project from the dropdown.

1.5.6 - Resume a run

Specify how a run should behave in the event that run stops or crashes. To resume or enable a run to automatically resume, you will need to specify the unique run ID associated with that run for the id parameter:

run = wandb.init(entity="<entity>", \ 
        project="<project>", id="<run ID>", resume="<resume>")

Pass one of the following arguments to the resume parameter to determine how W&B should respond. In each case, W&B first checks if the run ID already exists.

You can also specify resume="auto" to let W&B to automatically try to restart the run on your behalf. However, you will need to ensure that you restart your run from the same directory. See the Enable runs to automatically resume section for more information.

For all the examples below, replace values enclosed within <> with your own.

Resume a run that must use the same run ID

If a run is stopped, crashes, or fails, you can resume it using the same run ID. To do so, initialize a run and specify the following:

• Set the resume parameter to "must" (resume="must")
• Provide the run ID of the run that stopped or crashed

The following code snippet shows how to accomplish this with the W&B Python SDK:

run = wandb.init(entity="<entity>", \ 
        project="<project>", id="<run ID>", resume="must")

Resume a run without overriding the existing run

Resume a run that stopped or crashed without overriding the existing run. This is especially helpful if your process doesn’t exit successfully. The next time you start W&B, W&B will start logging from the last step.

Set the resume parameter to "allow" (resume="allow") when you initialize a run with W&B. Provide the run ID of the run that stopped or crashed. The following code snippet shows how to accomplish this with the W&B Python SDK:

import wandb

run = wandb.init(entity="<entity>", \ 
        project="<project>", id="<run ID>", resume="allow")

Enable runs to automatically resume

The following code snippet shows how to enable runs to automatically resume with the Python SDK or with environment variables.

• W&B Python SDK
• Shell script

run = wandb.init(entity="<entity>", \ 
        project="<project>", id="<run ID>", resume="<resume>")

RUN_ID="$1"

WANDB_RESUME=allow WANDB_RUN_ID="$RUN_ID" python eval.py

sh run_experiment.sh akj172 

For example, suppose you execute a python script called train.py in a directory called Users/AwesomeEmployee/Desktop/ImageClassify/training/. Within train.py, the script creates a run that enables automatic resuming. Suppose next that the training script is stopped. To resume this run, you would need to restart your train.py script within Users/AwesomeEmployee/Desktop/ImageClassify/training/ .

Resume preemptible Sweeps runs

Automatically requeue interrupted sweep runs. This is particularly useful if you run a sweep agent in a compute environment that is subject to preemption such as a SLURM job in a preemptible queue, an EC2 spot instance, or a Google Cloud preemptible VM.

Use the mark_preempting function to enable W&B to automatically requeue interrupted sweep runs. For example, the following code snippet

run = wandb.init()  # Initialize a run
run.mark_preempting()

The following table outlines how W&B handles runs based on the exit status of the a sweep run.

1.5.7 - Rewind a run

Rewind a run

Rewind a run to correct or modify the history of a run without losing the original data. In addition, when you rewind a run, you can log new data from that point in time. W&B recomputes the summary metrics for the run you rewind based on the newly logged history. This means the following behavior:

• History truncation: W&B truncates the history to the rewind point, allowing new data logging.
• Summary metrics: Recomputed based on the newly logged history.
• Configuration preservation: W&B preserves the original configurations and you can merge new configurations.

When you rewind a run, W&B resets the state of the run to the specified step, preserving the original data and maintaining a consistent run ID. This means that:

• Run archiving: W&B archives the original runs. Runs are accessible from the Run Overview tab.
• Artifact association: Associates artifacts with the run that produce them.
• Immutable run IDs: Introduced for consistent forking from a precise state.
• Copy immutable run ID: A button to copy the immutable run ID for improved run management.

Rewind a run

Use resume_from with wandb.init() to “rewind” a run’s history to a specific step. Specify the name of the run and the step you want to rewind from:

import wandb
import math

# Initialize the first run and log some metrics
# Replace with your_project_name and your_entity_name!
run1 = wandb.init(project="your_project_name", entity="your_entity_name")
for i in range(300):
    run1.log({"metric": i})
run1.finish()

# Rewind from the first run at a specific step and log the metric starting from step 200
run2 = wandb.init(project="your_project_name", entity="your_entity_name", resume_from=f"{run1.id}?_step=200")

# Continue logging in the new run
# For the first few steps, log the metric as is from run1
# After step 250, start logging the spikey pattern
for i in range(200, 300):
    if i < 250:
        run2.log({"metric": i, "step": i})  # Continue logging from run1 without spikes
    else:
        # Introduce the spikey behavior starting from step 250
        subtle_spike = i + (2 * math.sin(i / 3.0))  # Apply a subtle spikey pattern
        run2.log({"metric": subtle_spike, "step": i})
    # Additionally log the new metric at all steps
    run2.log({"additional_metric": i * 1.1, "step": i})
run2.finish()

View an archived run

After you rewind a run, you can explore archived run with the W&B App UI. Follow these steps to view archived runs:

1. Access the Overview Tab: Navigate to the Overview tab on the run’s page. This tab provides a comprehensive view of the run’s details and history.
2. Locate the Forked From field: Within the Overview tab, find the Forked From field. This field captures the history of the resumptions. The Forked From field includes a link to the source run, allowing you to trace back to the original run and understand the entire rewind history.

By using the Forked From field, you can effortlessly navigate the tree of archived resumptions and gain insights into the sequence and origin of each rewind.

Fork from a run that you rewind

To fork from a rewound run, use the fork_from argument in wandb.init() and specify the source run ID and the step from the source run to fork from:

import wandb

# Fork the run from a specific step
forked_run = wandb.init(
    project="your_project_name",
    entity="your_entity_name",
    fork_from=f"{rewind_run.id}?_step=500",
)

# Continue logging in the new run
for i in range(500, 1000):
    forked_run.log({"metric": i*3})
forked_run.finish()

1.5.8 - Send an alert

Create alerts with Slack or email if your run crashes or with a custom trigger. For example, you can create an alert if the gradient of your training loop starts to blow up (reports NaN) or a step in your ML pipeline completes. Alerts apply to all projects where you initialize runs, including both personal and team projects.

And then see W&B Alerts messages in Slack (or your email):

How to create an alert

There are two main steps to set up an alert:

1. Turn on Alerts in your W&B User Settings
2. Add run.alert() to your code
3. Confirm alert is set up properly

1. Turn on alerts in your W&B User Settings

In your User Settings:

• Scroll to the Alerts section
• Turn on Scriptable run alerts to receive alerts from run.alert()
• Use Connect Slack to pick a Slack channel to post alerts. We recommend the Slackbot channel because it keeps the alerts private.
• Email will go to the email address you used when you signed up for W&B. We recommend setting up a filter in your email so all these alerts go into a folder and don’t fill up your inbox.

You will only have to do this the first time you set up W&B Alerts, or when you’d like to modify how you receive alerts.

2. Add run.alert() to your code

Add run.alert() to your code (either in a Notebook or Python script) wherever you’d like it to be triggered

import wandb

run = wandb.init()
run.alert(title="High Loss", text="Loss is increasing rapidly")

3. Check your Slack or email

Check your Slack or emails for the alert message. If you didn’t receive any, make sure you’ve got emails or Slack turned on for Scriptable Alerts in your User Settings

Example

This simple alert sends a warning when accuracy falls below a threshold. In this example, it only sends alerts at least 5 minutes apart.

import wandb
from wandb import AlertLevel

run = wandb.init()

if acc < threshold:
    run.alert(
        title="Low accuracy",
        text=f"Accuracy {acc} is below the acceptable threshold {threshold}",
        level=AlertLevel.WARN,
        wait_duration=300,
    )

How to tag or mention users

Use the at sign @ followed by the Slack user ID to tag yourself or your colleagues in either the title or the text of the alert. You can find a Slack user ID from their Slack profile page.

run.alert(title="Loss is NaN", text=f"Hey <@U1234ABCD> loss has gone to NaN")

Team alerts

Team admins can set up alerts for the team on the team settings page: wandb.ai/teams/your-team.

Team alerts apply to everyone on your team. W&B recommends using the Slackbot channel because it keeps alerts private.

Change Slack channel to send alerts to

To change what channel alerts are sent to, click Disconnect Slack and then reconnect. After you reconnect, pick a different Slack channel.

1.6 - Log objects and media

Log a dictionary of metrics, media, or custom objects to a step with the W&B Python SDK. W&B collects the key-value pairs during each step and stores them in one unified dictionary each time you log data with wandb.log(). Data logged from your script is saved locally to your machine in a directory called wandb, then synced to the W&B cloud or your private server.

Each call to wandb.log is a new step by default. W&B uses steps as the default x-axis when it creates charts and panels. You can optionally create and use a custom x-axis or capture a custom summary metric. For more information, see Customize log axes.

Automatically logged data

W&B automatically logs the following information during a W&B Experiment:

• System metrics: CPU and GPU utilization, network, etc. These are shown in the System tab on the run page. For the GPU, these are fetched with nvidia-smi.
• Command line: The stdout and stderr are picked up and show in the logs tab on the run page.

Turn on Code Saving in your account’s Settings page to log:

• Git commit: Pick up the latest git commit and see it on the overview tab of the run page, as well as a diff.patch file if there are any uncommitted changes.
• Dependencies: The requirements.txt file will be uploaded and shown on the files tab of the run page, along with any files you save to the wandb directory for the run.

What data is logged with specific W&B API calls?

With W&B, you can decide exactly what you want to log. The following lists some commonly logged objects:

• Datasets: You have to specifically log images or other dataset samples for them to stream to W&B.
• Plots: Use wandb.plot with wandb.log to track charts. See Log Plots for more information.
• Tables: Use wandb.Table to log data to visualize and query with W&B. See Log Tables for more information.
• PyTorch gradients: Add wandb.watch(model) to see gradients of the weights as histograms in the UI.
• Configuration information: Log hyperparameters, a link to your dataset, or the name of the architecture you’re using as config parameters, passed in like this: wandb.init(config=your_config_dictionary). See the PyTorch Integrations page for more information.
• Metrics: Use wandb.log to see metrics from your model. If you log metrics like accuracy and loss from inside your training loop, you’ll get live updating graphs in the UI.

Common workflows

1. Compare the best accuracy: To compare the best value of a metric across runs, set the summary value for that metric. By default, summary is set to the last value you logged for each key. This is useful in the table in the UI, where you can sort and filter runs based on their summary metrics, to help compare runs in a table or bar chart based on their best accuracy, instead of final accuracy. For example: wandb.run.summary["best_accuracy"] = best_accuracy
2. View multiple metrics on one chart: Log multiple metrics in the same call to wandb.log, like this: wandb.log({"acc'": 0.9, "loss": 0.1}) and they will both be available to plot against in the UI
3. Customize the x-axis: Add a custom x-axis to the same log call to visualize your metrics against a different axis in the W&B dashboard. For example: wandb.log({'acc': 0.9, 'epoch': 3, 'batch': 117}). To set the default x-axis for a given metric use Run.define_metric()
4. Log rich media and charts: wandb.log supports the logging of a wide variety of data types, from media like images and videos to tables and charts.

Best practices and tips

For best practices and tips for Experiments and logging, see Best Practices: Experiments and Logging.

1.6.1 - Create and track plots from experiments

Using the methods in wandb.plot, you can track charts with wandb.log, including charts that change over time during training. To learn more about our custom charting framework, check out this guide.

Basic charts

These simple charts make it easy to construct basic visualizations of metrics and results.

• Line
• Scatter
• Bar
• Histogram
• Multi-line

wandb.plot.line()

Log a custom line plot—a list of connected and ordered points on arbitrary axes.

data = [[x, y] for (x, y) in zip(x_values, y_values)]
table = wandb.Table(data=data, columns=["x", "y"])
wandb.log(
    {
        "my_custom_plot_id": wandb.plot.line(
            table, "x", "y", title="Custom Y vs X Line Plot"
        )
    }
)

You can use this to log curves on any two dimensions. If you’re plotting two lists of values against each other, the number of values in the lists must match exactly. For example, each point must have an x and a y.

See in the app

Run the code

wandb.plot.scatter()

Log a custom scatter plot—a list of points (x, y) on a pair of arbitrary axes x and y.

data = [[x, y] for (x, y) in zip(class_x_scores, class_y_scores)]
table = wandb.Table(data=data, columns=["class_x", "class_y"])
wandb.log({"my_custom_id": wandb.plot.scatter(table, "class_x", "class_y")})

You can use this to log scatter points on any two dimensions. If you’re plotting two lists of values against each other, the number of values in the lists must match exactly. For example, each point must have an x and a y.

See in the app

Run the code

wandb.plot.bar()

Log a custom bar chart—a list of labeled values as bars—natively in a few lines:

data = [[label, val] for (label, val) in zip(labels, values)]
table = wandb.Table(data=data, columns=["label", "value"])
wandb.log(
    {
        "my_bar_chart_id": wandb.plot.bar(
            table, "label", "value", title="Custom Bar Chart"
        )
    }
)

You can use this to log arbitrary bar charts. The number of labels and values in the lists must match exactly. Each data point must have both.

See in the app

Run the code

wandb.plot.histogram()

Log a custom histogram—sort a list of values into bins by count/frequency of occurrence—natively in a few lines. Let’s say I have a list of prediction confidence scores (scores) and want to visualize their distribution:

data = [[s] for s in scores]
table = wandb.Table(data=data, columns=["scores"])
wandb.log({"my_histogram": wandb.plot.histogram(table, "scores", title="Histogram")})

You can use this to log arbitrary histograms. Note that data is a list of lists, intended to support a 2D array of rows and columns.

See in the app

Run the code

wandb.plot.line_series()

Plot multiple lines, or multiple different lists of x-y coordinate pairs, on one shared set of x-y axes:

wandb.log(
    {
        "my_custom_id": wandb.plot.line_series(
            xs=[0, 1, 2, 3, 4],
            ys=[[10, 20, 30, 40, 50], [0.5, 11, 72, 3, 41]],
            keys=["metric Y", "metric Z"],
            title="Two Random Metrics",
            xname="x units",
        )
    }
)

Note that the number of x and y points must match exactly. You can supply one list of x values to match multiple lists of y values, or a separate list of x values for each list of y values.

See in the app

Model evaluation charts

These preset charts have built-in wandb.plot methods that make it quick and easy to log charts directly from your script and see the exact information you’re looking for in the UI.

• Precision-recall curves
• ROC curves
• Confusion matrix

wandb.plot.pr_curve()

Create a Precision-Recall curve in one line:

wandb.log({"pr": wandb.plot.pr_curve(ground_truth, predictions)})

You can log this whenever your code has access to:

• a model’s predicted scores (predictions) on a set of examples
• the corresponding ground truth labels (ground_truth) for those examples
• (optionally) a list of the labels/class names (labels=["cat", "dog", "bird"...] if label index 0 means cat, 1 = dog, 2 = bird, etc.)
• (optionally) a subset (still in list format) of the labels to visualize in the plot

See in the app

Run the code

wandb.plot.roc_curve()

Create an ROC curve in one line:

wandb.log({"roc": wandb.plot.roc_curve(ground_truth, predictions)})

You can log this whenever your code has access to:

• a model’s predicted scores (predictions) on a set of examples
• the corresponding ground truth labels (ground_truth) for those examples
• (optionally) a list of the labels/ class names (labels=["cat", "dog", "bird"...] if label index 0 means cat, 1 = dog, 2 = bird, etc.)
• (optionally) a subset (still in list format) of these labels to visualize on the plot

See in the app

Run the code

wandb.plot.confusion_matrix()

Create a multi-class confusion matrix in one line:

cm = wandb.plot.confusion_matrix(
    y_true=ground_truth, preds=predictions, class_names=class_names
)

wandb.log({"conf_mat": cm})

You can log this wherever your code has access to:

• a model’s predicted labels on a set of examples (preds) or the normalized probability scores (probs). The probabilities must have the shape (number of examples, number of classes). You can supply either probabilities or predictions but not both.
• the corresponding ground truth labels for those examples (y_true)
• a full list of the labels/class names as strings of class_names. Examples: class_names=["cat", "dog", "bird"] if index 0 is cat, 1 is dog, 2 is bird.

​See in the app​

​Run the code

Interactive custom charts

For full customization, tweak a built-in Custom Chart preset or create a new preset, then save the chart. Use the chart ID to log data to that custom preset directly from your script.

# Create a table with the columns to plot
table = wandb.Table(data=data, columns=["step", "height"])

# Map from the table's columns to the chart's fields
fields = {"x": "step", "value": "height"}

# Use the table to populate the new custom chart preset
# To use your own saved chart preset, change the vega_spec_name
# To edit the title, change the string_fields
my_custom_chart = wandb.plot_table(
    vega_spec_name="carey/new_chart",
    data_table=table,
    fields=fields,
    string_fields={"title": "Height Histogram"},
)

Run the code

Matplotlib and Plotly plots

Instead of using W&B Custom Charts with wandb.plot, you can log charts generated with matplotlib and Plotly.

import matplotlib.pyplot as plt

plt.plot([1, 2, 3, 4])
plt.ylabel("some interesting numbers")
wandb.log({"chart": plt})

Just pass a matplotlib plot or figure object to wandb.log(). By default we’ll convert the plot into a Plotly plot. If you’d rather log the plot as an image, you can pass the plot into wandb.Image. We also accept Plotly charts directly.

Log custom HTML to W&B Tables

W&B supports logging interactive charts from Plotly and Bokeh as HTML and adding them to Tables.

Log Plotly figures to Tables as HTML

You can log interactive Plotly charts to wandb Tables by converting them to HTML.

import wandb
import plotly.express as px

# Initialize a new run
run = wandb.init(project="log-plotly-fig-tables", name="plotly_html")

# Create a table
table = wandb.Table(columns=["plotly_figure"])

# Create path for Plotly figure
path_to_plotly_html = "./plotly_figure.html"

# Example Plotly figure
fig = px.scatter(x=[0, 1, 2, 3, 4], y=[0, 1, 4, 9, 16])

# Write Plotly figure to HTML
# Set auto_play to False prevents animated Plotly charts
# from playing in the table automatically
fig.write_html(path_to_plotly_html, auto_play=False)

# Add Plotly figure as HTML file into Table
table.add_data(wandb.Html(path_to_plotly_html))

# Log Table
run.log({"test_table": table})
wandb.finish()

Log Bokeh figures to Tables as HTML

You can log interactive Bokeh charts to wandb Tables by converting them to HTML.

from scipy.signal import spectrogram
import holoviews as hv
import panel as pn
from scipy.io import wavfile
import numpy as np
from bokeh.resources import INLINE

hv.extension("bokeh", logo=False)
import wandb


def save_audio_with_bokeh_plot_to_html(audio_path, html_file_name):
    sr, wav_data = wavfile.read(audio_path)
    duration = len(wav_data) / sr
    f, t, sxx = spectrogram(wav_data, sr)
    spec_gram = hv.Image((t, f, np.log10(sxx)), ["Time (s)", "Frequency (hz)"]).opts(
        width=500, height=150, labelled=[]
    )
    audio = pn.pane.Audio(wav_data, sample_rate=sr, name="Audio", throttle=500)
    slider = pn.widgets.FloatSlider(end=duration, visible=False)
    line = hv.VLine(0).opts(color="white")
    slider.jslink(audio, value="time", bidirectional=True)
    slider.jslink(line, value="glyph.location")
    combined = pn.Row(audio, spec_gram * line, slider).save(html_file_name)


html_file_name = "audio_with_plot.html"
audio_path = "hello.wav"
save_audio_with_bokeh_plot_to_html(audio_path, html_file_name)

wandb_html = wandb.Html(html_file_name)
run = wandb.init(project="audio_test")
my_table = wandb.Table(columns=["audio_with_plot"], data=[[wandb_html], [wandb_html]])
run.log({"audio_table": my_table})
run.finish()

1.6.2 - Customize log axes

Use define_metric to set a custom x axis.Custom x-axes are useful in contexts where you need to log to different time steps in the past during training, asynchronously. For example, this can be useful in RL where you may track the per-episode reward and a per-step reward.

Try define_metric live in Google Colab →

Customize axes

By default, all metrics are logged against the same x-axis, which is the W&B internal step. Sometimes, you might want to log to a previous step, or use a different x-axis.

Here’s an example of setting a custom x-axis metric, instead of the default step.

import wandb

wandb.init()
# define our custom x axis metric
wandb.define_metric("custom_step")
# define which metrics will be plotted against it
wandb.define_metric("validation_loss", step_metric="custom_step")

for i in range(10):
    log_dict = {
        "train_loss": 1 / (i + 1),
        "custom_step": i**2,
        "validation_loss": 1 / (i + 1),
    }
    wandb.log(log_dict)

The x-axis can be set using globs as well. Currently, only globs that have string prefixes are available. The following example will plot all logged metrics with the prefix "train/" to the x-axis "train/step":

import wandb

wandb.init()
# define our custom x axis metric
wandb.define_metric("train/step")
# set all other train/ metrics to use this step
wandb.define_metric("train/*", step_metric="train/step")

for i in range(10):
    log_dict = {
        "train/step": 2**i,  # exponential growth w/ internal W&B step
        "train/loss": 1 / (i + 1),  # x-axis is train/step
        "train/accuracy": 1 - (1 / (1 + i)),  # x-axis is train/step
        "val/loss": 1 / (1 + i),  # x-axis is internal wandb step
    }
    wandb.log(log_dict)

1.6.3 - Log distributed training experiments

In distributed training, models are trained using multiple GPUs in parallel. W&B supports two patterns to track distributed training experiments:

1. One process: Initialize W&B (wandb.init) and log experiments (wandb.log) from a single process. This is a common solution for logging distributed training experiments with the PyTorch Distributed Data Parallel (DDP) Class. In some cases, users funnel data over from other processes using a multiprocessing queue (or another communication primitive) to the main logging process.
2. Many processes: Initialize W&B (wandb.init) and log experiments (wandb.log) in every process. Each process is effectively a separate experiment. Use the group parameter when you initialize W&B (wandb.init(group='group-name')) to define a shared experiment and group the logged values together in the W&B App UI.

The proceeding examples demonstrate how to track metrics with W&B using PyTorch DDP on two GPUs on a single machine. PyTorch DDP (DistributedDataParallel intorch.nn) is a popular library for distributed training. The basic principles apply to any distributed training setup, but the details of implementation may differ.

Method 1: One process

In this method we track only a rank 0 process. To implement this method, initialize W&B (wandb.init), commence a W&B Run, and log metrics (wandb.log) within the rank 0 process. This method is simple and robust, however, this method does not log model metrics from other processes (for example, loss values or inputs from their batches). System metrics, such as usage and memory, are still logged for all GPUs since that information is available to all processes.

Within our sample Python script (log-ddp.py), we check to see if the rank is 0. To do so, we first launch multiple processes with torch.distributed.launch. Next, we check the rank with the --local_rank command line argument. If the rank is set to 0, we set up wandb logging conditionally in the train() function. Within our Python script, we use the following check:

if __name__ == "__main__":
    # Get args
    args = parse_args()

    if args.local_rank == 0:  # only on main process
        # Initialize wandb run
        run = wandb.init(
            entity=args.entity,
            project=args.project,
        )
        # Train model with DDP
        train(args, run)
    else:
        train(args)

Explore the W&B App UI to view an example dashboard of metrics tracked from a single process. The dashboard displays system metrics such as temperature and utilization, that were tracked for both GPUs.

However, the loss values as a function epoch and batch size were only logged from a single GPU.

Method 2: Many processes

In this method, we track each process in the job, calling wandb.init() and wandb.log() from each process separately. We suggest you call wandb.finish() at the end of training, to mark that the run has completed so that all processes exit properly.

This method makes more information accessible for logging. However, note that multiple W&B Runs are reported in the W&B App UI. It might be difficult to keep track of W&B Runs across multiple experiments. To mitigate this, provide a value to the group parameter when you initialize W&B to keep track of which W&B Run belongs to a given experiment. For more information about how to keep track of training and evaluation W&B Runs in experiments, see Group Runs.

The following Python code snippet demonstrates how to set the group parameter when you initialize W&B:

if __name__ == "__main__":
    # Get args
    args = parse_args()
    # Initialize run
    run = wandb.init(
        entity=args.entity,
        project=args.project,
        group="DDP",  # all runs for the experiment in one group
    )
    # Train model with DDP
    train(args, run)

Explore the W&B App UI to view an example dashboard of metrics tracked from multiple processes. Note that there are two W&B Runs grouped together in the left sidebar. Click on a group to view the dedicated group page for the experiment. The dedicated group page displays metrics from each process separately.

The preceding image demonstrates the W&B App UI dashboard. On the sidebar we see two experiments. One labeled ’null’ and a second (bound by a yellow box) called ‘DPP’. If you expand the group (select the Group dropdown) you will see the W&B Runs that are associated to that experiment.

Use W&B Service to avoid common distributed training issues

There are two common issues you might encounter when using W&B and distributed training:

1. Hanging at the beginning of training - A wandb process can hang if the wandb multiprocessing interferes with the multiprocessing from distributed training.
2. Hanging at the end of training - A training job might hang if the wandb process does not know when it needs to exit. Call the wandb.finish() API at the end of your Python script to tell W&B that the Run finished. The wandb.finish() API will finish uploading data and will cause W&B to exit.

We recommend using the wandb service to improve the reliability of your distributed jobs. Both of the preceding training issues are commonly found in versions of the W&B SDK where wandb service is unavailable.

Enable W&B Service

Depending on your version of the W&B SDK, you might already have W&B Service enabled by default.

W&B SDK 0.13.0 and above

W&B Service is enabled by default for versions of the W&B SDK 0.13.0 and above.

W&B SDK 0.12.5 and above

Modify your Python script to enable W&B Service for W&B SDK version 0.12.5 and above. Use the wandb.require method and pass the string "service" within your main function:

if __name__ == "__main__":
    main()


def main():
    wandb.require("service")
    # rest-of-your-script-goes-here

For optimal experience we do recommend you upgrade to the latest version.

W&B SDK 0.12.4 and below

Set the WANDB_START_METHOD environment variable to "thread" to use multithreading instead if you use a W&B SDK version 0.12.4 and below.

Example use cases for multiprocessing

The following code snippets demonstrate common methods for advanced distributed use cases.

Spawn process

Use the wandb.setup()method in your main function if you initiate a W&B Run in a spawned process:

import multiprocessing as mp


def do_work(n):
    run = wandb.init(config=dict(n=n))
    run.log(dict(this=n * n))


def main():
    wandb.setup()
    pool = mp.Pool(processes=4)
    pool.map(do_work, range(4))


if __name__ == "__main__":
    main()

Share a W&B Run

Pass a W&B Run object as an argument to share W&B Runs between processes:

def do_work(run):
    run.log(dict(this=1))


def main():
    run = wandb.init()
    p = mp.Process(target=do_work, kwargs=dict(run=run))
    p.start()
    p.join()


if __name__ == "__main__":
    main()

1.6.4 - Log media and objects

We support images, video, audio, and more. Log rich media to explore your results and visually compare your runs, models, and datasets. Read on for examples and how-to guides.

Pre-requisites

In order to log media objects with the W&B SDK, you may need to install additional dependencies. You can install these dependencies by running the following command:

pip install wandb[media]

Images

Log images to track inputs, outputs, filter weights, activations, and more.

Images can be logged directly from NumPy arrays, as PIL images, or from the filesystem.

Each time you log images from a step, we save them to show in the UI. Expand the image panel, and use the step slider to look at images from different steps. This makes it easy to compare how a model’s output changes during training.

• Logging arrays as Images
• Logging PIL Images
• Logging Images from Files

images = wandb.Image(image_array, caption="Top: Output, Bottom: Input")

wandb.log({"examples": images})

images = [PIL.Image.fromarray(image) for image in image_array]

wandb.log({"examples": [wandb.Image(image) for image in images]})

im = PIL.fromarray(...)
rgb_im = im.convert("RGB")
rgb_im.save("myimage.jpg")

wandb.log({"example": wandb.Image("myimage.jpg")})

Image overlays

• Segmentation Masks
• Bounding Boxes

Log semantic segmentation masks and interact with them (altering opacity, viewing changes over time, and more) via the W&B UI.

To log an overlay, you’ll need to provide a dictionary with the following keys and values to the masks keyword argument of wandb.Image:

• one of two keys representing the image mask:
  • "mask_data": a 2D NumPy array containing an integer class label for each pixel
  • "path": (string) a path to a saved image mask file
• "class_labels": (optional) a dictionary mapping the integer class labels in the image mask to their readable class names

To log multiple masks, log a mask dictionary with multiple keys, as in the code snippet below.

See a live example

Sample code

mask_data = np.array([[1, 2, 2, ..., 2, 2, 1], ...])

class_labels = {1: "tree", 2: "car", 3: "road"}

mask_img = wandb.Image(
    image,
    masks={
        "predictions": {"mask_data": mask_data, "class_labels": class_labels},
        "ground_truth": {
            # ...
        },
        # ...
    },
)

Log bounding boxes with images, and use filters and toggles to dynamically visualize different sets of boxes in the UI.

See a live example

To log a bounding box, you’ll need to provide a dictionary with the following keys and values to the boxes keyword argument of wandb.Image:

• box_data: a list of dictionaries, one for each box. The box dictionary format is described below.
  • position: a dictionary representing the position and size of the box in one of two formats, as described below. Boxes need not all use the same format.
    • Option 1: {"minX", "maxX", "minY", "maxY"}. Provide a set of coordinates defining the upper and lower bounds of each box dimension.
    • Option 2: {"middle", "width", "height"}. Provide a set of coordinates specifying the middle coordinates as [x,y], and width and height as scalars.
  • class_id: an integer representing the class identity of the box. See class_labels key below.
  • scores: a dictionary of string labels and numeric values for scores. Can be used for filtering boxes in the UI.
  • domain: specify the units/format of the box coordinates. Set this to “pixel” if the box coordinates are expressed in pixel space, such as integers within the bounds of the image dimensions. By default, the domain is assumed to be a fraction/percentage of the image, expressed as a floating point number between 0 and 1.
  • box_caption: (optional) a string to be displayed as the label text on this box
• class_labels: (optional) A dictionary mapping class_ids to strings. By default we will generate class labels class_0, class_1, etc.

Check out this example:

class_id_to_label = {
    1: "car",
    2: "road",
    3: "building",
    # ...
}

img = wandb.Image(
    image,
    boxes={
        "predictions": {
            "box_data": [
                {
                    # one box expressed in the default relative/fractional domain
                    "position": {"minX": 0.1, "maxX": 0.2, "minY": 0.3, "maxY": 0.4},
                    "class_id": 2,
                    "box_caption": class_id_to_label[2],
                    "scores": {"acc": 0.1, "loss": 1.2},
                    # another box expressed in the pixel domain
                    # (for illustration purposes only, all boxes are likely
                    # to be in the same domain/format)
                    "position": {"middle": [150, 20], "width": 68, "height": 112},
                    "domain": "pixel",
                    "class_id": 3,
                    "box_caption": "a building",
                    "scores": {"acc": 0.5, "loss": 0.7},
                    # ...
                    # Log as many boxes an as needed
                }
            ],
            "class_labels": class_id_to_label,
        },
        # Log each meaningful group of boxes with a unique key name
        "ground_truth": {
            # ...
        },
    },
)

wandb.log({"driving_scene": img})

Image overlays in Tables

• Segmentation Masks
• Bounding Boxes

To log Segmentation Masks in tables, you will need to provide a wandb.Image object for each row in the table.

An example is provided in the Code snippet below:

table = wandb.Table(columns=["ID", "Image"])

for id, img, label in zip(ids, images, labels):
    mask_img = wandb.Image(
        img,
        masks={
            "prediction": {"mask_data": label, "class_labels": class_labels}
            # ...
        },
    )

    table.add_data(id, img)

wandb.log({"Table": table})

To log Images with Bounding Boxes in tables, you will need to provide a wandb.Image object for each row in the table.

An example is provided in the code snippet below:

table = wandb.Table(columns=["ID", "Image"])

for id, img, boxes in zip(ids, images, boxes_set):
    box_img = wandb.Image(
        img,
        boxes={
            "prediction": {
                "box_data": [
                    {
                        "position": {
                            "minX": box["minX"],
                            "minY": box["minY"],
                            "maxX": box["maxX"],
                            "maxY": box["maxY"],
                        },
                        "class_id": box["class_id"],
                        "box_caption": box["caption"],
                        "domain": "pixel",
                    }
                    for box in boxes
                ],
                "class_labels": class_labels,
            }
        },
    )

Histograms

• Basic Histogram Logging
• Flexible Histogram Logging
• Histograms in Summary

If a sequence of numbers, such as a list, array, or tensor, is provided as the first argument, we will construct the histogram automatically by calling np.histogram. All arrays/tensors are flattened. You can use the optional num_bins keyword argument to override the default of 64 bins. The maximum number of bins supported is 512.

In the UI, histograms are plotted with the training step on the x-axis, the metric value on the y-axis, and the count represented by color, to ease comparison of histograms logged throughout training. See the “Histograms in Summary” tab of this panel for details on logging one-off histograms.

wandb.log({"gradients": wandb.Histogram(grads)})

If you want more control, call np.histogram and pass the returned tuple to the np_histogram keyword argument.

np_hist_grads = np.histogram(grads, density=True, range=(0.0, 1.0))
wandb.log({"gradients": wandb.Histogram(np_hist_grads)})

wandb.run.summary.update(  # if only in summary, only visible on overview tab
    {"final_logits": wandb.Histogram(logits)}
)

Log files in the formats 'obj', 'gltf', 'glb', 'babylon', 'stl', 'pts.json', and we will render them in the UI when your run finishes.

wandb.log(
    {
        "generated_samples": [
            wandb.Object3D(open("sample.obj")),
            wandb.Object3D(open("sample.gltf")),
            wandb.Object3D(open("sample.glb")),
        ]
    }
)

See a live example

If histograms are in your summary they will appear on the Overview tab of the Run Page. If they are in your history, we plot a heatmap of bins over time on the Charts tab.

3D visualizations

point_cloud = np.array([[0, 0, 0, COLOR]])

wandb.log({"point_cloud": wandb.Object3D(point_cloud)})

point_list = [
    [
        2566.571924017235, # x
        746.7817289698219, # y
        -15.269245470863748,# z
        76.5, # red
        127.5, # green
        89.46617199365393 # blue
    ],
    [ 2566.592983606823, 746.6791987335685, -15.275803826279521, 76.5, 127.5, 89.45471117247024 ],
    [ 2566.616361739416, 746.4903185513501, -15.28628929674075, 76.5, 127.5, 89.41336375503832 ],
    [ 2561.706014951675, 744.5349468458361, -14.877496818222781, 76.5, 127.5, 82.21868245418283 ],
    [ 2561.5281847916694, 744.2546118233013, -14.867862032341005, 76.5, 127.5, 81.87824684536432 ],
    [ 2561.3693562897465, 744.1804761656741, -14.854129178142523, 76.5, 127.5, 81.64137897587152 ],
    [ 2561.6093071504515, 744.0287526628543, -14.882135189841177, 76.5, 127.5, 81.89871499537098 ],
    # ... and so on
]

run.log({"my_first_point_cloud": wandb.Object3D.from_point_cloud(
     points = point_list,
     boxes = [{
         "corners": [
                [ 2601.2765123137915, 767.5669506323393, -17.816764802288663 ],
                [ 2599.7259021588347, 769.0082337923552, -17.816764802288663 ],
                [ 2599.7259021588347, 769.0082337923552, -19.66876480228866 ],
                [ 2601.2765123137915, 767.5669506323393, -19.66876480228866 ],
                [ 2604.8684867834395, 771.4313904894723, -17.816764802288663 ],
                [ 2603.3178766284827, 772.8726736494882, -17.816764802288663 ],
                [ 2603.3178766284827, 772.8726736494882, -19.66876480228866 ],
                [ 2604.8684867834395, 771.4313904894723, -19.66876480228866 ]
        ],
         "color": [0, 0, 255], # color in RGB of the bounding box
         "label": "car", # string displayed on the bounding box
         "score": 0.6 # numeric displayed on the bounding box
     }],
     vectors = [
        {"start": [0, 0, 0], "end": [0.1, 0.2, 0.5], "color": [255, 0, 0]}, # color is optional
     ],
     point_cloud_type = "lidar/beta",
)})

run.log({"my_cloud_from_file": wandb.Object3D.from_file(
     "./my_point_cloud.pts.json"
)})

{
    "boxes": [
        {
            "color": [
                0,
                255,
                0
            ],
            "score": 0.35,
            "label": "My label",
            "corners": [
                [
                    2589.695869075582,
                    760.7400443552185,
                    -18.044831294622487
                ],
                [
                    2590.719039645323,
                    762.3871153874499,
                    -18.044831294622487
                ],
                [
                    2590.719039645323,
                    762.3871153874499,
                    -19.54083129462249
                ],
                [
                    2589.695869075582,
                    760.7400443552185,
                    -19.54083129462249
                ],
                [
                    2594.9666662674313,
                    757.4657929961453,
                    -18.044831294622487
                ],
                [
                    2595.9898368371723,
                    759.1128640283766,
                    -18.044831294622487
                ],
                [
                    2595.9898368371723,
                    759.1128640283766,
                    -19.54083129462249
                ],
                [
                    2594.9666662674313,
                    757.4657929961453,
                    -19.54083129462249
                ]
            ]
        }
    ],
    "points": [
        [
            2566.571924017235,
            746.7817289698219,
            -15.269245470863748,
            76.5,
            127.5,
            89.46617199365393
        ],
        [
            2566.592983606823,
            746.6791987335685,
            -15.275803826279521,
            76.5,
            127.5,
            89.45471117247024
        ],
        [
            2566.616361739416,
            746.4903185513501,
            -15.28628929674075,
            76.5,
            127.5,
            89.41336375503832
        ]
    ],
    "type": "lidar/beta"
}

run.log({"my_cloud_from_numpy_xyz": wandb.Object3D.from_numpy(
     np.array(  
        [
            [0.4, 1, 1.3], # x, y, z
            [1, 1, 1], 
            [1.2, 1, 1.2]
        ]
    )
)})

run.log({"my_cloud_from_numpy_cat": wandb.Object3D.from_numpy(
     np.array(  
        [
            [0.4, 1, 1.3, 1], # x, y, z, category 
            [1, 1, 1, 1], 
            [1.2, 1, 1.2, 12], 
            [1.2, 1, 1.3, 12], 
            [1.2, 1, 1.4, 12], 
            [1.2, 1, 1.5, 12], 
            [1.2, 1, 1.6, 11], 
            [1.2, 1, 1.7, 11], 
        ]
    )
)})

run.log({"my_cloud_from_numpy_rgb": wandb.Object3D.from_numpy(
     np.array(  
        [
            [0.4, 1, 1.3, 255, 0, 0], # x, y, z, r, g, b 
            [1, 1, 1, 0, 255, 0], 
            [1.2, 1, 1.3, 0, 255, 255],
            [1.2, 1, 1.4, 0, 255, 255],
            [1.2, 1, 1.5, 0, 0, 255],
            [1.2, 1, 1.1, 0, 0, 255],
            [1.2, 1, 0.9, 0, 0, 255],
        ]
    )
)})

wandb.log({"protein": wandb.Molecule("6lu7.pdb")})

resveratrol = rdkit.Chem.MolFromSmiles("Oc1ccc(cc1)C=Cc1cc(O)cc(c1)O")

wandb.log(
    {
        "resveratrol": wandb.Molecule.from_rdkit(resveratrol),
        "green fluorescent protein": wandb.Molecule.from_rdkit("2b3p.mol"),
        "acetaminophen": wandb.Molecule.from_smiles("CC(=O)Nc1ccc(O)cc1"),
    }
)

wandb.log({"example": wandb.Image(...)})
# Or multiple images
wandb.log({"example": [wandb.Image(...) for img in images]})

wandb.log({"example": wandb.Video("myvideo.mp4")})

molecule = rdkit.Chem.MolFromSmiles("CC(=O)O")
rdkit.Chem.AllChem.Compute2DCoords(molecule)
rdkit.Chem.AllChem.GenerateDepictionMatching2DStructure(molecule, molecule)
pil_image = rdkit.Chem.Draw.MolToImage(molecule, size=(300, 300))

wandb.log({"acetic_acid": wandb.Image(pil_image)})

wandb.log({"whale songs": wandb.Audio(np_array, caption="OooOoo", sample_rate=32)})

wandb.log({"video": wandb.Video(numpy_array_or_path_to_video, fps=4, format="gif")})

columns = ["Text", "Predicted Sentiment", "True Sentiment"]
# Method 1
data = [["I love my phone", "1", "1"], ["My phone sucks", "0", "-1"]]
table = wandb.Table(data=data, columns=columns)
wandb.log({"examples": table})

# Method 2
table = wandb.Table(columns=columns)
table.add_data("I love my phone", "1", "1")
table.add_data("My phone sucks", "0", "-1")
wandb.log({"examples": table})

table = wandb.Table(dataframe=my_dataframe)

wandb.log({"custom_file": wandb.Html(open("some.html"))})
wandb.log({"custom_string": wandb.Html('<a href="https://mysite">Link</a>')})

wandb.log({"custom_file": wandb.Html(open("some.html"), inject=False)})

1.6.5 - Log models

Log models

The following guide describes how to log models to a W&B run and interact with them.

Log a model to a run

Use the log_model to log a model artifact that contains content within a directory you specify. The log_model method also marks the resulting model artifact as an output of the W&B run.

You can track a model’s dependencies and the model’s associations if you mark the model as the input or output of a W&B run. View the lineage of the model within the W&B App UI. See the Explore and traverse artifact graphs page within the Artifacts chapter for more information.

Provide the path where your model files are saved to the path parameter. The path can be a local file, directory, or reference URI to an external bucket such as s3://bucket/path.

Ensure to replace values enclosed in <> with your own.

import wandb

# Initialize a W&B run
run = wandb.init(project="<your-project>", entity="<your-entity>")

# Log the model
run.log_model(path="<path-to-model>", name="<name>")

Optionally provide a name for the model artifact for the name parameter. If name is not specified, W&B will use the basename of the input path prepended with the run ID as the name.

See log_model in the API Reference guide for more information on possible parameters.

import os
import wandb
from tensorflow import keras
from tensorflow.keras import layers

config = {"optimizer": "adam", "loss": "categorical_crossentropy"}

# Initialize a W&B run
run = wandb.init(entity="charlie", project="mnist-experiments", config=config)

# Hyperparameters
loss = run.config["loss"]
optimizer = run.config["optimizer"]
metrics = ["accuracy"]
num_classes = 10
input_shape = (28, 28, 1)

# Training algorithm
model = keras.Sequential(
    [
        layers.Input(shape=input_shape),
        layers.Conv2D(32, kernel_size=(3, 3), activation="relu"),
        layers.MaxPooling2D(pool_size=(2, 2)),
        layers.Conv2D(64, kernel_size=(3, 3), activation="relu"),
        layers.MaxPooling2D(pool_size=(2, 2)),
        layers.Flatten(),
        layers.Dropout(0.5),
        layers.Dense(num_classes, activation="softmax"),
    ]
)

# Configure the model for training
model.compile(loss=loss, optimizer=optimizer, metrics=metrics)

# Save model
model_filename = "model.h5"
local_filepath = "./"
full_path = os.path.join(local_filepath, model_filename)
model.save(filepath=full_path)

# Log the model to the W&B run
run.log_model(path=full_path, name="MNIST")
run.finish()

View run different-surf-5 at: https://wandb.ai/charlie/mnist-experiments/runs/wlby6fuw
Synced 5 W&B file(s), 0 media file(s), 1 artifact file(s) and 0 other file(s)
Find logs at: ./wandb/run-20231206_103511-wlby6fuw/logs

Download and use a logged model

Use the use_model function to access and download models files previously logged to a W&B run.

Provide the name of the model artifact where the model files you are want to retrieve are stored. The name you provide must match the name of an existing logged model artifact.

If you did not define name when originally logged the files with log_model, the default name assigned is the basename of the input path, prepended with the run ID.

Ensure to replace other the values enclosed in <> with your own:

import wandb

# Initialize a run
run = wandb.init(project="<your-project>", entity="<your-entity>")

# Access and download model. Returns path to downloaded artifact
downloaded_model_path = run.use_model(name="<your-model-name>")

The use_model function returns the path of downloaded model files. Keep track of this path if you want to link this model later. In the preceding code snippet, the returned path is stored in a variable called downloaded_model_path.

import wandb

entity = "luka"
project = "NLP_Experiments"
alias = "latest"  # semantic nickname or identifier for the model version
model_artifact_name = "fine-tuned-model"

# Initialize a run
run = wandb.init(project=project, entity=entity)
# Access and download model. Returns path to downloaded artifact
downloaded_model_path = run.use_model(name = f"{model_artifact_name}:{alias}") 

See use_model in the API Reference guide for more information on possible parameters and return type.

Log and link a model to the W&B Model Registry

Use the link_model method to log model files to a W&B run and link it to the W&B Model Registry. If no registered model exists, W&B will create a new one for you with the name you provide for the registered_model_name parameter.

A Registered Model is a collection or folder of linked model versions in the Model Registry. Registered models typically represent candidate models for a single modeling use case or task.

The proceeding code snippet shows how to link a model with the link_model API. Ensure to replace other the values enclosed in <> with your own:

import wandb

run = wandb.init(entity="<your-entity>", project="<your-project>")
run.link_model(path="<path-to-model>", registered_model_name="<registered-model-name>")
run.finish()

See link_model in the API Reference guide for more information on optional parameters.

If the registered-model-name matches the name of a registered model that already exists within the Model Registry, the model will be linked to that registered model. If no such registered model exists, a new one will be created and the model will be the first one linked.

For example, suppose you have an existing registered model named “Fine-Tuned-Review-Autocompletion” in your Model Registry (see example here). And suppose that a few model versions are already linked to it: v0, v1, v2. If you call link_model with registered-model-name="Fine-Tuned-Review-Autocompletion", the new model will be linked to this existing registered model as v3. If no registered model with this name exists, a new one will be created and the new model will be linked as v0.

import wandb

path = "/local/dir/model.pt"
registered_model_name = "Fine-Tuned-Review-Autocompletion"

run = wandb.init(project="llm-evaluation", entity="noa")
run.link_model(path=path, registered_model_name=registered_model_name)
run.finish()

1.6.6 - Log summary metrics

In addition to values that change over time during training, it is often important to track a single value that summarizes a model or a preprocessing step. Log this information in a W&B Run’s summary dictionary. A Run’s summary dictionary can handle numpy arrays, PyTorch tensors or TensorFlow tensors. When a value is one of these types we persist the entire tensor in a binary file and store high level metrics in the summary object, such as min, mean, variance, percentiles, and more.

The last value logged with wandb.log is automatically set as the summary dictionary in a W&B Run. If a summary metric dictionary is modified, the previous value is lost.

The proceeding code snippet demonstrates how to provide a custom summary metric to W&B:

wandb.init(config=args)

best_accuracy = 0
for epoch in range(1, args.epochs + 1):
    test_loss, test_accuracy = test()
    if test_accuracy > best_accuracy:
        wandb.summary["best_accuracy"] = test_accuracy
        best_accuracy = test_accuracy

You can update the summary attribute of an existing W&B Run after training has completed. Use the W&B Public API to update the summary attribute:

api = wandb.Api()
run = api.run("username/project/run_id")
run.summary["tensor"] = np.random.random(1000)
run.summary.update()

Customize summary metrics

Custom summary metrics are useful for capturing model performance at the best step of training in your wandb.summary. For example, you might want to capture the maximum accuracy or the minimum loss value, instead of the final value.

By default, the summary uses the final value from history. To customize summary metrics, pass the summary argument in define_metric. It accepts the following values:

• "min"
• "max"
• "mean"
• "best"
• "last"
• "none"

You can use "best" only when you also set the optional objective argument to "minimize" or "maximize".

The following example adds the min and max values of loss and accuracy to the summary:

import wandb
import random

random.seed(1)
wandb.init()

# Min and max summary values for loss
wandb.define_metric("loss", summary="min")
wandb.define_metric("loss", summary="max")

# Min and max summary values for accuracy
wandb.define_metric("acc", summary="min")
wandb.define_metric("acc", summary="max")

for i in range(10):
    log_dict = {
        "loss": random.uniform(0, 1 / (i + 1)),
        "acc": random.uniform(1 / (i + 1), 1),
    }
    wandb.log(log_dict)

View summary metrics

View summary values in a run’s Overview page or the project’s runs table.

• Run Overview
• Run Table
• W&B Public API

1. Navigate to the W&B App.
2. Select the Workspace tab.
3. From the list of runs, click the name of the run that logged the summary values.
4. Select the Overview tab.
5. View the summary values in the Summary section.

1. Navigate to the W&B App.
2. Select the Runs tab.
3. Within the runs table, you can view the summary values within the columns based on the name of the summary value.

You can use the W&B Public API to fetch the summary values of a run.

The following code example demonstrates one way to retrieve the summary values logged to a specific run using the W&B Public API and pandas:

import wandb
import pandas

entity = "<your-entity>"
project = "<your-project>"
run_name = "<your-run-name>" # Name of run with summary values

all_runs = []

for run in api.runs(f"{entity}/{project_name}"):
  print("Fetching details for run: ", run.id, run.name)
  run_data = {
            "id": run.id,
            "name": run.name,
            "url": run.url,
            "state": run.state,
            "tags": run.tags,
            "config": run.config,
            "created_at": run.created_at,
            "system_metrics": run.system_metrics,
            "summary": run.summary,
            "project": run.project,
            "entity": run.entity,
            "user": run.user,
            "path": run.path,
            "notes": run.notes,
            "read_only": run.read_only,
            "history_keys": run.history_keys,
            "metadata": run.metadata,
        }
  all_runs.append(run_data)
  
# Convert to DataFrame  
df = pd.DataFrame(all_runs)

# Get row based on the column name (run) and convert to dictionary
df[df['name']==run_name].summary.reset_index(drop=True).to_dict()

1.6.7 - Log tables

1. Create Tables
2. Add Data
3. Retrieve Data
4. Save Tables

Create tables

To define a Table, specify the columns you want to see for each row of data. Each row might be a single item in your training dataset, a particular step or epoch during training, a prediction made by your model on a test item, an object generated by your model, etc. Each column has a fixed type: numeric, text, boolean, image, video, audio, etc. You do not need to specify the type in advance. Give each column a name, and make sure to only pass data of that type into that column index. For a more detailed example, see this report.

Use the wandb.Table constructor in one of two ways:

1. List of Rows: Log named columns and rows of data. For example the proceeding code snippet generates a table with two rows and three columns:

wandb.Table(columns=["a", "b", "c"], data=[["1a", "1b", "1c"], ["2a", "2b", "2c"]])

2. Pandas DataFrame: Log a DataFrame using wandb.Table(dataframe=my_df). Column names will be extracted from the DataFrame.

From an existing array or dataframe

# assume a model has returned predictions on four images
# with the following fields available:
# - the image id
# - the image pixels, wrapped in a wandb.Image()
# - the model's predicted label
# - the ground truth label
my_data = [
    [0, wandb.Image("img_0.jpg"), 0, 0],
    [1, wandb.Image("img_1.jpg"), 8, 0],
    [2, wandb.Image("img_2.jpg"), 7, 1],
    [3, wandb.Image("img_3.jpg"), 1, 1],
]

# create a wandb.Table() with corresponding columns
columns = ["id", "image", "prediction", "truth"]
test_table = wandb.Table(data=my_data, columns=columns)

Add data

Tables are mutable. As your script executes you can add more data to your table, up to 200,000 rows. There are two ways to add data to a table:

1. Add a Row: table.add_data("3a", "3b", "3c"). Note that the new row is not represented as a list. If your row is in list format, use the star notation, * ,to expand the list to positional arguments: table.add_data(*my_row_list). The row must contain the same number of entries as there are columns in the table.
2. Add a Column: table.add_column(name="col_name", data=col_data). Note that the length of col_data must be equal to the table’s current number of rows. Here, col_data can be a list data, or a NumPy NDArray.

Adding data incrementally

This code sample shows how to create and populate a W&B table incrementally. You define the table with predefined columns, including confidence scores for all possible labels, and add data row by row during inference. You can also add data to tables incrementally when resuming runs.

# Define the columns for the table, including confidence scores for each label
columns = ["id", "image", "guess", "truth"]
for digit in range(10):  # Add confidence score columns for each digit (0-9)
    columns.append(f"score_{digit}")

# Initialize the table with the defined columns
test_table = wandb.Table(columns=columns)

# Iterate through the test dataset and add data to the table row by row
# Each row includes the image ID, image, predicted label, true label, and confidence scores
for img_id, img in enumerate(mnist_test_data):
    true_label = mnist_test_data_labels[img_id]  # Ground truth label
    guess_label = my_model.predict(img)  # Predicted label
    test_table.add_data(
        img_id, wandb.Image(img), guess_label, true_label
    )  # Add row data to the table

Adding data to resumed runs

You can incrementally update a W&B table in resumed runs by loading an existing table from an artifact, retrieving the last row of data, and adding the updated metrics. Then, reinitialize the table for compatibility and log the updated version back to W&B.

# Load the existing table from the artifact
best_checkpt_table = wandb.use_artifact(table_tag).get(table_name)

# Get the last row of data from the table for resuming
best_iter, best_metric_max, best_metric_min = best_checkpt_table.data[-1]

# Update the best metrics as needed

# Add the updated data to the table
best_checkpt_table.add_data(best_iter, best_metric_max, best_metric_min)

# Reinitialize the table with its updated data to ensure compatibility
best_checkpt_table = wandb.Table(
    columns=["col1", "col2", "col3"], data=best_checkpt_table.data
)

# Log the updated table to Weights & Biases
wandb.log({table_name: best_checkpt_table})

Retrieve data

Once data is in a Table, access it by column or by row:

1. Row Iterator: Users can use the row iterator of Table such as for ndx, row in table.iterrows(): ... to efficiently iterate over the data’s rows.
2. Get a Column: Users can retrieve a column of data using table.get_column("col_name") . As a convenience, users can pass convert_to="numpy" to convert the column to a NumPy NDArray of primitives. This is useful if your column contains media types such as wandb.Image so that you can access the underlying data directly.

Save tables

After you generate a table of data in your script, for example a table of model predictions, save it to W&B to visualize the results live.

Log a table to a run

Use wandb.log() to save your table to the run, like so:

run = wandb.init()
my_table = wandb.Table(columns=["a", "b"], data=[["1a", "1b"], ["2a", "2b"]])
run.log({"table_key": my_table})