Automations

• 1: Create an automation
• 1.1: Create a Slack automation
• 1.2: Create a webhook automation
• 2: Automation events and scopes

This feature requires a Pro or Enterprise plan.

This page describes automations in W&B. Create an automation to trigger workflow steps, such as automated model testing and deployment, based on an event in W&B, such as when an artifact artifact version is created or when a run metric meets or changes by a threshold.

For example, an automation can notify a Slack channel when a new version is created, trigger an automated testing webhook when the production alias is added to an artifact, or start a validation job only when a run’s loss is within acceptable bounds.

Overview

An automation can start when a specific event occurs in a registry or project.

For an artifact in a Registry, an automation can start:

• When a new artifact version is linked to a collection. For example, trigger testing and validation workflows for new candidate models.
• When an alias is added to an artifact version. For example, trigger a deployment workflow when an alias is added to a model version.

For an artifact in a project, an automation can start:

• When a new version is added to an artifact. For example, start a training job when a new version of a dataset artifact is added to a given collection.
• When an alias is added to an artifact version. For example, trigger a PII redaction workflow when the alias “redaction” is added to a dataset artifact.
• When a metric for a run meets or exceeds a configured threshold.
• When a metric for a run changes by a configured threshold.

This diagram shows the relationship between automation events and actions.

For more details, refer to Automation events and scopes.

To create an automation, you:

1. If required, configure secrets for sensitive strings the automation requires, such as access tokens, passwords, or sensitive configuration details. Secrets are defined in your Team Settings. Secrets are most commonly used in webhook automations to securely pass credentials or tokens to the webhook’s external service without exposing it in plain text or hard-coding it in the webhook’s payload.
2. Configure the webhook or Slack notification to authorize W&B to post to Slack or run the webhook on your behalf. A single automation action (webhook or Slack notification) can be used by multiple automations. These actions are defined in your Team Settings.
3. In the project or registry, create the automation:
   1. Define the event to watch for, such as when a new artifact version is added.
   2. Define the action to take when the event occurs (posting to a Slack channel or running a webhook). For a webhook, specify a secret to use for the access token and/or a secret to send with the payload, if required.

Limitations

Run metric automations are currently supported only in W&B Multi-tenant Cloud.

Next steps

• Create an automation.
• Learn about Automation events and scopes.
• Create a secret.

1 - Create an automation

This page gives an overview of creating and managing W&B automations. For more detailed instructions, refer to Create a Slack automation or Create a webhook automation.

Requirements

• A team admin can create and manage automations for the team’s projects, as well as components of their automations, such as webhooks, secrets, and Slack integrations. Refer to Team settings.
• To create a registry automation, you must have access to the registry. Refer to Configure Registry access.
• To create a Slack automation, you must have permission to post to the Slack instance and channel you select.

Create an automation

Create an automation from the project or registry’s Automations tab. At a high level, to create an automation, follow these steps:

1. If necessary, create a W&B secret for each sensitive string required by the automation, such as an access token, password, or SSH key. Secrets are defined in your Team Settings. Secrets are most commonly used in webhook automations.
2. Configure the webhook or Slack integration to authorize W&B to post to Slack or run the webhook on your behalf. A single webhook or Slack integration can be used by multiple automations. These actions are defined in your Team Settings.
3. In the project or registry, create the automation, which specifies the event to watch for and the action to take (such as posting to Slack or running a webhook). When you create a webhook automation, you configure the payload it sends.

Or, from a line plot in the workspace, you can quickly create a run metric automation for the metric it shows:

1. Hover over the panel, then click the bell icon at the top of the panel.

   

2. Configure the automation using the basic or advanced configuration controls. For example, apply a run filter to limit the scope of the automation, or configure an absolute threshold.

For details, refer to:

• Create a Slack automation
• Create a webhook automation

View and manage automations

View and manage automations from a project or registry’s Automations tab.

• To view an automation’s details, click its name.
• To edit an automation, click its action ... menu, then click Edit automation.
• To delete an automation, click its action ... menu, then click Delete automation.

Next steps

• Create a Slack automation.
• Create a webhook automation.
• Create a secret.

1.1 - Create a Slack automation

This page shows how to create a Slack automation. To create a webhook automation, refer to Create a webhook automation instead.

At a high level, to create a Slack automation, you take these steps:

1. Add a Slack integration, which authorizes W&B to post to the Slack instance and channel.
2. Create the automation, which defines the event to watch for and the channel to notify.

Add a Slack integration

A team admin can add a Slack integration to the team.

1. Log in to W&B and go to Team Settings.

2. In the Slack channel integrations section, click Connect Slack to add a new Slack instance. To add a channel for an existing Slack instance, click New integration.

   If necessary, sign in to Slack in your browser. When prompted, grant W&B permission to post to the Slack channel you select. Read the page, then click Search for a channel and begin typing the channel name. Select the channel from the list, then click Allow.

3. In Slack, go to the channel you selected. If you see a post like [Your Slack handle] added an integration to this channel: Weights & Biases, the integration is configured correctly.

Now you can create an automation that notifies the Slack channel you configured.

View and manage Slack integrations

A team admin can view and manage the team’s Slack instances and channels.

1. Log in to W&B and go to Team Settings.
2. View each Slack destination in the Slack channel integrations section.
3. Delete a destination by clicking its trash icon.

Create an automation

After you add a Slack integration, select Registry or Project, then follow these steps to create an automation that notifies the Slack channel.

• Registry
• Project

View and manage automations

• Registry
• Project

1.2 - Create a webhook automation

This page shows how to create a webhook automation. To create a Slack automation, refer to Create a Slack automation instead.

At a high level, to create a webhook automation, you take these steps:

1. If necessary, create a W&B secret for each sensitive string required by the automation, such as an access token, password, or SSH key. Secrets are defined in your Team Settings.
2. Create a webhook to define the endpoint and authorization details and grant the integration access to any secrets it needs.
3. Create the automation to define the event to watch for and the payload W&B will send. Grant the automation access to any secrets it needs for the payload.

Create a webhook

A team admin can add a webhook for the team.

1. Log in to W&B and go to Team Settings page.

2. In the Webhooks section, click New webhook.

3. Provide a name for the webhook.

4. Provide the endpoint URL for the webhook.

5. If the webhook requires a Bearer token, set Access token to the secret that contains it. When using the webhook automation, W&B sets the Authorization: Bearer HTTP header to the access token, and you can access the token in the ${ACCESS_TOKEN} payload variable. Learn more about the structure of the POST request W&B sends to the webhook service in Troubleshoot your webhook.

6. If the webhook requires a password or other sensitive string in its payload, set Secret to the secret that contains it. When you configure the automation that uses the webhook, you can access the secret as a payload variable by prefixing its name with $.

   If the webhook’s access token is stored in a secret, you must also complete the next step to specify the secret as the access token.

7. To verify that the W&B can connect and authenticate to the endpoint:

   1. Optionally, provide a payload to test. To refer to a secret the webhook has access to in the payload, prefix its name with $. This payload is only used for testing and is not saved. You configure an automation’s payload when you create the automation. See Troubleshoot your webhook to view where the secret and access token are specified in the POST request.
   2. Click Test. W&B attempts to connect to the webhook’s endpoint using the credentials you configured. If you provided a payload, W&B sends it.

   If the test does not succeed, verify the webhook’s configuration and try again. If necessary, refer to Troubleshoot your webhook.

Now you can create an automation that uses the webhook.

Create an automation

After you configure a webhook, select Registry or Project, then follow these steps to create an automation that triggers the webhook.

• Registry
• Project

View and manage automations

• Registry
• Project

Payload reference

Use these sections to construct your webhoook’s payload. For details about testing your webhook and its payload, refer to Troubleshoot your webhook.

Payload variables

This section describes the variables you can use to construct your webhook’s payload.

Payload examples

This section includes examples of webhook payloads for some common use cases. The examples demonstrate how to use payload variables.

• GitHub repository dispatch
• Microsoft Teams notification
• Slack notifications

on:
repository_dispatch:
  types: BUILD_AND_DEPLOY

{
  "event_type": "BUILD_AND_DEPLOY",
  "client_payload": 
  {
    "event_author": "${event_author}",
    "artifact_version": "${artifact_version}",
    "artifact_version_string": "${artifact_version_string}",
    "artifact_collection_name": "${artifact_collection_name}",
    "project_name": "${project_name}",
    "entity_name": "${entity_name}"
    }
}

${event_type} --> "LINK_ARTIFACT" or "ADD_ARTIFACT_ALIAS"
${event_author} --> "<wandb-user>"
${artifact_version} --> "wandb-artifact://_id/QXJ0aWZhY3Q6NTE3ODg5ODg3""
${artifact_version_string} --> "<entity>/model-registry/<registered_model_name>:<alias>"
${artifact_collection_name} --> "<registered_model_name>"
${project_name} --> "model-registry"
${entity_name} --> "<entity>"

{
"@type": "MessageCard",
"@context": "http://schema.org/extensions",
"summary": "New Notification",
"sections": [
  {
    "activityTitle": "Notification from WANDB",
    "text": "This is an example message sent via Teams webhook.",
    "facts": [
      {
        "name": "Author",
        "value": "${event_author}"
      },
      {
        "name": "Event Type",
        "value": "${event_type}"
      }
    ],
    "markdown": true
  }
]
}

{
    "text": "New alert from WANDB!",
"blocks": [
    {
            "type": "section",
        "text": {
            "type": "mrkdwn",
            "text": "Registry event: ${event_type}"
        }
    },
        {
            "type":"section",
            "text": {
            "type": "mrkdwn",
            "text": "New version: ${artifact_version_string}"
        }
        },
        {
        "type": "divider"
    },
        {
            "type": "section",
        "text": {
            "type": "mrkdwn",
            "text": "Author: ${event_author}"
        }
        }
    ]
}

Troubleshoot your webhook

Interactively troubleshoot your webhook with the W&B App UI or programmatically with a Bash script. You can troubleshoot a webhook when you create a new webhook or edit an existing webhook.

For details about the format W&B uses for the POST request, refer to the Bash script tab.

• W&B App UI
• Bash script

2 - Automation events and scopes

An automation can start when a specific event occurs within a project or registry. This diagram shows the relationship between automation events and actions.

This page describes the events that can trigger an automation within each scope. Learn more about automations in the Automations overview or Create an automation.

Registry

This section describes the scopes and events for an automation in a Registry.

1. Navigate to the Registry App at https://wandb.ai/registry/.
2. Click the name of a registry, then view and create automations in the Automations tab.

Learn more about creating automations.

Scopes

You can create a Registry automation at these scopes:

• Registry level: The automation watches for the event taking place on any collection within a specific registry, including collections added in the future.
• Collection level: A single collection in a specific registry.

Events

A Registry automation can watch for these events:

• Linking a new artifact to a collection: Test and validate new models or datasets when they are added to a registry.
• Adding a new alias to a version of an artifact: Trigger a specific step of your workflow when a new artifact version has a specific alias applied. For example, deploy a model when it has the production alias applied.

Project

This section describes the scopes and events for an automation in a project.

1. Navigate to your W&B project on the W&B App at https://wandb.ai/<team>/<project-name>.
2. View and create automations in the Automations tab.

Learn more about creating automations.

Scopes

You can create a project automation at these scopes:

• Project level: The automation watches for the event taking place on any collection in the project.
• Collection level: All collections in the project that match the filter you specify.

Artifact events

This section describes the events related to an artifact that can trigger an automation.

• Linking a new artifact: Test and validate new models or datasets automatically.
• Creating a new version of an artifact: Apply recurring actions to each version of an artifact. For example, start a training job when a new dataset artifact version is created.
• Adding a new alias to a version of an artifact: Trigger a specific step of your workflow when a new artifact version in a project or collection has a specific label or alias applied. For example, run a series of downstream processing steps when an artifact has the test-set-quality-check alias applied.

Run events

From the project’s Automations tab or directly from a line plot panel, you can create a run metric automation triggered by:

• A metric in a run’s history.
• A system metric such as cpu, which tracks the percentage of CPU utilization. W&B logs system metrics automatically every 15 seconds.

The notification can watch the metric for these events:

• Run metrics threshold met: Trigger a workflow when for a given metric, a single logged value or the average logged values meets the threshold you specify.
• Run metrics change threshold met: Trigger a workflow when the average logged values of a run change by the absolute or relative threshold you specify.

To set up a run metric automation, you configure how to compare the metric’s value with the threshold you specify. Your choices depend on the event type and on any filters you specify.

Threshold

For Run metrics threshold met events, you configure:

1. The number of logged values to average across (defaults to 5).
2. How to compare the values with the threshold.

For example, trigger an automation when accuracy exceeds .6.

Change threshold

For Run metrics change threshold met events, the automation uses two “windows” of values to check whether to start:

• The current window averages the 10 most recent values by default.
• The prior window averages the 50 most recent logged values prior to the current window.

The windows are consecutive and do not overlap.

To create the automation, you configure:

1. The current window (defaults to 10).
2. The prior window (defaults to 50).
3. Whether to evaluate the values as relative or absolute (defaults to Relative).
4. How to compare the values with the threshold:
   • Increases by at least
   • Decreases by at least
   • Increases or decreases by at least

Run filters

This section describes how the automation selects runs to evaluate.

• By default, any run in the project triggers the animation when the event occurs. To consider only specific runs, specify a run filter.
• Each run is considered individually and can potentially trigger the automation.
• Each run’s values are put into a separate window and compared to the threshold separately.
• In a 24 hour period, a particular automation can fire at most once per run.

Next steps

• Create a Slack automation
• Create a webhook automation