Advanced Configuration

Search…
How to configure the W&B Local Server installation
W&B Server starts ready-to-use on boot using wandb server start. However, several advanced configuration options are available using the /system-admin page on your server once it's up and running. You can email [email protected] to request a trial license to enable more users and teams.
The following is detailed information about the advanced configuration of a local server. When possible we suggest you use our existing Terraform to configure your instance.
Configuration as code
All configuration settings can be set via the UI however if you would like to manage these configuration options via code you can set the following environment variables:
Environment Variable
Description
LICENSE
Your wandb/local license
MYSQL
The MySQL connection string
BUCKET
The S3 / GCS bucket for storing data
BUCKET_QUEUE
The SQS / Google PubSub queue for object creation events
NOTIFICATIONS_QUEUE
The SQS queue on which to publish run events
AWS_REGION
The AWS Region where your bucket lives
HOST
The FQD of your instance, i.e. https://my.domain.net​
OIDC_ISSUER
A url to your Open ID Connect identity provider, i.e. https://cognito-idp.us-east-1.amazonaws.com/us-east-1_uiIFNdacd​
OIDC_CLIENT_ID
The Client ID of application in your identity provider
OIDC_AUTH_METHOD
Implicit (default) or pkce, see below for more context
SLACK_CLIENT_ID
The client ID of the Slack application you want to use for alerts
SLACK_SECRET
The secret of the Slack application you want to use for alerts
LOCAL_RESTORE
You can temporarily set this to true if you're unable to access your instance. Check the logs from the container for temporary credentials.
Host Configuration
To change the host and port that you want to deploy your wandb server instance then you can run the command
wandb server -e HOST=http://<HOST>:<PORT>
You can connect to this instance by then explicitly defining the HOST for our authentication method for wandb client. Here are various ways to perform this action.
1.wandb login --host=<HOST>:<PORT>
2.wandb.login(host="<HOST>:<PORT>")
3.export WANDB_BASE_URL=<HOST>:<PORT>
export WANDB_API_KEY=<API-KEY> 
SSO & Authentication
By default, a W&B Server runs with manual user management. Licensed versions of wandb/local also unlock SSO. Email  [email protected] to schedule a time with us to configure an Auth0 tenant for you with any Identity provider they support such as SAML, Ping Federate, Active Directory, etc. 
If you already use Auth0 or have an Open ID Connect compatible server, you can follow the instructions below.
Open ID Connect
wandb/local uses Open ID Connect for authentication. When creating an application client in your IPD you should choose Web Application or Public Client. For example, if you're using AWS Cognito as an identity provider you would choose Public Client:
Because we're only using OIDC for authentication and not authorization, public clients simplify setup
To configure an application client in your identity provider you'll need to provide an allowed callback url:
Add the following allowed Callback URL http(s)://YOUR-W&B-HOST/oidc/callback
If your IDP supports universal logout, set Logout URL to http(s)://YOUR-W&B-HOST
For example, in AWS Cognito if your application was running at https://wandb.mycompany.com:
If your instance is accessible from multiple hosts, be sure to include all of them here.
wandb/local will use the "implicit" grant with the "form_post" response type by default. You can also configure wandb/local to perform an "authorization_code" grant using the PKCE Code Exchange flow. We request the following scopes for the grant: "openid", "profile", and "email". Your identity provider will need to allow these scopes. For example in AWS Cognito the application should look like:
openid, profile, and email are required
To tell wandb/local which grant to use you can select the Auth Method in the settings page or set the OIDC_AUTH_METHOD environment variable.
For AWS Cognito providers you must set the Auth Method to "pkce"
You'll need a Client ID and the url of your OIDC issuer. The OpenID discovery document must be available at $OIDC_ISSUER/.well-known/openid-configuration For example, when using AWS Cognito you can generate your issuer url by appending your User Pool ID to the Cognito IDP url from the User Pools > App Integration tab:
The issuer URL would be https://cognito-idp.us-east-1.amazonaws.com/us-east-1_uiIFNdacd
Do not use the "Cognito domain" for the IDP url. Cognito provides it's discovery document at https://cognito-idp.$REGION.amazonaws.com/$USER_POOL_ID
Once you have everything configured you can provide the Issuer, Client ID, and Auth method to wandb/local via /system-admin or the environment variables and SSO will be configured.
If you're unable to login to your instance after configuring SSO, you can restart the instance with the LOCAL_RESTORE=true environment variable set. This will output a temporary password to the containers logs and disable SSO. Once you've resolved any issues with SSO, you must remove that environment variable to enable SSO again.
File Storage
By default, a W&B Enterprise Server saves files to a local data disk with a capacity that you set when you provision your instance. To support limitless file storage, you may configure your server to use an external cloud file storage bucket with an S3-compatible API.
You should always specify the bucket you're using with the BUCKET environment variable. This removes the need for a persistent volume as all settings can then be persisted to your bucket.
Amazon Web Services
To use an AWS S3 bucket as the file storage backend for W&B, you'll need to create a bucket, along with an SQS queue configured to receive object creation notifications from that bucket. Your instance will need permissions to read from this queue.
Create an S3 Bucket and Bucket Notifications
Then, create an S3 bucket. Under the bucket properties page in the console, in the "Events" section of "Advanced Settings", click "Add notification", and configure all object creation events to be sent to the SQS Queue you configured earlier.
Enterprise file storage settings
Enable CORS access: your CORS configuration should look like the following:
1
<?xml version="1.0" encoding="UTF-8"?>
2
<CORSConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
3
<CORSRule>
4
    <AllowedOrigin>http://YOUR-W&B-SERVER-IP</AllowedOrigin>
5
    <AllowedMethod>GET</AllowedMethod>
6
    <AllowedMethod>PUT</AllowedMethod>
7
    <AllowedHeader>*</AllowedHeader>
8
</CORSRule>
9
</CORSConfiguration>
Copied!
Create an SQS Queue
First, create an SQS Standard Queue. Add a permission for all principals for the SendMessage, ReceiveMessage, ChangeMessageVisibility, DeleteMessage, and GetQueueUrl actions. If you'd like you can further lock this down using an advanced policy document. For instance, the policy for accessing SQS with a statement is as follows:
1
{
2
    "Version" : "2012-10-17",
3
    "Statement" : [
4
      {
5
        "Effect" : "Allow",
6
        "Principal" : "*",
7
        "Action" : ["sqs:SendMessage"],
8
        "Resource" : "<sqs-queue-arn>",
9
        "Condition" : {
10
          "ArnEquals" : { "aws:SourceArn" : "<s3-bucket-arn>" }
11
        }
12
      }
13
    ]
14
}
Copied!
Grant Permissions to Node Running W&B
The node on which W&B Server is running must be configured to permit access to S3 and SQS. Depending on the type of server deployment you've opted for, you may need to add the following policy statements to your node role:
1
{
2
   "Statement":[
3
      {
4
         "Sid":"",
5
         "Effect":"Allow",
6
         "Action":"s3:*",
7
         "Resource":"arn:aws:s3:::<WANDB_BUCKET>"
8
      },
9
      {
10
         "Sid":"",
11
         "Effect":"Allow",
12
         "Action":[
13
            "sqs:*"
14
         ],
15
         "Resource":"arn:aws:sqs:<REGION>:<ACCOUNT>:<WANDB_QUEUE>"
16
      }
17
   ]
18
}
Copied!
Configure W&B Server
Finally, navigate to the W&B settings page at http(s)://YOUR-W&B-SERVER-HOST/system-admin. Enable the "Use an external file storage backend" option, and fill in the s3 bucket, region, and SQS queue in the following format:
File Storage Bucket: s3://<bucket-name>
File Storage Region (AWS only): <region>
Notification Subscription: sqs://<queue-name>
Press "Update settings" to apply the new settings.
Google Cloud Platform
To use a GCP Storage bucket as a file storage backend for W&B, you'll need to create a bucket, along with a PubSub topic and subscription configured to receive object creation messages from that bucket.
Create PubSub Topic and Subscription
Navigate to Pub/Sub > Topics in the GCP Console, and click "Create topic". Choose a name and create a topic.
Then click "Create subscription" in the subscriptions table at the bottom of the page. Choose a name, and make sure Delivery Type is set to "Pull". Click "Create".
Make sure the service account or account that your instance is running as has access to this subscription.
Create Storage Bucket
Navigate to Storage > Browser in the GCP Console, and click "Create bucket". Make sure to choose "Standard" storage class.
Make sure the service account or account that your instance is running as has access to this bucket.
Create PubSub Notification
Creating a notification stream from the Storage Bucket to the PubSub Topic can unfortunately only be done in the console. Make sure you have gsutil installed, and logged into the correct GCP Project, then run the following:
1
gcloud pubsub topics list  # list names of topics for reference
2
gsutil ls                  # list names of buckets for reference
3
​
4
# create bucket notification
5
gsutil notification create -t <TOPIC-NAME> -f json gs://<BUCKET-NAME>
Copied!
​Further reference is available on the Cloud Storage website.​
Add Signing Permissions
To create signed file URLs, your W&B Server also needs the iam.serviceAccounts.signBlob permission in GCP. You can add it by adding the Service Account Token Creator role to the service account or IAM member that your instance is running as.
Grant Permissions to Node Running W&B Server
The node on which W&B Server is running must be configured to permit access to S3 and SQS. Depending on the type of server deployment you've opted for, you may need to add the following policy statements to your node role:
1
{
2
   "Statement":[
3
      {
4
         "Sid":"",
5
         "Effect":"Allow",
6
         "Action":"s3:*",
7
         "Resource":"arn:aws:s3:::<WANDB_BUCKET>"
8
      },
9
      {
10
         "Sid":"",
11
         "Effect":"Allow",
12
         "Action":[
13
            "sqs:*"
14
         ],
15
         "Resource":"arn:aws:sqs:<REGION>:<ACCOUNT>:<WANDB_QUEUE>"
16
      }
17
   ]
18
}
Copied!
Configure W&B Server
Finally, navigate to the W&B settings page at http(s)://YOUR-W&B-SERVER-HOST/system-admin. Enable the "Use an external file storage backend" option, and fill in the s3 bucket, region, and SQS queue in the following format:
File Storage Bucket: gs://<bucket-name>
File Storage Region: blank
Notification Subscription: pubsub:/<project-name>/<topic-name>/<subscription-name>
Press "update settings" to apply the new settings.
Azure
To use an Azure blob container as the file storage for W&B, you'll need to create a storage account (if you don't already have one you want to use), create a blob container and a queue within that storage account, and then create an event subscription that sends "blob created" notifications to the queue from the blob container.
Create a Storage Account
If you have a storage account you want to use already, you can skip this step.
Navigate to Storage Accounts > Add in the Azure portal. Select an Azure subscription, and select any resource group or create a new one. Enter a name for your storage account.
Azure storage account setup
Click Review and Create, and then, on the summary screen, click Create:
Azure storage account details review
Creating the blob container
Go to Storage Accounts in the Azure portal, and click on your new storage account. In the storage account dashboard, click on Blob service > Containers in the menu:
Create a new container, and set it to Private:
Go to Settings > CORS > Blob service, and enter the IP of your wandb server as an allowed origin, with allowed methods GET and PUT, and all headers allowed and exposed, then save your CORS settings.
Creating the Queue
Go to Queue service > Queues in your storage account, and create a new Queue:
Go to Events in your storage account, and create an event subscription:
Give the event subscription the Event Schema "Event Grid Schema", filter to only the "Blob Created" event type, set the Endpoint Type to Storage Queues, and then select the storage account/queue as the endpoint.
In the Filters tab, enable subject filtering for subjects beginning with /blobServices/default/containers/your-blob-container-name/blobs/
Configure W&B Server
Go to Settings > Access keys in your storage account, click "Show keys", and then copy either key1 > Key or key2 > Key. Set this key on your W&B server as the environment variable AZURE_STORAGE_KEY.
Finally, navigate to the W&B settings page at http(s)://YOUR-W&B-SERVER-HOST/system-admin. Enable the "Use an external file storage backend" option, and fill in the s3 bucket, region, and SQS queue in the following format:
File Storage Bucket: az://<storage-account-name>/<blob-container-name>
Notification Subscription: az://<storage-account-name>/<queue-name>
Press "Update settings" to apply the new settings.
Slack
In order to integrate your local W&B installation with Slack, you'll need to create a suitable Slack application.
Creating the Slack application
Visit https://api.slack.com/apps and select Create New App in the top right.
You can name it whatever you like, but what's important is to select the same Slack workspace as the one you intend to use for alerts.
Configuring the Slack application
Now that we have a Slack application ready, we need to authorize for use as an OAuth bot. Select OAuth & Permissions in the sidebar to the left.
Under Scopes, supply the bot with the incoming_webhook scope.
Finally, configure the Redirect URL to point to your W&B installation. You should use the same value as what you set Frontend Host to in your local system settings. You can specify multiple URLs if you have different DNS mappings to your instance.
Hit Save URLs once finished.
To further secure your Slack application and prevent abuse, you can specify an IP range under Restrict API Token Usage, whitelisting the IP or IP range of your W&B instance(s).
Register your Slack application with W&B
Navigate to the System Settings page of your W&B instance. Check the box to enable a custom Slack application:
You'll need to supply your Slack application's client ID and secret, which you can find in the Basic Information tab.
That's it! You can now verify that everything is working by setting up a Slack integration in the W&B app. Visit this page for more detailed information.
On Prem / Baremetal
FAQ
Last modified 5d ago
Copy link
Contents
Configuration as code
Google Cloud Platform
Azure
Slack