W&B recommends fully managed deployment options such as W&B Multi-tenant Cloud or W&B Dedicated Cloud deployment types. W&B fully managed services are simple and secure to use, with minimum to no configuration required.
Your IT/DevOps/MLOps team is responsible for provisioning your deployment, managing upgrades, and continuously maintaining your self managed W&B Server instance.
Deploy W&B Server within self managed cloud accounts
W&B recommends that you use official W&B Terraform scripts to deploy W&B Server into your AWS, GCP, or Azure cloud account.
See specific cloud provider documentation for more information on how to set up W&B Server in AWS, GCP or Azure.
Deploy W&B Server in on-prem infrastructure
You need to configure several infrastructure components in order to set up W&B Server in your on-prem infrastructure. Some of those components include include, but are not limited to:
(Strongly recommended) Kubernetes cluster
MySQL 8 database cluster
Amazon S3-compatible object storage
Redis cache cluster
See Install on on-prem infrastructure for more information on how to install W&B Server on your on-prem infrastructure. W&B can provide recommendations for the different components and provide guidance through the installation process.
Deploy W&B Server on a custom cloud platform
You can deploy W&B Server to a cloud platform that is not AWS, GCP, or Azure. Requirements for that are similar to that for deploying in on-prem infrastructure.
Obtain your W&B Server license
You need a W&B trial license to complete your configuration of the W&B server. Open the Deploy Manager to generate a free trial license.
If you do not already have a W&B account, create one to generate your free license.
If you need an enterprise license for W&B Server which includes support for important security & other enterprise-friendly capabilities, submit this form or reach out to your W&B team.
The URL redirects you to a Get a License for W&B Local form. Provide the following information:
Choose a deployment type from the Choose Platform step.
Select the owner of the license or add a new organization in the Basic Information step.
Provide a name for the instance in the Name of Instance field and optionally provide a description in the Description field in the Get a License step.
Select the Generate License Key button.
A page displays with an overview of your deployment along with the license associated with the instance.
1 - Tutorial: Run a W&B Server using Docker
Run Weights and Biases on your own machines using Docker
Follow this “Hello, world!” example to learn the general workflow to install W&B Server for Dedicated Cloud and Self Managed hosting options. By the end of this demo, you will know how to host W&B Server on your local machine using a Trial Mode W&B license.
For demonstration purposes, this demo uses a local development server on port 8080 (localhost:8080).
Trial Mode vs. Production Setup
In Trial Mode, you run the Docker container on a single machine. This setup is ideal for testing the product, but it is not scalable.
For production work, set up a scalable file system to avoid data loss. W&B strongly recommends that you:
Allocate extra space in advance,
Resize the file system proactively as you log more data
Configure external metadata and object stores for backup.
Prerequisites
Before you get started, ensure your local machine satisfies the following requirements:
Select a license owner from the Owner of license dropdown.
Click Next.
Provide a name for your license in the Name of Instance field.
(Optional) Provide a description about your license in the Description field.
Click the Generate License Key button.
After you click Generate License Key, W&B redirects you to a Deployment License page. Within the Deployment License page you can view information about your license instance such as the Deployment ID, the organization the license belongs to, and more.
View a specific license instance in one of two ways:
Navigate to the Deploy Manager UI and then click the name of the license instance.
Directly navigate to a specific license instance at https://deploy.wandb.ai/DeploymentID where DeploymentID is the unique ID assigned to your license instance.
4. Add trial license to your local host
Within the Deployment License page of your license instance, click the Copy License button.
Navigate to http://localhost:8080/system-admin/
Paste your license into to License field.
Click the Update settings button.
5. Check your browser is running the W&B App UI
Check that W&B is running on your local machine. Navigate to http://localhost:8080/home. You should see the W&B App UI in your browser.
6. Add programmatic access to your local W&B instance
Navigate to http://localhost:8080/authorize to obtain your API key.
Within your terminal, execute the following:
wandb login --host=http://localhost:8080/
If you are already logged into W&B with a different count, add the relogin flag:
W&B appends a localhost profile and your API key to your .netrc profile at /Users/username/.netrc for future automatic logins.
Add a volume to retain data
All metadata and files you log to W&B are temporarily stored in the https://deploy.wandb.ai/vol directory.
Mount a volume, or external storage, to your Docker container to retain files and metadata you store in your local W&B instance. W&B recommends that you store metadata in an external MySQL database and files in an external storage bucket such as Amazon S3.
Recall that your local W&B instance (created using a Trial W&B License), uses Docker to run W&B in your local browser. By default, data is not retained if a Docker container no longer exists. Data is lost when a Docker process dies if you do not mount a volume at https://deploy.wandb.ai/vol.
For more information on how to mount a volume and for information on how Docker manages data, see Manage data in Docker page in the Docker documentation.
Volume considerations
The underlying file store should be resizable.
W&B recommends that you set up alerts to inform you when you are close to reaching minimum storage thresholds so you can resize the underlying file system.
For enterprise trials, W&B recommends at least 100 GB free space in the volume for non-image/video/audio heavy workloads.
2 - Run W&B Server on Kubernetes
Deploy W&B Platform with Kubernetes Operator
W&B Kubernetes Operator
Use the W&B Kubernetes Operator to simplify deploying, administering, troubleshooting, and scaling your W&B Server deployments on Kubernetes. You can think of the operator as a smart assistant for your W&B instance.
The W&B Server architecture and design continuously evolves to expand AI developer tooling capabilities, and to provide appropriate primitives for high performance, better scalability, and easier administration. That evolution applies to the compute services, relevant storage and the connectivity between them. To help facilitate continuous updates and improvements across deployment types, W&B users a Kubernetes operator.
W&B uses the operator to deploy and manage Dedicated cloud instances on AWS, GCP and Azure public clouds.
For more information about Kubernetes operators, see Operator pattern in the Kubernetes documentation.
Reasons for the architecture shift
Historically, the W&B application was deployed as a single deployment and pod within a Kubernetes Cluster or a single Docker container. W&B has, and continues to recommend, to externalize the Database and Object Store. Externalizing the Database and Object store decouples the application’s state.
As the application grew, the need to evolve from a monolithic container to a distributed system (microservices) was apparent. This change facilitates backend logic handling and seamlessly introduces built-in Kubernetes infrastructure capabilities. Distributed systems also supports deploying new services essential for additional features that W&B relies on.
Before 2024, any Kubernetes-related change required manually updating the terraform-kubernetes-wandb Terraform module. Updating the Terraform module ensures compatibility across cloud providers, configuring necessary Terraform variables, and executing a Terraform apply for each backend or Kubernetes-level change.
This process was not scalable since W&B Support had to assist each customer with upgrading their Terraform module.
The solution was to implement an operator that connects to a central deploy.wandb.ai server to request the latest specification changes for a given release channel and apply them. Updates are received as long as the license is valid. Helm is used as both the deployment mechanism for the W&B operator and the means for the operator to handle all configuration templating of the W&B Kubernetes stack, Helm-ception.
How it works
You can install the operator with helm or from the source. See charts/operator for detailed instructions.
The installation process creates a deployment called controller-manager and uses a custom resource definition named weightsandbiases.apps.wandb.com (shortName: wandb), that takes a single spec and applies it to the cluster:
The controller-manager installs charts/operator-wandb based on the spec of the custom resource, release channel, and a user defined config. The configuration specification hierarchy enables maximum configuration flexibility at the user end and enables W&B to release new images, configurations, features, and Helm updates automatically.
Configuration specifications follow a hierarchical model where higher-level specifications override lower-level ones. Here’s how it works:
Release Channel Values: This base level configuration sets default values and configurations based on the release channel set by W&B for the deployment.
User Input Values: Users can override the default settings provided by the Release Channel Spec through the System Console.
Custom Resource Values: The highest level of specification, which comes from the user. Any values specified here override both the User Input and Release Channel specifications. For a detailed description of the configuration options, see Configuration Reference.
This hierarchical model ensures that configurations are flexible and customizable to meet varying needs while maintaining a manageable and systematic approach to upgrades and changes.
Requirements to use the W&B Kubernetes Operator
Satisfy the following requirements to deploy W&B with the W&B Kubernetes operator:
This section describes different ways to deploy the W&B Kubernetes operator.
The W&B Operator is the default and recommended installation method for W&B Server
Choose one of the following:
If you have provisioned all required external services and want to deploy W&B onto Kubernetes with Helm CLI, continue here.
If you prefer managing infrastructure and the W&B Server with Terraform, continue here.
If you want to utilize the W&B Cloud Terraform Modules, continue here.
Deploy W&B with Helm CLI
W&B provides a Helm Chart to deploy the W&B Kubernetes operator to a Kubernetes cluster. This approach allows you to deploy W&B Server with Helm CLI or a continuous delivery tool like ArgoCD. Make sure that the above mentioned requirements are in place.
Follow those steps to install the W&B Kubernetes Operator with Helm CLI:
Add the W&B Helm repository. The W&B Helm chart is available in the W&B Helm repository. Add the repo with the following commands:
Configure the W&B operator custom resource to trigger the W&B Server installation. Create an operator.yaml file to customize the W&B Operator deployment, specifying your custom configuration. See Configuration Reference for details.
Once you have the specification YAML created and filled with your values, run the following and the operator applies the configuration and install the W&B Server application based on your configuration.
kubectl apply -f operator.yaml
Wait until the deployment completes. This takes a few minutes.
To verify the installation using the web UI, create the first admin user account, then follow the verification steps outlined in Verify the installation.
Deploy W&B with Helm Terraform Module
This method allows for customized deployments tailored to specific requirements, leveraging Terraform’s infrastructure-as-code approach for consistency and repeatability. The official W&B Helm-based Terraform Module is located here.
The following code can be used as a starting point and includes all necessary configuration options for a production grade deployment.
Note that the configuration options are the same as described in Configuration Reference, but that the syntax has to follow the HashiCorp Configuration Language (HCL). The Terraform module creates the W&B custom resource definition (CRD).
To see how W&B&Biases themselves use the Helm Terraform module to deploy “Dedicated cloud” installations for customers, follow those links:
W&B provides a set of Terraform Modules for AWS, GCP and Azure. Those modules deploy entire infrastructures including Kubernetes clusters, load balancers, MySQL databases and so on as well as the W&B Server application. The W&B Kubernetes Operator is already pre-baked with those official W&B cloud-specific Terraform Modules with the following versions:
This integration ensures that W&B Kubernetes Operator is ready to use for your instance with minimal setup, providing a streamlined path to deploying and managing W&B Server in your cloud environment.
For a detailed description on how to use these modules, refer to this section to self-managed installations section in the docs.
Verify the installation
To verify the installation, W&B recommends using the W&B CLI. The verify command executes several tests that verify all components and configurations.
This step assumes that the first admin user account is created with the browser.
Follow these steps to verify the installation:
Install the W&B CLI:
pip install wandb
Log in to W&B:
wandb login --host=https://YOUR_DNS_DOMAIN
For example:
wandb login --host=https://wandb.company-name.com
Verify the installation:
wandb verify
A successful installation and fully working W&B deployment shows the following output:
Default host selected: https://wandb.company-name.com
Find detailed logs for this test at: /var/folders/pn/b3g3gnc11_sbsykqkm3tx5rh0000gp/T/tmpdtdjbxua/wandb
Checking if logged in...................................................✅
Checking signed URL upload..............................................✅
Checking ability to send large payloads through proxy...................✅
Checking requests to base url...........................................✅
Checking requests made over signed URLs.................................✅
Checking CORs configuration of the bucket...............................✅
Checking wandb package version is up to date............................✅
Checking logged metrics, saving and downloading a file..................✅
Checking artifact save and download workflows...........................✅
Access the W&B Management Console
The W&B Kubernetes operator comes with a management console. It is located at ${HOST_URI}/console, for example https://wandb.company-name.com/ console.
There are two ways to log in to the management console:
Open the W&B application in the browser and login. Log in to the W&B application with ${HOST_URI}/, for example https://wandb.company-name.com/
Access the console. Click on the icon in the top right corner and then click System console. Only users with admin privileges can see the System console entry.
W&B recommends you access the console using the following steps only if Option 1 does not work.
Open console application in browser. Open the above described URL, which redirects you to the login screen:
Retrieve the password from the Kubernetes secret that the installation generates:
kubectl get secret wandb-password -o jsonpath='{.data.password}' | base64 -d
Copy the password.
Login to the console. Paste the copied password, then click Login.
Update the W&B Kubernetes operator
This section describes how to update the W&B Kubernetes operator.
Updating the W&B Kubernetes operator does not update the W&B server application.
See the instructions here if you use a Helm chart that does not user the W&B Kubernetes operator before you follow the proceeding instructions to update the W&B operator.
Copy and paste the code snippets below into your terminal.
You no longer need to update W&B Server application if you use the W&B Kubernetes operator.
The operator automatically updates your W&B Server application when a new version of the software of W&B is released.
Migrate self-managed instances to W&B Operator
The proceeding section describe how to migrate from self-managing your own W&B Server installation to using the W&B Operator to do this for you. The migration process depends on how you installed W&B Server:
The W&B Operator is the default and recommended installation method for W&B Server. Reach out to Customer Support or your W&B team if you have any questions.
If you used the official W&B Cloud Terraform Modules, navigate to the appropriate documentation and follow the steps there:
Configure the new helm chart and trigger W&B application deployment. Apply the new configuration.
kubectl apply -f operator.yaml
The deployment takes a few minutes to complete.
Verify the installation. Make sure that everything works by following the steps in Verify the installation.
Remove to old installation. Uninstall the old helm chart or delete the resources that were created with manifests.
Migrate to Operator-based Terraform Helm chart
Follow these steps to migrate to the Operator-based Helm chart:
Prepare Terraform config. Replace the Terraform code from the old deployment in your Terraform config with the one that is described here. Set the same variables as before. Do not change .tfvars file if you have one.
Execute Terraform run. Execute terraform init, plan and apply
Verify the installation. Make sure that everything works by following the steps in Verify the installation.
Remove to old installation. Uninstall the old helm chart or delete the resources that were created with manifests.
Configuration Reference for W&B Server
This section describes the configuration options for W&B Server application. The application receives its configuration as custom resource definition named WeightsAndBiases. Some configuration options are exposed with the below configuration, some need to be set as environment variables.
The documentation has two lists of environment variables: basic and advanced. Only use environment variables if the configuration option that you need are not exposed using Helm Chart.
The W&B Server application configuration file for a production deployment requires the following contents. This YAML file defines the desired state of your W&B deployment, including the version, environment variables, external resources like databases, and other necessary settings.
The variable contains a connection string in this form:
s3://$ACCESS_KEY:$SECRET_KEY@$HOST/$BUCKET_NAME
You can optionally tell W&B to only connect over TLS if you configure a trusted SSL certificate for your object store. To do so, add the tls query parameter to the url:
This works only for a trusted SSL certificate. W&B does not support self-signed certificates.
MySQL
global:
mysql:
# Example values, replace with your owndatabase: wandb_localhost: 10.218.0.2name: wandb_localpassword: 8wtX6cJH...ZcUarK4zZGjpVport: 3306user: wandb
License
global:
# Example license, replace with your ownlicense: eyJhbGnUzaHgyQjQy...VFnPS_KETXg1hi
Ingress
To identify the ingress class, see this FAQ entry.
Without TLS
global:
# IMPORTANT: Ingress is on the same level in the YAML as ‘global’ (not a child)ingress:
class: ""
global:
# IMPORTANT: Ingress is on the same level in the YAML as ‘global’ (not a child)ingress:
class: ""annotations:
{}
# kubernetes.io/ingress.class: nginx# kubernetes.io/tls-acme: "true"tls:
- secretName: wandb-ingress-tlshosts:
- <HOST_URI>
In case of Nginx you might have to add the following annotation:
global:
ldap:
enabled: true# LDAP server address including "ldap://" or "ldaps://"host:
# LDAP search base to use for finding usersbaseDN:
# LDAP user to bind with (if not using anonymous bind)bindDN:
# Secret name and key with LDAP password to bind with (if not using anonymous bind)bindPW:
# LDAP attribute for email and group ID attribute names as comma separated string values.attributes:
# LDAP group allow listgroupAllowList:
# Enable LDAP TLStls: false
With TLS
The LDAP TLS cert configuration requires a config map pre-created with the certificate content.
To create the config map you can use the following command:
And use the config map in the YAML like the example below
global:
ldap:
enabled: true# LDAP server address including "ldap://" or "ldaps://"host:
# LDAP search base to use for finding usersbaseDN:
# LDAP user to bind with (if not using anonymous bind)bindDN:
# Secret name and key with LDAP password to bind with (if not using anonymous bind)bindPW:
# LDAP attribute for email and group ID attribute names as comma separated string values.attributes:
# LDAP group allow listgroupAllowList:
# Enable LDAP TLStls: true# ConfigMap name and key with CA certificate for LDAP servertlsCert:
configMap:
name: "ldap-tls-cert"key: "certificate.crt"
This section describes configuration options for W&B Kubernetes operator (wandb-controller-manager). The operator receives its configuration in the form of a YAML file.
By default, the W&B Kubernetes operator does not need a configuration file. Create a configuration file if required. For example, you might need a configuration file to specify custom certificate authorities, deploy in an air gap environment and so forth.
A custom certificate authority (customCACerts), is a list and can take many certificates. Those certificate authorities when added only apply to the W&B Kubernetes operator (wandb-controller-manager).
You can get the ingress class installed in your cluster by running
kubectl get ingressclass
2.1 - Kubernetes operator for air-gapped instances
Deploy W&B Platform with Kubernetes Operator (Airgapped)
Introduction
This guide provides step-by-step instructions to deploy the W&B Platform in air-gapped customer-managed environments.
Use an internal repository or registry to host the Helm charts and container images. Run all commands in a shell console with proper access to the Kubernetes cluster.
You could utilize similar commands in any continuous delivery tooling that you use to deploy Kubernetes applications.
Step 1: Prerequisites
Before starting, make sure your environment meets the following requirements:
Kubernetes version >= 1.28
Helm version >= 3
Access to an internal container registry with the required W&B images
Access to an internal Helm repository for W&B Helm charts
Step 2: Prepare internal container registry
Before proceeding with the deployment, you must ensure that the following container images are available in your internal container registry.
These images are critical for the successful deployment of W&B components.
The operator chart is used to deploy the W&B Operator, or the Controller Manager. While the platform chart is used to deploy the W&B Platform using the values configured in the custom resource definition (CRD).
Step 4: Set up Helm repository
Now, configure the Helm repository to pull the W&B Helm charts from your internal repository. Run the following commands to add and update the Helm repository:
The W&B Kubernetes operator, also known as the controller manager, is responsible for managing the W&B platform components. To install it in an air-gapped environment,
you must configure it to use your internal container registry.
To do so, you must override the default image settings to use your internal container registry and set the key airgapped: true to indicate the expected deployment type. Update the values.yaml file as shown below:
After installing the W&B Kubernetes operator, you must configure the Custom Resource Definitions (CRDs) to point to your internal Helm repository and container registry.
This configuration ensures that the Kubernetes operators uses your internal registry and repository are when it deploys the required components of the W&B platform.
To deploy the W&B platform, the Kubernetes Operator uses the operator-wandb chart from your internal repository and use the values from your CRD to configure the Helm chart.
Finally, after setting up the Kubernetes operator and the CRD, deploy the W&B platform using the following command:
kubectl apply -f wandb.yaml
FAQ
Refer to the below frequently asked questions (FAQs) and troubleshooting tips during the deployment process:
There is another ingress class. Can that class be used?
Yes, you can configure your ingress class by modifying the ingress settings in values.yaml.
The certificate bundle has more than one certificate. Would that work?
You must split the certificates into multiple entries in the customCACerts section of values.yaml.
How do you prevent the Kubernetes operator from applying unattended updates. Is that possible?
You can turn off auto-updates from the W&B console. Reach out to your W&B team for any questions on the supported versions. Also, note that W&B supports platform versions released in last 6 months. W&B recommends performing periodic upgrades.
Does the deployment work if the environment has no connection to public repositories?
As long as you have enabled the airgapped: true configuration, the Kubernetes operator does not attempt to reach public repositories. The Kubernetes operator attempts to use your internal resources.
3 - Install on public cloud
3.1 - Deploy W&B Platform on AWS
Hosting W&B Server on AWS.
W&B recommends fully managed deployment options such as W&B Multi-tenant Cloud or W&B Dedicated Cloud deployment types. W&B fully managed services are simple and secure to use, with minimum to no configuration required.
Before you start, W&B recommends that you choose one of the remote backends available for Terraform to store the State File.
The State File is the necessary resource to roll out upgrades or make changes in your deployment without recreating all components.
The Terraform Module deploys the following mandatory components:
Load Balancer
AWS Identity & Access Management (IAM)
AWS Key Management System (KMS)
Amazon Aurora MySQL
Amazon VPC
Amazon S3
Amazon Route53
Amazon Certificate Manager (ACM)
Amazon Elastic Load Balancing (ALB)
Amazon Secrets Manager
Other deployment options can also include the following optional components:
Elastic Cache for Redis
SQS
Pre-requisite permissions
The account that runs Terraform needs to be able to create all components described in the Introduction and permission to create IAM Policies and IAM Roles and assign roles to resources.
General steps
The steps on this topic are common for any deployment option covered by this documentation.
Ensure to define variables in your tvfars file before you deploy because the namespace variable is a string that prefixes all resources created by Terraform.
The combination of subdomain and domain will form the FQDN that W&B will be configured. In the example above, the W&B FQDN will be wandb-aws.wandb.ml and the DNS zone_id where the FQDN record will be created.
Both allowed_inbound_cidr and allowed_inbound_ipv6_cidr also require setting. In the module, this is a mandatory input. The proceeding example permits access from any source to the W&B installation.
Create the file versions.tf
This file will contain the Terraform and Terraform provider versions required to deploy W&B in AWS
provider "aws"{ region ="eu-central-1" default_tags { tags ={ GithubRepo ="terraform-aws-wandb" GithubOrg ="wandb" Enviroment ="Example" Example ="PublicDnsExternal"}}}
Optionally, but highly recommended, add the remote backend configuration mentioned at the beginning of this documentation.
Create the file variables.tf
For every option configured in the terraform.tfvars Terraform requires a correspondent variable declaration.
variable "namespace" {
type = string
description = "Name prefix used for resources"
}
variable "domain_name" {
type = string
description = "Domain name used to access instance."
}
variable "subdomain" {
type = string
default = null
description = "Subdomain for accessing the Weights & Biases UI."
}
variable "license" {
type = string
}
variable "zone_id" {
type = string
description = "Domain for creating the Weights & Biases subdomain on."
}
variable "allowed_inbound_cidr" {
description = "CIDRs allowed to access wandb-server."
nullable = false
type = list(string)
}
variable "allowed_inbound_ipv6_cidr" {
description = "CIDRs allowed to access wandb-server."
nullable = false
type = list(string)
}
Recommended deployment option
This is the most straightforward deployment option configuration that creates all Mandatory components and installs in the Kubernetes Cluster the latest version of W&B.
Create the main.tf
In the same directory where you created the files in the General Steps, create a file main.tf with the following content:
module "wandb_infra" {
source = "wandb/wandb/aws"
version = "~>2.0"
namespace = var.namespace
domain_name = var.domain_name
subdomain = var.subdomain
zone_id = var.zone_id
allowed_inbound_cidr = var.allowed_inbound_cidr
allowed_inbound_ipv6_cidr = var.allowed_inbound_ipv6_cidr
public_access = true
external_dns = true
kubernetes_public_access = true
kubernetes_public_access_cidrs = ["0.0.0.0/0"]
}
data "aws_eks_cluster" "app_cluster" {
name = module.wandb_infra.cluster_id
}
data "aws_eks_cluster_auth" "app_cluster" {
name = module.wandb_infra.cluster_id
}
provider "kubernetes" {
host = data.aws_eks_cluster.app_cluster.endpoint
cluster_ca_certificate = base64decode(data.aws_eks_cluster.app_cluster.certificate_authority.0.data)
token = data.aws_eks_cluster_auth.app_cluster.token
}
module "wandb_app" {
source = "wandb/wandb/kubernetes"
version = "~>1.0"
license = var.license
host = module.wandb_infra.url
bucket = "s3://${module.wandb_infra.bucket_name}"
bucket_aws_region = module.wandb_infra.bucket_region
bucket_queue = "internal://"
database_connection_string = "mysql://${module.wandb_infra.database_connection_string}"
# TF attempts to deploy while the work group is
# still spinning up if you do not wait
depends_on = [module.wandb_infra]
}
output "bucket_name" {
value = module.wandb_infra.bucket_name
}
output "url" {
value = module.wandb_infra.url
}
Another deployment option uses Redis to cache the SQL queries and speed up the application response when loading the metrics for the experiments.
You need to add the option create_elasticache_subnet = true to the same main.tf file described in the Recommended deployment section to enable the cache.
Deployment option 3 consists of enabling the external message broker. This is optional because the W&B brings embedded a broker. This option doesn’t bring a performance improvement.
The AWS resource that provides the message broker is the SQS, and to enable it, you will need to add the option use_internal_queue = false to the same main.tf described in the Recommended deployment section.
You can combine all three deployment options adding all configurations to the same file.
The Terraform Module provides several options that can be combined along with the standard options and the minimal configuration found in Deployment - Recommended
Manual configuration
To use an Amazon S3 bucket as a file storage backend for W&B, you will need to:
you’ll need to create a bucket, along with an SQS queue configured to receive object creation notifications from that bucket. Your instance will need permissions to read from this queue.
Create an S3 Bucket and Bucket Notifications
Follow the procedure bellow to create an Amazon S3 bucket and enable bucket notifications.
Navigate to Amazon S3 in the AWS Console.
Select Create bucket.
Within the Advanced settings, select Add notification within the Events section.
Configure all object creation events to be sent to the SQS Queue you configured earlier.
Enable CORS access. Your CORS configuration should look like the following:
Follow the procedure below to create an SQS Queue:
Navigate to Amazon SQS in the AWS Console.
Select Create queue.
From the Details section, select a Standard queue type.
Within the Access policy section, add permission to the following principals:
SendMessage
ReceiveMessage
ChangeMessageVisibility
DeleteMessage
GetQueueUrl
Optionally add an advanced access policy in the Access Policy section. For example, the policy for accessing Amazon SQS with a statement is as follows:
The node where W&B server is running must be configured to permit access to Amazon S3 and Amazon SQS. Depending on the type of server deployment you have opted for, you may need to add the following policy statements to your node role:
Navigate to the W&B settings page at http(s)://YOUR-W&B-SERVER-HOST/system-admin.
Enable the **Use an external file storage backend option
Provide information about your Amazon S3 bucket, region, and Amazon SQS queue in the following format:
File Storage Bucket: s3://<bucket-name>
File Storage Region (AWS only): <region>
Notification Subscription: sqs://<queue-name>
Select Update settings to apply the new settings.
Upgrade your W&B version
Follow the steps outlined here to update W&B:
Add wandb_version to your configuration in your wandb_app module. Provide the version of W&B you want to upgrade to. For example, the following line specifies W&B version 0.48.1:
Alternatively, you can add the wandb_version to the terraform.tfvars and create a variable with the same name and instead of using the literal value, use the var.wandb_version
This section details the steps required to upgrade from pre-operator to post-operator environments using the terraform-aws-wandb module.
The transition to a Kubernetes operator pattern is necessary for the W&B architecture. See this section for a detailed explanation for the architecture shift.
Before and after architecture
Previously, the W&B architecture used:
module"wandb_infra" {
source ="wandb/wandb/aws" version ="1.16.10" ...
}
to control the infrastructure:
and this module to deploy the W&B Server:
module"wandb_app" {
source ="wandb/wandb/kubernetes" version ="1.12.0"}
Post-transition, the architecture uses:
module"wandb_infra" {
source ="wandb/wandb/aws" version ="4.7.2" ...
}
to manage both the installation of infrastructure and the W&B Server to the Kubernetes cluster, thus eliminating the need for the module "wandb_app" in post-operator.tf.
This architectural shift enables additional features (like OpenTelemetry, Prometheus, HPAs, Kafka, and image updates) without requiring manual Terraform operations by SRE/Infrastructure teams.
To commence with a base installation of the W&B Pre-Operator, ensure that post-operator.tf has a .disabled file extension and pre-operator.tf is active (that does not have a .disabled extension). Those files can be found here.
Prerequisites
Before initiating the migration process, ensure the following prerequisites are met:
Egress: The deployment can’t be airgapped. It needs access to deploy.wandb.ai to get the latest spec for the Release Channel.
AWS Credentials: Proper AWS credentials configured to interact with your AWS resources.
Terraform Installed: The latest version of Terraform should be installed on your system.
Route53 Hosted Zone: An existing Route53 hosted zone corresponding to the domain under which the application will be served.
Pre-Operator Terraform Files: Ensure pre-operator.tf and associated variable files like pre-operator.tfvars are correctly set up.
Pre-Operator set up
Execute the following Terraform commands to initialize and apply the configuration for the Pre-Operator setup:
The pre-operator.tf configuration calls two modules:
module"wandb_infra" {
source ="wandb/wandb/aws" version ="1.16.10" ...
}
This module spins up the infrastructure.
module"wandb_app" {
source ="wandb/wandb/kubernetes" version ="1.12.0"}
This module deploys the application.
Post-Operator Setup
Make sure that pre-operator.tf has a .disabled extension, and post-operator.tf is active.
The post-operator.tfvars includes additional variables:
...# wandb_version = "0.51.2" is now managed via the Release Channel or set in the User Spec.# Required Operator Variables for Upgrade:size="small"enable_dummy_dns=trueenable_operator_alb=truecustom_domain_filter="sandbox-aws.wandb.ml"
Run the following commands to initialize and apply the Post-Operator configuration:
module"wandb_infra" {
source ="wandb/wandb/aws" version ="4.7.2" ...
}
Changes in the post-operator configuration:
Update Required Providers: Change required_providers.aws.version from 3.6 to 4.0 for provider compatibility.
DNS and Load Balancer Configuration: Integrate enable_dummy_dns and enable_operator_alb to manage DNS records and AWS Load Balancer setup through an Ingress.
License and Size Configuration: Transfer the license and size parameters directly to the wandb_infra module to match new operational requirements.
Custom Domain Handling: If necessary, use custom_domain_filter to troubleshoot DNS issues by checking the External DNS pod logs within the kube-system namespace.
Helm Provider Configuration: Enable and configure the Helm provider to manage Kubernetes resources effectively:
This comprehensive setup ensures a smooth transition from the Pre-Operator to the Post-Operator configuration, leveraging new efficiencies and capabilities enabled by the operator model.
3.2 - Deploy W&B Platform on GCP
Hosting W&B Server on GCP.
W&B recommends fully managed deployment options such as W&B Multi-tenant Cloud or W&B Dedicated Cloud deployment types. W&B fully managed services are simple and secure to use, with minimum to no configuration required.
If you’ve determined to self-managed W&B Server, W&B recommends using the W&B Server GCP Terraform Module to deploy the platform on GCP.
The module documentation is extensive and contains all available options that can be used.
Before you start, W&B recommends that you choose one of the remote backends available for Terraform to store the State File.
The State File is the necessary resource to roll out upgrades or make changes in your deployment without recreating all components.
The Terraform Module will deploy the following mandatory components:
VPC
Cloud SQL for MySQL
Cloud Storage Bucket
Google Kubernetes Engine
KMS Crypto Key
Load Balancer
Other deployment options can also include the following optional components:
Memory store for Redis
Pub/Sub messages system
Pre-requisite permissions
The account that will run the terraform need to have the role roles/owner in the GCP project used.
General steps
The steps on this topic are common for any deployment option covered by this documentation.
The variables defined here need to be decided before the deployment because. The namespace variable will be a string that will prefix all resources created by Terraform.
The combination of subdomain and domain will form the FQDN that W&B will be configured. In the example above, the W&B FQDN will be wandb-gcp.wandb.ml
Create the file variables.tf
For every option configured in the terraform.tfvars Terraform requires a correspondent variable declaration.
variable "project_id" {
type = string
description = "Project ID"
}
variable "region" {
type = string
description = "Google region"
}
variable "zone" {
type = string
description = "Google zone"
}
variable "namespace" {
type = string
description = "Namespace prefix used for resources"
}
variable "domain_name" {
type = string
description = "Domain name for accessing the Weights & Biases UI."
}
variable "subdomain" {
type = string
description = "Subdomain for access the Weights & Biases UI."
}
variable "license" {
type = string
description = "W&B License"
}
Deployment - Recommended (~20 mins)
This is the most straightforward deployment option configuration that will create all Mandatory components and install in the Kubernetes Cluster the latest version of W&B.
Create the main.tf
In the same directory where you created the files in the General Steps, create a file main.tf with the following content:
provider "google" {
project = var.project_id
region = var.region
zone = var.zone
}
provider "google-beta" {
project = var.project_id
region = var.region
zone = var.zone
}
data "google_client_config" "current" {}
provider "kubernetes" {
host = "https://${module.wandb.cluster_endpoint}"
cluster_ca_certificate = base64decode(module.wandb.cluster_ca_certificate)
token = data.google_client_config.current.access_token
}
# Spin up all required services
module "wandb" {
source = "wandb/wandb/google"
version = "~> 5.0"
namespace = var.namespace
license = var.license
domain_name = var.domain_name
subdomain = var.subdomain
}
# You'll want to update your DNS with the provisioned IP address
output "url" {
value = module.wandb.url
}
output "address" {
value = module.wandb.address
}
output "bucket_name" {
value = module.wandb.bucket_name
}
Deployment option 3 consists of enabling the external message broker. This is optional because the W&B brings embedded a broker. This option doesn’t bring a performance improvement.
The GCP resource that provides the message broker is the Pub/Sub, and to enable it, you will need to add the option use_internal_queue = false to the same main.tf specified in the recommended Deployment option section
You can combine all three deployment options adding all configurations to the same file.
The Terraform Module provides several options that can be combined along with the standard options and the minimal configuration found in Deployment - Recommended
Manual configuration
To use a GCP Storage bucket as a file storage backend for W&B, you will need to create a:
Your instance also needs the iam.serviceAccounts.signBlob permission in GCP to create signed file URLs. Add Service Account Token Creator role to the service account or IAM member that your instance is running as to enable permission.
Enable CORS access. This can only be done using the command line. First, create a JSON file with the following CORS configuration.
cors:
- maxAgeSeconds: 3600
method:
- GET
- PUT
origin:
- '<YOUR_W&B_SERVER_HOST>'
responseHeader:
- Content-Type
Note that the scheme, host, and port of the values for the origin must match exactly.
Make sure you have gcloud installed, and logged into the correct GCP Project.
Follow the procedure below in your command line to create a notification stream from the Storage Bucket to the Pub/Sub topic.
You must use the CLI to create a notification stream. Ensure you have gcloud installed.
Log into your GCP Project.
Run the following in your terminal:
gcloud pubsub topics list # list names of topics for referencegcloud storage ls # list names of buckets for reference# create bucket notificationgcloud storage buckets notifications create gs://<BUCKET_NAME> --topic=<TOPIC_NAME>
Finally, navigate to the W&B System Connections page at http(s)://YOUR-W&B-SERVER-HOST/console/settings/system.
Select the provider Google Cloud Storage (gcs),
Provide the name of the GCS bucket
Press Update settings to apply the new settings.
Upgrade W&B Server
Follow the steps outlined here to update W&B:
Add wandb_version to your configuration in your wandb_app module. Provide the version of W&B you want to upgrade to. For example, the following line specifies W&B version 0.48.1:
Alternatively, you can add the wandb_version to the terraform.tfvars and create a variable with the same name and instead of using the literal value, use the var.wandb_version
W&B recommends fully managed deployment options such as W&B Multi-tenant Cloud or W&B Dedicated Cloud deployment types. W&B fully managed services are simple and secure to use, with minimum to no configuration required.
If you’ve determined to self-managed W&B Server, W&B recommends using the W&B Server Azure Terraform Module to deploy the platform on Azure.
The module documentation is extensive and contains all available options that can be used. We will cover some deployment options in this document.
Before you start, we recommend you choose one of the remote backends available for Terraform to store the State File.
The State File is the necessary resource to roll out upgrades or make changes in your deployment without recreating all components.
The Terraform Module will deploy the following mandatory components:
Azure Resource Group
Azure Virtual Network (VPC)
Azure MySQL Fliexible Server
Azure Storage Account & Blob Storage
Azure Kubernetes Service
Azure Application Gateway
Other deployment options can also include the following optional components:
Azure Cache for Redis
Azure Event Grid
Pre-requisite permissions
The simplest way to get the AzureRM provider configured is via Azure CLI but the incase of automation using Azure Service Principal can also be useful.
Regardless the authentication method used, the account that will run the Terraform needs to be able to create all components described in the Introduction.
General steps
The steps on this topic are common for any deployment option covered by this documentation.
We recommend creating a Git repository with the code that will be used, but you can keep your files locally.
Create the terraform.tfvars file The tvfars file content can be customized according to the installation type, but the minimum recommended will look like the example below.
The variables defined here need to be decided before the deployment because. The namespace variable will be a string that will prefix all resources created by Terraform.
The combination of subdomain and domain will form the FQDN that W&B will be configured. In the example above, the W&B FQDN will be wandb-aws.wandb.ml and the DNS zone_id where the FQDN record will be created.
Create the file versions.tf This file will contain the Terraform and Terraform provider versions required to deploy W&B in AWS
Optionally, but highly recommended, you can add the remote backend configuration mentioned at the beginning of this documentation.
Create the filevariables.tf. For every option configured in the terraform.tfvars Terraform requires a correspondent variable declaration.
variable "namespace"{ type = string
description ="String used for prefix resources."} variable "location"{ type = string
description ="Azure Resource Group location"} variable "domain_name"{ type = string
description ="Domain for accessing the Weights & Biases UI."} variable "subdomain"{ type = string
default = null
description ="Subdomain for accessing the Weights & Biases UI. Default creates record at Route53 Route."} variable "license"{ type = string
description ="Your wandb/local license"}
Recommended deployment
This is the most straightforward deployment option configuration that will create all Mandatory components and install in the Kubernetes Cluster the latest version of W&B.
Create the main.tf In the same directory where you created the files in the General Steps, create a file main.tf with the following content:
Deployment option 3 consists of enabling the external message broker. This is optional because the W&B brings embedded a broker. This option doesn’t bring a performance improvement.
The Azure resource that provides the message broker is the Azure Event Grid, and to enable it, you must add the option use_internal_queue = false to the same main.tf that you used in the recommended deployment
You can combine all three deployment options adding all configurations to the same file.
The Terraform Module provides several options that you can combine along with the standard options and the minimal configuration found in recommended deployment
3.4 - Reference Architecture
W&B Reference Architecture
This page describes a reference architecture for a Weights & Biases deployment and outlines the recommended infrastructure and resources to support a production deployment of the platform.
Depending on your chosen deployment environment for Weights & Biases (W&B), various services can help to enhance the resiliency of your deployment.
For instance, major cloud providers offer robust managed database services which help to reduce the complexity of database configuration, maintenance, high availability, and resilience.
This reference architecture addresses some common deployment scenarios and shows how you can integrate your W&B deployment with cloud vendor services for optimal performance and reliability.
Before you start
Running any application in production comes with its own set of challenges, and W&B is no exception. While we aim to streamline the process, certain complexities may arise depending on your unique architecture and design decisions. Typically, managing a production deployment involves overseeing various components, including hardware, operating systems, networking, storage, security, the W&B platform itself, and other dependencies. This responsibility extends to both the initial setup of the environment and its ongoing maintenance.
Consider carefully whether a self-managed approach with W&B is suitable for your team and specific requirements.
A strong understanding of how to run and maintain production-grade application is an important prerequisite before you deploy self-managed W&B. If your team needs assistance, our Professional Services team and partners offer support for implementation and optimization.
The application layer consists of a multi-node Kubernetes cluster, with resilience against node failures. The Kubernetes cluster runs and maintains W&B’s pods.
Storage layer
The storage layer consists of a MySQL database and object storage. The MySQL database stores metadata and the object storage stores artifacts such as models and datasets.
Infrastructure requirements
Kubernetes
The W&B Server application is deployed as a Kubernetes Operator that deploys multiple Pods. For this reason, W&B requires a Kubernetes cluster with:
A fully configured and functioning Ingress controller
The capability to provision Persistent Volumes.
MySQL
W&B stores metadata in a MySQL database. The database’s performance and storage requirements depend on the shapes of the model parameters and related metadata. For example, the database grows in size as you track more training runs, and load on the database increases based on queries in run tables, user workspaces, and reports.
Consider the following when you deploy a self-managed MySQL database:
Backups. You should periodically back up the database to a separate facility. W&B recommends daily backups with at least 1 week of retention.
Performance. The disk the server is running on should be fast. W&B recommends running the database on an SSD or accelerated NAS.
Monitoring. The database should be monitored for load. If CPU usage is sustained at > 40% of the system for more than 5 minutes it is likely a good indication the server is resource starved.
Availability. Depending on your availability and durability requirements you might want to configure a hot standby on a separate machine that streams all updates in realtime from the primary server and can be used to failover to in the event that the primary server crashes or become corrupted.
Object storage
W&B requires object storage with Pre-signed URL and CORS support, deployed in Amazon S3, Azure Cloud Storage, Google Cloud Storage, or a storage service compatible with Amazon S3.service)
Versions
Kubernetes: at least version 1.29.
MySQL: at least 8.0.
Networking
In a deployment connected a public or private network, egress to the following endpoints is required during installation and during runtime:
* https://deploy.wandb.ai
* https://charts.wandb.ai
* https://docker.io
* https://quay.io
* https://gcr.io
Access to W&B and to the object storage is required for the training infrastructure and for each system that tracks the needs of experiments.
DNS
The fully qualified domain name (FQDN) of the W&B deployment must resolve to the IP address of the ingress/load balancer using an A record.
SSL/TLS
W&B requires a valid signed SSL/TLS certificate for secure communication between clients and the server. SSL/TLS termination must occur on the ingress/load balancer. The W&B Server application does not terminate SSL or TLS connections.
Please note: W&B does not recommend the use self-signed certificates and custom CAs.
Supported CPU architectures
W&B runs on the Intel (x86) CPU architecture. ARM is not supported.
Infrastructure provisioning
Terraform is the recommended way to deploy W&B for production. Using Terraform, you define the required resources, their references to other resources, and their dependencies. W&B provides Terraform modules for the major cloud providers. For details, refer to Deploy W&B Server within self managed cloud accounts.
Sizing
Use the following general guidelines as a starting point when planning a deployment. W&B recommends that you monitor all components of a new deployment closely and that you make adjustments based on observed usage patterns. Continue to monitor production deployments over time and make adjustments as needed to maintain optimal performance.
Models only
Kubernetes
Environment
CPU
Memory
Disk
Test/Dev
2 cores
16 GB
100 GB
Production
8 cores
64 GB
100 GB
Numbers are per Kubernetes worker node.
MySQL
Environment
CPU
Memory
Disk
Test/Dev
2 cores
16 GB
100 GB
Production
8 cores
64 GB
500 GB
Numbers are per MySQL node.
Weave only
Kubernetes
Environment
CPU
Memory
Disk
Test/Dev
4 cores
32 GB
100 GB
Production
12 cores
96 GB
100 GB
Numbers are per Kubernetes worker node.
MySQL
Environment
CPU
Memory
Disk
Test/Dev
2 cores
16 GB
100 GB
Production
8 cores
64 GB
500 GB
Numbers are per MySQL node.
Models and Weave
Kubernetes
Environment
CPU
Memory
Disk
Test/Dev
4 cores
32 GB
100 GB
Production
16 cores
128 GB
100 GB
Numbers are per Kubernetes worker node.
MySQL
Environment
CPU
Memory
Disk
Test/Dev
2 cores
16 GB
100 GB
Production
8 cores
64 GB
500 GB
Numbers are per MySQL node.
Cloud provider instance recommendations
Services
Cloud
Kubernetes
MySQL
Object Storage
AWS
EKS
RDS Aurora
S3
GCP
GKE
Google Cloud SQL - Mysql
Google Cloud Storage (GCS)
Azure
AKS
Azure Database for Mysql
Azure Blob Storage
Machine types
These recommendations apply to each node of a self-managed deployment of W&B in cloud infrastructure.
AWS
Environment
K8s (Models only)
K8s (Weave only)
K8s (Models&Weave)
MySQL
Test/Dev
r6i.large
r6i.xlarge
r6i.xlarge
db.r6g.large
Production
r6i.2xlarge
r6i.4xlarge
r6i.4xlarge
db.r6g.2xlarge
GCP
Environment
K8s (Models only)
K8s (Weave only)
K8s (Models&Weave)
MySQL
Test/Dev
n2-highmem-2
n2-highmem-4
n2-highmem-4
db-n1-highmem-2
Production
n2-highmem-8
n2-highmem-16
n2-highmem-16
db-n1-highmem-8
Azure
Environment
K8s (Models only)
K8s (Weave only)
K8s (Models&Weave)
MySQL
Test/Dev
Standard_E2_v5
Standard_E4_v5
Standard_E4_v5
MO_Standard_E2ds_v4
Production
Standard_E8_v5
Standard_E16_v5
Standard_E16_v5
MO_Standard_E8ds_v4
4 - Deploy W&B Platform On-premises
Hosting W&B Server on on-premises infrastructure
W&B recommends fully managed deployment options such as W&B Multi-tenant Cloud or W&B Dedicated Cloud deployment types. W&B fully managed services are simple and secure to use, with minimum to no configuration required.
Reach out to the W&B Sales Team for related question: contact@wandb.com.
Infrastructure guidelines
Before you start deploying W&B, refer to the reference architecture, especially the infrastructure requirements.
W&B strongly recommends to deploy W&B Server into a Kubernetes cluster using the W&B Kubernetes Operator. Deploying to a Kubernetes cluster with the operator ensures that you can use all the existing and latest W&B features.
W&B application performance depends on scalable data stores that your operations team must configure and manage. The team must provide a MySQL 8 database cluster and an AWS S3 compatible object store for the application to scale properly.
Application server
W&B recommends deploying W&B Server into its own namespace and a two availability zone node group with the following specifications to provide the best performance, reliability, and availability:
Specification
Value
Bandwidth
Dual 10 Gigabit+ Ethernet Network
Root Disk Bandwidth (Mbps)
4,750+
Root Disk Provision (GB)
100+
Core Count
4
Memory (GiB)
8
This ensures that W&B Server has sufficient disk space to process the application data and store temporary logs before they are externalized.
It also ensures fast and reliable data transfer, the necessary processing power and memory for smooth operation, and that W&B will not be affected by any noisy neighbors.
It is important to keep in mind that these specifications are minimum requirements, and actual resource needs may vary depending on the specific usage and workload of the W&B application. Monitoring the resource usage and performance of the application is critical to ensure that it operates optimally and to make adjustments as necessary.
Database server
W&B recommends a MySQL 8 database as a metadata store. The shape of the model parameters and related metadata impact the performance of the database. The database size grows as the ML practitioners track more training runs, and incurs read heavy load when queries are executed in run tables, users workspaces, and reports.
To ensure optimal performance W&B recommends deploying the W&B database on to a server with the following starting specs:
Specification
Value
Bandwidth
Dual 10 Gigabit+ Ethernet Network
Root Disk Bandwidth (Mbps)
4,750+
Root Disk Provision (GB)
1000+
Core Count
4
Memory (GiB)
32
Again, W&B recommends monitoring the resource usage and performance of the database to ensure that it operates optimally and to make adjustments as necessary.
Additionally, W&B recommends the following parameter overrides to tune the DB for MySQL 8.
Object storage
W&B is compatible with an object storage that supports S3 API interface, Signed URLs and CORS. W&B recommends specifying the storage array to the current needs of your practitioners and to capacity plan on a regular cadence.
More details on object store configuration can be found in the how-to section.
The Secure Storage Connector is not available for teams at this time for bare metal deployments.
MySQL database
W&B does not recommend using MySQL 5.7. If you are using MySQL 5.7, migrate to MySQL 8 for best compatibility with latest versions of W&B Server. The W&B Server currently only supports MySQL 8 versions 8.0.28 and above.
There are a number of enterprise services that make operating a scalable MySQL database simpler. W&B recommends looking into one of the following solutions:
Due to some changes in the way that MySQL 8.0 handles sort_buffer_size, you might need to update the sort_buffer_size parameter from its default value of 262144. The recommendation is to set the value to 67108864 (64MiB) to ensure that MySQL works efficiently with W&B. MySQL supports this configuration starting with v8.0.28.
Database considerations
Create a database and a user with the following SQL query. Replace SOME_PASSWORD with password of your choice:
CREATEUSER'wandb_local'@'%' IDENTIFIED BY'SOME_PASSWORD';
CREATEDATABASE wandb_local CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;
GRANTALLON wandb_local.*TO'wandb_local'@'%'WITHGRANTOPTION;
Parameter group configuration
Ensure that the following parameter groups are set to tune the database performance:
The object store can be externally hosted on a Minio cluster, or any Amazon S3 compatible object store that has support for signed URLs. Run the following script to check if your object store supports signed URLs.
Additionally, the following CORS policy needs to be applied to the object store.
You can specify your credentials in a connection string when you connect to an Amazon S3 compatible object store. For example, you can specify the following:
s3://$ACCESS_KEY:$SECRET_KEY@$HOST/$BUCKET_NAME
You can optionally tell W&B to only connect over TLS if you configure a trusted SSL certificate for your object store. To do so, add the tls query parameter to the URL. For example, the following URL example demonstrates how to add the TLS query parameter to an Amazon S3 URI:
This will only work if the SSL certificate is trusted. W&B does not support self-signed certificates.
Set BUCKET_QUEUE to internal:// if you use third-party object stores. This tells the W&B server to manage all object notifications internally instead of depending on an external SQS queue or equivalent.
The most important things to consider when running your own object store are:
Storage capacity and performance. It’s fine to use magnetic disks, but you should be monitoring the capacity of these disks. Average W&B usage results in 10’s to 100’s of Gigabytes. Heavy usage could result in Petabytes of storage consumption.
Fault tolerance. At a minimum, the physical disk storing the objects should be on a RAID array. If you use minio, consider running it in distributed mode.
Availability. Monitoring should be configured to ensure the storage is available.
There are many enterprise alternatives to running your own object storage service such as:
Ensure that all machines used to execute machine learning payloads, and the devices used to access the service through web browsers, can communicate to this endpoint.
SSL / TLS
W&B Server does not stop SSL. If your security policies require SSL communication within your trusted networks consider using a tool like Istio and side car containers. The load balancer itself should terminate SSL with a valid certificate. Using self-signed certificates is not supported and will cause a number of challenges for users. If possible using a service like Let’s Encrypt is a great way to provided trusted certificates to your load balancer. Services like Caddy and Cloudflare manage SSL for you.
Example nginx configuration
The following is an example configuration using nginx as a reverse proxy.
events {}
http {
# If we receive X-Forwarded-Proto, pass it through; otherwise, pass along the
# scheme used to connect to this server
map $http_x_forwarded_proto $proxy_x_forwarded_proto {
default $http_x_forwarded_proto;
'' $scheme;
}
# Also, in the above case, force HTTPS
map $http_x_forwarded_proto $sts {
default'';
"https""max-age=31536000; includeSubDomains";
}
# If we receive X-Forwarded-Host, pass it though; otherwise, pass along $http_host
map $http_x_forwarded_host $proxy_x_forwarded_host {
default $http_x_forwarded_host;
'' $http_host;
}
# If we receive X-Forwarded-Port, pass it through; otherwise, pass along the
# server port the client connected to
map $http_x_forwarded_port $proxy_x_forwarded_port {
default $http_x_forwarded_port;
'' $server_port;
}
# If we receive Upgrade, set Connection to "upgrade"; otherwise, delete any
# Connection header that may have been passed to this server
map $http_upgrade $proxy_connection {
defaultupgrade;
''close;
}
server {
listen443ssl;
server_namewww.example.com;
ssl_certificatewww.example.com.crt;
ssl_certificate_keywww.example.com.key;
proxy_http_version1.1;
proxy_bufferingoff;
proxy_set_headerHost $http_host;
proxy_set_headerUpgrade $http_upgrade;
proxy_set_headerConnection $proxy_connection;
proxy_set_headerX-Real-IP $remote_addr;
proxy_set_headerX-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_headerX-Forwarded-Proto $proxy_x_forwarded_proto;
proxy_set_headerX-Forwarded-Host $proxy_x_forwarded_host;
location/ {
proxy_passhttp://$YOUR_UPSTREAM_SERVER_IP:8080/;
}
keepalive_timeout10;
}
}
Verify your installation
Very your W&B Server is configured properly. Run the following commands in your terminal:
Check log files to view any errors the W&B Server hits at startup. Run the following commands:
docker logs wandb-local
kubectl get pods
kubectl logs wandb-XXXXX-XXXXX
Contact W&B Support if you encounter errors.
5 - Update W&B license and version
Guide for updating W&B (Weights & Biases) version and license across different installation methods.
Update your W&B Server Version and License with the same method you installed W&B Server with. The following table lists how to update your license and version based on different deployment methods:
First, navigate to the W&B maintained Terraform module for your appropriate cloud provider. See the preceding table to find the appropriate Terraform module based on your cloud provider.
Within your Terraform configuration, update wandb_version and license in your Terraform wandb_app module configuration:
module"wandb_app" {
source ="wandb/wandb/<cloud-specific-module>" version ="new_version" license ="new_license_key" # Your new license key
wandb_version ="new_wandb_version" # Desired W&B version
...
}
Apply the Terraform configuration with terraform plan and terraform apply.
terraform init
terraform apply
(Optional) If you use a terraform.tfvars or other .tfvars file.
Update or create a terraform.tfvars file with the new W&B version and license key.
terraform plan -var-file="terraform.tfvars"
Apply the configuration. In your Terraform workspace directory execute:
terraform apply -var-file="terraform.tfvars"
Update with Helm
Update W&B with spec
Specify a new version by modifying the image.tag and/or license values in your Helm chart *.yaml configuration file:
For more details, see the upgrade guide in the public repository.
Update with admin UI
This method is only works for updating licenses that are not set with an environment variable in the W&B server container, typically in self-hosted Docker installations.
Obtain a new license from the W&B Deployment Page, ensuring it matches the correct organization and deployment ID for the deployment you are looking to upgrade.
Access the W&B Admin UI at <host-url>/system-settings.