Deploy on Air-Gapped Kubernetes - Weights & Biases Documentation

Introduction

This guide provides step-by-step instructions to deploy the W&B Platform in air-gapped, disconnected, or restricted network customer-managed environments. By following this guide, you set up an internal container registry and Helm repository to host W&B images and charts, install the W&B Kubernetes Operator, and deploy the W&B Platform without requiring outbound internet connectivity. This guide targets platform administrators and DevOps engineers who manage Kubernetes infrastructure in regulated or isolated networks. Air-gapped deployments are common in the following environments:

Secure government facilities.
Financial institutions with strict network isolation.
Healthcare organizations with compliance requirements.
Industrial control systems (ICS) environments.
Research facilities with classified networks.

Run these commands in a shell console with proper access to the Kubernetes cluster. You can adapt these commands to work with any CI/CD tooling you use to deploy Kubernetes applications. For standard on-premises Kubernetes deployments with internet connectivity, see Deploy W&B with Kubernetes Operator.

Prerequisites

Before starting, ensure your air-gapped environment meets the following requirements.

Version requirements

Software	Minimum version
Kubernetes	v1.34 or newer (Supported Kubernetes versions)
Helm	v3.x
MySQL	W&B Self-Managed deployments must run a supported MySQL version that receives security patches and critical bug fixes. Install or upgrade to MySQL 8.4.x, or use a managed service version that your provider documents as supported and patched. Aurora MySQL version strings differ from community MySQL versions. Use `SELECT version()` to see the full engine version string and `SELECT aurora_version()` to see the Aurora version. Aurora MySQL version 3 is compatible with MySQL 8.0.x and is still supported. See Amazon Aurora versioning and your cloud provider’s documentation when you choose a target version.
Redis	v7.x
ClickHouse	Required for Weave and some Models OLAP features on Self-Managed deployments. Minimum version depends on which features are enabled and your W&B Server version. See ClickHouse compatibility for upgrades and Supported W&B Server releases.

SSL/TLS requirements

W&B requires a valid signed SSL/TLS certificate for secure communication between clients and the server. SSL/TLS termination must occur on the ingress or load balancer. The W&B Server application doesn’t terminate SSL or TLS connections.

W&B doesn’t support self-signed certificates or custom CAs. Self-signed certificates cause challenges for users and aren’t supported.

If possible, use a service like Let’s Encrypt to provide trusted certificates to your load balancer. Services like Caddy and Cloudflare manage SSL for you. If your security policies require SSL communication within your trusted networks, consider using a tool like Istio and sidecar containers.

Hardware requirements

CPU Architecture: W&B runs on Intel (x86) CPU architecture only. ARM is not supported. Sizing: For CPU, memory, and disk sizing recommendations for Kubernetes nodes and MySQL, see the Sizing section in the reference architecture. Requirements vary based on whether you’re running Models, Weave, or both.

MySQL database

W&B requires an external MySQL database. For production, W&B recommends using managed database services:

Managed database services provide automated backups, monitoring, high availability, patching, and reduce operational overhead. See the reference architecture for MySQL requirements, including sizing recommendations and configuration parameters. For the SQL to create the database, see the bare-metal guide. For questions about your deployment’s database configuration, contact support or your AISE. For MySQL configuration parameters for self-managed instances, see the reference architecture MySQL configuration section.

Redis

W&B depends on a single-node Redis 7.x deployment, which W&B’s components use for job queuing and data caching. For testing and proof-of-concept work, W&B Self-Managed includes a local Redis deployment. This bundled deployment isn’t appropriate for production use. For production deployments, W&B can connect to a Redis instance in the following environments:

Amazon ElastiCache
Google Cloud Memorystore
Azure Cache for Redis
Self-hosted Redis in your cloud or on-premises infrastructure

Object storage

W&B requires object storage with pre-signed URL and CORS support. W&B recommends the following storage providers:

Amazon S3: Object storage service offering scalability, data availability, security, and performance.
Google Cloud Storage: Managed service for storing unstructured data at scale.
Azure Blob Storage: Cloud-based object storage for unstructured data at scale.
CoreWeave AI Object Storage: S3-compatible object storage optimized for AI workloads.
Enterprise S3-compatible storage such as MinIO Enterprise (AIStor), NetApp StorageGRID, or other enterprise solutions.

MinIO Open Source is in maintenance mode with no active development or pre-compiled binaries. For production deployments, W&B recommends managed object storage services or enterprise S3-compatible solutions such as MinIO Enterprise (AIStor).

After you select a provider, configure the bucket so that W&B can access it. For detailed bucket provisioning instructions including IAM policies, CORS configuration, and access setup, see the Bring Your Own Bucket (BYOB) guide. For the complete list of object storage requirements, including capacity and performance guidance, see the reference architecture object storage section. For detailed object storage provisioning guidance, see the Bring Your Own Bucket (BYOB) guide. In air-gapped environments, you’ll typically use on-premises S3-compatible storage such as MinIO Enterprise, NetApp StorageGRID, or Dell ECS.

Air-gapped specific requirements

In addition to the preceding standard requirements, air-gapped deployments require the following:

Internal container registry: Access to a private container registry such as Harbor, JFrog Artifactory, or Nexus, with all required W&B images.
Internal Helm repository: Access to a private Helm chart repository with W&B Helm charts.
Image transfer capability: A method to transfer container images from an internet-connected system to your air-gapped registry.
License file: A valid W&B Enterprise license. To obtain a license (for example, from an internet-connected machine), see the License section on the Requirements page, or contact your W&B account team.

For complete infrastructure requirements, including networking and load balancer configuration, see the reference architecture.

Prepare your air-gapped environment

The following steps prepare your air-gapped environment to host the W&B container images and Helm charts. Complete these steps before installing the operator or deploying the platform.

Step 1: Set up internal container registry

Because the Kubernetes cluster cannot pull images from public registries, all required container images must be available in your internal air-gapped container registry before deployment.

You are responsible for tracking the W&B Operator’s requirements and maintaining your container registry with updated images regularly. For the most current list of required container images and versions, refer to the Helm chart, or contact W&B Support or your assigned W&B support engineer.

Core W&B component containers

The following core images are required:

docker.io/wandb/controller: W&B Kubernetes Operator.
docker.io/wandb/local: W&B application server.
docker.io/wandb/console: W&B management console.
docker.io/wandb/megabinary: W&B microservices (API, executor, glue, parquet).

Dependency containers

The following third-party dependency images are required:

docker.io/bitnamilegacy/redis: Required for local Redis deployment during testing and development. For production Redis requirements, see the Redis section in Prerequisites.
docker.io/otel/opentelemetry-collector-contrib: OpenTelemetry agent for collecting metrics and logs.
quay.io/prometheus/prometheus: Prometheus for metrics collection.
quay.io/prometheus-operator/prometheus-config-reloader: Prometheus dependency.

Get the complete image list

To extract the complete list of required images and versions from the Helm chart:

On an internet-connected system, download the W&B Helm charts from the W&B Helm charts repository:

# Clone the helm-charts repository
git clone https://github.com/wandb/helm-charts.git
cd helm-charts

Inspect the values.yaml files to identify all container images and their versions:

# Extract image references from the operator chart
helm show values charts/operator | grep -E "repository:|tag:" | grep -v "^#"

# Extract image references from the platform chart
helm show values charts/operator-wandb | grep -E "repository:|tag:" | grep -v "^#"

Alternatively, use this command to extract only the repository names (without version tags):

helm show values charts/operator-wandb \
  | awk -F': *' '/^[[:space:]]*repository:/{print $2}' \
  | grep -v "^#" \
  | sort -u

The list of repositories looks similar to the following:

wandb/controller
wandb/local
wandb/console
wandb/megabinary
wandb/weave-python
wandb/weave-trace
otel/opentelemetry-collector-contrib
prometheus/prometheus
prometheus-operator/prometheus-config-reloader
bitnamilegacy/redis

To get the specific version tags for each image, use the preceding first command (grep -E "repository:|tag:"), which shows both repository names and their corresponding version tags.

Transfer images to air-gapped registry

On an internet-connected system, pull and save all required images.

Replace version numbers in the following examples with the actual versions from your Helm chart inspection in the preceding step. The versions shown here are examples and become outdated over time.

Use shell variables to manage versions consistently:

# Set version variables (update these based on your Helm chart versions)
CONTROLLER_VERSION="1.13.3"
APP_VERSION="0.59.2"
CONSOLE_VERSION="2.12.2"

# Pull images
docker pull wandb/controller:${CONTROLLER_VERSION}
docker pull wandb/local:${APP_VERSION}
docker pull wandb/console:${CONSOLE_VERSION}
docker pull wandb/megabinary:${APP_VERSION}
# ... pull all other required images with their versions

# Save images to .tar files
docker save wandb/controller:${CONTROLLER_VERSION} -o wandb-controller-${CONTROLLER_VERSION}.tar
docker save wandb/local:${APP_VERSION} -o wandb-local-${APP_VERSION}.tar
docker save wandb/console:${CONSOLE_VERSION} -o wandb-console-${CONSOLE_VERSION}.tar
docker save wandb/megabinary:${APP_VERSION} -o wandb-megabinary-${APP_VERSION}.tar
# ... save all other images

Transfer the .tar files to your air-gapped environment using your approved method, such as a USB drive or secure file transfer.

In your air-gapped environment, load and push images to your internal registry:

# Set the same version variables used above
CONTROLLER_VERSION="1.13.3"
APP_VERSION="0.59.2"
CONSOLE_VERSION="2.12.2"
INTERNAL_REGISTRY="registry.yourdomain.com"

# Load images
docker load -i wandb-controller-${CONTROLLER_VERSION}.tar
docker load -i wandb-local-${APP_VERSION}.tar
docker load -i wandb-console-${CONSOLE_VERSION}.tar
docker load -i wandb-megabinary-${APP_VERSION}.tar
# ... load all other images

# Tag for internal registry
docker tag wandb/controller:${CONTROLLER_VERSION} ${INTERNAL_REGISTRY}/wandb/controller:${CONTROLLER_VERSION}
docker tag wandb/local:${APP_VERSION} ${INTERNAL_REGISTRY}/wandb/local:${APP_VERSION}
docker tag wandb/console:${CONSOLE_VERSION} ${INTERNAL_REGISTRY}/wandb/console:${CONSOLE_VERSION}
docker tag wandb/megabinary:${APP_VERSION} ${INTERNAL_REGISTRY}/wandb/megabinary:${APP_VERSION}
# ... tag all other images

# Push to internal registry
docker push ${INTERNAL_REGISTRY}/wandb/controller:${CONTROLLER_VERSION}
docker push ${INTERNAL_REGISTRY}/wandb/local:${APP_VERSION}
docker push ${INTERNAL_REGISTRY}/wandb/console:${CONSOLE_VERSION}
docker push ${INTERNAL_REGISTRY}/wandb/megabinary:${APP_VERSION}
# ... push all other images

Step 2: Set up internal Helm chart repository

With the container images in place, the Kubernetes Operator also needs access to the W&B Helm charts. Ensure the following Helm charts are available in your internal Helm repository:

On an internet-connected system, download the charts:

# Add W&B Helm repository
helm repo add wandb https://wandb.github.io/helm-charts
helm repo update

# Download the charts
helm pull wandb/operator --version 1.13.3
helm pull wandb/operator-wandb --version 0.18.0

Transfer the .tgz chart files to your air-gapped environment and upload them to your internal Helm repository according to your repository’s procedures. The operator chart deploys the W&B Kubernetes Operator (Controller Manager). The operator-wandb chart deploys the W&B Platform using the values configured in the Custom Resource (CR).

Step 3: Configure Helm repository access

Configure your local Helm client in the air-gapped environment to point to your internal repository so that subsequent install commands can locate the charts.

In your air-gapped environment, configure Helm to use your internal repository:

helm repo add local-repo https://charts.yourdomain.com
helm repo update

Verify the charts are available:

helm search repo local-repo/operator
helm search repo local-repo/operator-wandb

Deploy W&B in air-gapped environment

With your internal registry and Helm repository in place, you can now install the Kubernetes Operator, configure external services, and deploy the W&B Platform.

Step 4: Install the Kubernetes Operator

The W&B Kubernetes Operator (controller manager) manages the W&B Platform components. To install it in an air-gapped environment, configure it to use your internal container registry.

Create a values.yaml file with the following content:
image: repository: registry.yourdomain.com/wandb/controller tag: 1.13.3 airgapped: true
Replace the repository and tag with the actual versions you transferred to your internal registry in Step 1. The version shown here (1.13.3) is an example and becomes outdated over time.

Install the operator and Custom Resource Definition (CRD):

helm upgrade --install operator local-repo/operator \
  --namespace wandb \
  --create-namespace \
  --values values.yaml

Verify the operator is running:
kubectl get pods -n wandb
You should see the operator pod in a Running state.

The W&B Kubernetes Operator is now installed and ready to deploy the W&B Platform from your internal chart repository. For full details about supported values, refer to the Kubernetes operator GitHub repository values file.

Step 5: Set up MySQL database

Before configuring the W&B Custom Resource, set up an external MySQL database. For production deployments, W&B strongly recommends using managed database services where available. However, if you’re running your own MySQL instance, create the database and user: Create a database and a user with the following SQL commands. Replace [PASSWORD] with a secure password:

CREATE USER 'wandb_local'@'%' IDENTIFIED BY '[PASSWORD]';
CREATE DATABASE wandb_local CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;
GRANT ALL ON wandb_local.* TO 'wandb_local'@'%' WITH GRANT OPTION;

For MySQL configuration parameters, see the reference architecture MySQL configuration section.

Step 6: Configure W&B Custom Resource

After installing the W&B Kubernetes Operator, configure the Custom Resource (CR) to point to your internal Helm repository and container registry. This configuration ensures the Kubernetes operator uses your internal registry and repository when deploying the required components of the W&B Platform, instead of attempting to reach public sources.

The following example configuration includes image version tags that become outdated over time. Replace all tag: values with the actual versions you transferred to your internal registry in Step 1.

Create a file named wandb.yaml with the following content:

apiVersion: apps.wandb.com/v1
kind: WeightsAndBiases
metadata:
  labels:
    app.kubernetes.io/instance: wandb
    app.kubernetes.io/name: weightsandbiases
  name: wandb
  namespace: wandb

spec:
  chart:
    url: https://charts.yourdomain.com
    name: operator-wandb
    version: 0.18.0

  values:
    global:
      host: https://wandb.yourdomain.com
      license: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
      
      bucket:
        accessKey: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
        secretKey: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
        name: s3.yourdomain.com:9000
        path: wandb
        provider: s3
        region: us-east-1
      
      mysql:
        database: wandb
        host: mysql.yourdomain.com
        password: [YOUR-MYSQL-PASSWORD]
        port: 3306
        user: wandb
      
      redis:
        host: redis.yourdomain.com
        port: 6379
        password: [YOUR-REDIS-PASSWORD]
      
      api:
        enabled: true
      
      glue:
        enabled: true
      
      executor:
        enabled: true
      
      extraEnv:
        ENABLE_REGISTRY_UI: 'true'

    # Configure all component images to use internal registry
    app:
      image:
        repository: registry.yourdomain.com/wandb/local
        tag: 0.59.2

    console:
      image:
        repository: registry.yourdomain.com/wandb/console
        tag: 2.12.2

    api:
      image:
        repository: registry.yourdomain.com/wandb/megabinary
        tag: 0.59.2

    executor:
      image:
        repository: registry.yourdomain.com/wandb/megabinary
        tag: 0.59.2

    glue:
      image:
        repository: registry.yourdomain.com/wandb/megabinary
        tag: 0.59.2

    parquet:
      image:
        repository: registry.yourdomain.com/wandb/megabinary
        tag: 0.59.2

    weave:
      image:
        repository: registry.yourdomain.com/wandb/weave-python
        tag: 0.59.2

    otel:
      image:
        repository: registry.yourdomain.com/otel/opentelemetry-collector-contrib
        tag: 0.97.0

    prometheus:
      server:
        image:
          repository: registry.yourdomain.com/prometheus/prometheus
          tag: v2.47.0
      configmapReload:
        prometheus:
          image:
            repository: registry.yourdomain.com/prometheus-operator/prometheus-config-reloader
            tag: v0.67.0

    ingress:
      annotations:
        nginx.ingress.kubernetes.io/proxy-body-size: "0"
      class: nginx

Replace all placeholder values, such as hostnames, passwords, and tags, with your actual configuration values. The preceding example shows the most commonly used components.

Depending on your deployment needs, you may also need to configure image repositories for additional components such as the following:

settingsMigrationJob
weave-trace
filestream
flat-runs-table

Refer to the W&B Helm repository values file for the complete list of configurable components.

Step 7: Deploy the W&B Platform

Applying the Custom Resource triggers the operator to install the W&B Platform components defined in the operator-wandb chart, using the configuration and image references from wandb.yaml.

Apply the W&B Custom Resource to deploy the platform:
kubectl apply -f wandb.yaml

Monitor the deployment progress:

# Watch pods being created
kubectl get pods -n wandb --watch

# Check deployment status
kubectl get weightsandbiases -n wandb

# View operator logs
kubectl logs -n wandb deployment/wandb-operator-controller-manager

The deployment may take several minutes as the operator creates all necessary components.

OpenShift configuration

W&B supports deployment on air-gapped OpenShift Kubernetes clusters. OpenShift deployments require additional security context configurations because of OpenShift’s stricter security policies. If you’re deploying on OpenShift, apply the configurations in this section in addition to the preceding steps.

OpenShift security context constraints

OpenShift uses Security Context Constraints (SCCs) to control pod permissions. By default, OpenShift assigns the restricted SCC to pods, which prevents running as root and requires specific user IDs.

Option 1: Use restricted SCC (recommended)

Configure W&B components to run with the restricted SCC by setting appropriate security contexts in your Custom Resource:

spec:
  values:
    # Configure security contexts for all pods
    app:
      podSecurityContext:
        fsGroup: 1000
        runAsUser: 1000
        runAsNonRoot: true
      securityContext:
        allowPrivilegeEscalation: false
        capabilities:
          drop:
            - ALL
        runAsNonRoot: true
        seccompProfile:
          type: RuntimeDefault

    console:
      podSecurityContext:
        fsGroup: 1000
        runAsUser: 1000
        runAsNonRoot: true
      securityContext:
        allowPrivilegeEscalation: false
        capabilities:
          drop:
            - ALL
        runAsNonRoot: true
        seccompProfile:
          type: RuntimeDefault

    # Repeat for other components: api, executor, glue, parquet, weave

Option 2: Create custom SCC (if required)

If your deployment requires capabilities not available in the restricted SCC, create a custom SCC:

apiVersion: security.openshift.io/v1
kind: SecurityContextConstraints
metadata:
  name: wandb-scc
allowHostDirVolumePlugin: false
allowHostIPC: false
allowHostNetwork: false
allowHostPID: false
allowHostPorts: false
allowPrivilegeEscalation: false
allowPrivilegedContainer: false
allowedCapabilities: []
defaultAddCapabilities: []
fsGroup:
  type: MustRunAs
  ranges:
    - min: 1000
      max: 65535
readOnlyRootFilesystem: false
requiredDropCapabilities:
  - ALL
runAsUser:
  type: MustRunAsRange
  uidRangeMin: 1000
  uidRangeMax: 65535
seLinuxContext:
  type: MustRunAs
supplementalGroups:
  type: RunAsAny
volumes:
  - configMap
  - downwardAPI
  - emptyDir
  - persistentVolumeClaim
  - projected
  - secret

Apply the SCC:
oc apply -f wandb-scc.yaml

Bind the SCC to the W&B service accounts:

oc adm policy add-scc-to-user wandb-scc -z wandb-app -n wandb
oc adm policy add-scc-to-user wandb-scc -z wandb-console -n wandb

OpenShift routes

OpenShift uses Routes instead of standard Kubernetes Ingress. Configure W&B to use OpenShift Routes:

spec:
  values:
    ingress:
      enabled: false
    
    route:
      enabled: true
      host: wandb.apps.openshift.yourdomain.com
      tls:
        enabled: true
        termination: edge
        insecureEdgeTerminationPolicy: Redirect

OpenShift image pull configuration

If your OpenShift cluster uses an internal image registry with authentication:

Create an image pull secret:

kubectl create secret docker-registry wandb-registry-secret \
  --docker-server=registry.yourdomain.com \
  --docker-username=[USERNAME] \
  --docker-password=[PASSWORD] \
  --namespace=wandb

Reference the secret in your Custom Resource:

spec:
  values:
    imagePullSecrets:
      - name: wandb-registry-secret

OpenShift complete example

The following example shows a complete CR for OpenShift air-gapped deployment:

Replace all tag: values in this example with the actual versions you transferred to your internal registry in Step 1. The versions shown are examples and become outdated over time.

apiVersion: apps.wandb.com/v1
kind: WeightsAndBiases
metadata:
  name: wandb
  namespace: wandb

spec:
  chart:
    url: https://charts.yourdomain.com
    name: operator-wandb
    version: 0.18.0

  values:
    global:
      host: https://wandb.apps.openshift.yourdomain.com
      license: [YOUR-LICENSE]
      
      bucket:
        accessKey: [YOUR-ACCESS-KEY]
        secretKey: [YOUR-SECRET-KEY]
        name: s3.yourdomain.com:9000
        path: wandb
        provider: s3
        region: us-east-1
      
      mysql:
        database: wandb
        host: mysql.yourdomain.com
        password: [YOUR-MYSQL-PASSWORD]
        port: 3306
        user: wandb
      
      redis:
        host: redis.yourdomain.com
        port: 6379
        password: [YOUR-REDIS-PASSWORD]

    # OpenShift-specific: Use Routes instead of Ingress
    ingress:
      enabled: false
    
    route:
      enabled: true
      host: wandb.apps.openshift.yourdomain.com
      tls:
        enabled: true
        termination: edge

    # Image pull secret for internal registry
    imagePullSecrets:
      - name: wandb-registry-secret

    # Security contexts for OpenShift restricted SCC
    app:
      image:
        repository: registry.yourdomain.com/wandb/local
        tag: 0.59.2
      podSecurityContext:
        fsGroup: 1000
        runAsUser: 1000
        runAsNonRoot: true
      securityContext:
        allowPrivilegeEscalation: false
        capabilities:
          drop:
            - ALL
        runAsNonRoot: true
        seccompProfile:
          type: RuntimeDefault

    console:
      image:
        repository: registry.yourdomain.com/wandb/console
        tag: 2.12.2
      podSecurityContext:
        fsGroup: 1000
        runAsUser: 1000
        runAsNonRoot: true
      securityContext:
        allowPrivilegeEscalation: false
        capabilities:
          drop:
            - ALL
        runAsNonRoot: true
        seccompProfile:
          type: RuntimeDefault

    # Repeat security contexts for: api, executor, glue, parquet, weave
    # (abbreviated for clarity)

Contact W&B Support or your assigned W&B support engineer for comprehensive OpenShift configuration examples tailored to your security requirements.

Verify your installation

After deploying W&B, verify the installation is working correctly so that you can confirm the platform is reachable, pods are healthy, and the deployment is using only your internal resources. Follow the general verification steps, then complete the additional air-gapped checks in the following section. To verify the installation, W&B recommends using the W&B CLI. The wandb verify command runs tests that confirm components and configurations work as expected.

This procedure assumes that you create the first admin user account in a browser.

To verify the installation:

Install the W&B CLI:
pip install wandb

wandb login --host=https://YOUR_DNS_DOMAIN

For example:

wandb login --host=https://wandb.company-name.com

Verify the installation:
wandb verify

After the command runs, a successful installation displays the following output:

Default host selected:  https://wandb.company-name.com
Find detailed logs for this test at: /var/folders/pn/b3g3gnc11_sbsykqkm3tx5rh0000gp/T/tmpdtdjbxua/wandb
Checking if logged in...................................................✅
Checking signed URL upload..............................................✅
Checking ability to send large payloads through proxy...................✅
Checking requests to base url...........................................✅
Checking requests made over signed URLs.................................✅
Checking CORs configuration of the bucket...............................✅
Checking wandb package version is up to date............................✅
Checking logged metrics, saving and downloading a file..................✅
Checking artifact save and download workflows...........................✅

Contact W&B Support if you encounter errors.

Additional air-gapped verification

For air-gapped deployments, also verify:

Image pull: Confirm all pods successfully pulled images from your internal registry:
kubectl get pods -n wandb -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.phase}{"\t"}{.status.containerStatuses[*].image}{"\n"}{end}'
All images should point to your internal registry and all pods should be in Running state.
External connectivity: Verify W&B is not attempting external connections (it shouldn’t in air-gapped mode):
kubectl logs -n wandb deployment/wandb-app --tail=100 | grep -i "connection"
License validation: Access the W&B console and verify your license is active.

Troubleshooting

Image pull errors

If pods fail to pull images, check the following:

Verify images exist in your internal registry.
Check that the image pull secret is correctly configured.
Verify network connectivity from Kubernetes nodes to the registry.
Check registry authentication credentials.

To test an image pull manually:

kubectl run test-pull --image=registry.yourdomain.com/wandb/local:0.59.2 --namespace=wandb
kubectl logs test-pull -n wandb
kubectl delete pod test-pull -n wandb

OpenShift SCC errors

If pods fail with permission errors on OpenShift:

# Check which SCC is being used
oc get pod [POD-NAME] -n wandb -o yaml | grep scc

# Check service account permissions
oc describe scc wandb-scc
oc get rolebinding -n wandb

Helm chart not found

If the operator can’t find the platform chart, check the following:

Verify the chart repository URL in the Custom Resource.
Check that the operator pod can reach your internal Helm repository.

Verify the chart exists in your repository:

helm search repo local-repo/operator-wandb

Frequently asked questions

Can I use a different ingress class?

Yes, configure your ingress class by modifying the ingress settings in your Custom Resource:

spec:
  values:
    ingress:
      class: your-ingress-class

How do I handle certificate bundles with multiple certificates?

Split the certificates into multiple entries in the customCACerts section:

spec:
  values:
    customCACerts:
      cert1.crt: |
        -----BEGIN CERTIFICATE-----
        ...
        -----END CERTIFICATE-----
      cert2.crt: |
        -----BEGIN CERTIFICATE-----
        ...
        -----END CERTIFICATE-----

How do I prevent automatic updates?

To configure the operator to not automatically update W&B, do the following:

Set airgapped: true in the operator installation (this disables automatic update checks).
Control version updates by manually updating the spec.chart.version in your Custom Resource.
Optionally, disable automatic updates from the W&B System Console.

See Disable automatic app version updates for more details.

W&B strongly recommends customers with Self-Managed instances update their deployments with the latest release at minimum once per quarter to maintain support and receive the latest features, performance improvements, and fixes. W&B supports a major release for 12 months from its initial release date. Refer to Release policies and processes.

Does the deployment work with no connection to public repositories?

Yes. When airgapped: true is set in the operator configuration, the Kubernetes operator uses only your internal resources and doesn’t attempt to connect to public repositories.

How do I update W&B in an air-gapped environment?

To update W&B:

Pull new container images on an internet-connected system.
Transfer images to your air-gapped registry.
Upload new Helm charts to your internal repository.
Update the spec.chart.version and image tags in your Custom Resource.
Apply the updated Custom Resource. The operator performs a rolling update of the W&B components.

Next steps

After successful deployment, complete the following tasks:

Configure user authentication: Set up SSO or other authentication methods.
Set up monitoring: Configure monitoring for your W&B instance and infrastructure.
Plan for updates: Review the Server upgrade process and establish an update cadence.
Configure backups: Establish backup procedures for your MySQL database.
Document your process: Create runbooks for your specific air-gapped update procedures.

Get help

If you encounter issues during deployment:

Review the Reference Architecture for infrastructure guidance.
Check the Operator guide for configuration details.
Contact W&B Support or your assigned W&B support engineer.
For OpenShift-specific issues, reference Red Hat OpenShift documentation.

​Introduction

​Prerequisites

​Version requirements

​SSL/TLS requirements

​Hardware requirements

​MySQL database

​Redis

​Object storage

​Air-gapped specific requirements

​Prepare your air-gapped environment

​Step 1: Set up internal container registry

​Core W&B component containers

​Dependency containers

​Get the complete image list

​Transfer images to air-gapped registry

​Step 2: Set up internal Helm chart repository

​Step 3: Configure Helm repository access

​Deploy W&B in air-gapped environment

​Step 4: Install the Kubernetes Operator

​Step 5: Set up MySQL database

​Step 6: Configure W&B Custom Resource

​Step 7: Deploy the W&B Platform

​OpenShift configuration

​OpenShift security context constraints

​Option 1: Use restricted SCC (recommended)

​Option 2: Create custom SCC (if required)

​OpenShift routes

​OpenShift image pull configuration

​OpenShift complete example

​Verify your installation

​Additional air-gapped verification

​Troubleshooting

​Image pull errors

​OpenShift SCC errors

​Helm chart not found

​Frequently asked questions

​Can I use a different ingress class?

​How do I handle certificate bundles with multiple certificates?

​How do I prevent automatic updates?

​Does the deployment work with no connection to public repositories?

​How do I update W&B in an air-gapped environment?

​Next steps

​Get help

Introduction

Prerequisites

Version requirements

SSL/TLS requirements

Hardware requirements

MySQL database

Redis

Object storage

Air-gapped specific requirements

Prepare your air-gapped environment

Step 1: Set up internal container registry

Core W&B component containers

Dependency containers

Get the complete image list

Transfer images to air-gapped registry

Step 2: Set up internal Helm chart repository

Step 3: Configure Helm repository access

Deploy W&B in air-gapped environment

Step 4: Install the Kubernetes Operator

Step 5: Set up MySQL database

Step 6: Configure W&B Custom Resource

Step 7: Deploy the W&B Platform

OpenShift configuration

OpenShift security context constraints

Option 1: Use restricted SCC (recommended)

Option 2: Create custom SCC (if required)

OpenShift routes

OpenShift image pull configuration

OpenShift complete example

Verify your installation

Additional air-gapped verification

Troubleshooting

Image pull errors

OpenShift SCC errors

Helm chart not found

Frequently asked questions

Can I use a different ingress class?

How do I handle certificate bundles with multiple certificates?

How do I prevent automatic updates?

Does the deployment work with no connection to public repositories?

How do I update W&B in an air-gapped environment?

Next steps

Get help