Skip to main content

Bring your own bucket (BYOB)

Bring your own bucket (BYOB) allows you to store W&B artifacts and other related sensitive data in your own cloud or on-prem infrastructure. In case of Dedicated Cloud or SaaS Cloud, data that you store in your bucket is not copied to the W&B managed infrastructure.

info
  • Communication between W&B SDK / CLI / UI and your buckets occurs using pre-signed URLs.
  • W&B uses a garbage collection process to delete W&B Artifacts. For more information, see Deleting Artifacts.
  • You can specify a sub-path when configuring a bucket, to ensure that W&B does not store any files in a folder at the root of the bucket. It can help you better conform to your organzation's bucket governance policy.

Configuration options​

There are two scopes you can configure your storage bucket to: at the Instance level or at a Team level.

  • Instance level: Any user that has relevant permissions within your organization can access files stored in your instance level storage bucket.
  • Team level: Members of a W&B Team can access files stored in the bucket configured at the Team level. Team level storage buckets allow greater data access control and data isolation for teams with highly sensitive data or strict compliance requirements.

You can configure your bucket at both the instance level and separately for one or more teams within your organization.

For example, suppose you have a team called Kappa in your organization. Your organization (and Team Kappa) use the Instance level storage bucket by default. Next, you create a team called Omega. When you create Team Omega, you configure a Team level storage bucket for that team. Files generated by Team Omega are not accessible by Team Kappa. However, files created by Team Kappa are accessible by Team Omega. If you want to isolate data for Team Kappa, you must configure a Team level storage bucket for them as well.

tip

Team level storage bucket provides the same benefits for Self-Managed instances, especially when different business units and departments share an instance to efficiently utilize the infrastructure and administrative resources. This also applies to firms that have separate project teams managing AI workflows for separate customer engagements.

Availability matrix​

The following table shows the availability of BYOB across different W&B Server deployment types. An X means the feature is available on the specific deployment type.

W&B Server deployment typeInstance levelTeam levelAdditional information
Dedicated CloudXXBoth the instance and team level BYOB are available for Amazon Web Services, Google Cloud Platform and Microsoft Azure. For the team-level BYOB, you can connect to a cloud-native storage bucket in the same or another cloud, or even a S3-compatible secure storage like MinIO hosted in your cloud or on-prem infrastructure.
SaaS CloudNot ApplicableXThe team level BYOB is available only for Amazon Web Services and Google Cloud Platform. W&B fully manages the default and only storage bucket for Microsoft Azure.
Self-managedXXInstance level BYOB is the default since the instance is fully managed by you. If your self-managed instance is in cloud, you can connect to a cloud-native storage bucket in the same or another cloud for the team-level BYOB. You can also use S3-compatible secure storage like MinIO for either of instance or team-level BYOB.
caution

Once you configure a instance or team level storage bucket for your Dedicated Cloud or Self-managed instance, or a team level storage bucket for your SaaS Cloud account, you can not change or reconfigure the storage bucket for any of those scopes. That includes the inability to migrate data to another bucket and remap relevant references in the main product storage. W&B recommends to plan your storage bucket layout carefully before configuring for either of the instance or team level scopes. Reach out to your W&B team for any questions.

Cross-cloud or S3-compatible storage for team-level BYOB​

You can connect to a cloud-native storage bucket in another cloud or to an S3-compatible storage bucket like MinIO for team-level BYOB in your Dedicated Cloud or Self-Managed instance.

To enable the use of cross-cloud or S3-compatible storage, specify the storage bucket including the relevant access key in one of the following formats, using the GORILLA_SUPPORTED_FILE_STORES environment variable for your W&B instance.

Configure an S3-compatible storage for team-level BYOB in Dedicated Cloud or Self-managed instance

Specify the path using the following format:

s3://<accessKey>:<secretAccessKey>@<url_endpoint>/<bucketName>?region=<region>?tls=true

The region parameter is mandatory, except for when your W&B instance is in AWS and the AWS_REGION configured on the W&B instance nodes matches the region configured for the S3-compatible storage.

Configure a cross-cloud native storage for team-level BYOB in Dedicated Cloud or Self-managed instance

Specify the path in a format specific to the locations of your W&B instance and storage bucket:

From W&B instance in GCP or Azure to a bucket in AWS:

s3://<accessKey>:<secretAccessKey>@<s3_regional_url_endpoint>/<bucketName>

From W&B instance in GCP or AWS to a bucket in Azure:

az://:<urlEncodedAccessKey>@<storageAccountName>/<containerName>

From W&B instance in AWS or Azure to a bucket in GCP:

gs://<serviceAccountEmail>:<urlEncodedPrivateKey>@<bucketName>
info

Connectivity to S3-compatible storage for team-level BYOB is not available in SaaS Cloud. Also, connectivity to an AWS bucket for team-level BYOB is considered cross-cloud in SaaS Cloud, as that instance is hosted in GCP. That cross-cloud connectivity doesn't use the access key and environment variable based mechanism as outlined above for Dedicated Cloud and Self-Managed instances.

Reach out to W&B Support at support@wandb.com for more information.

Cloud storage in same cloud as W&B platform​

Based on your use case, configure a storage bucket at the team or instance level. How a storage bucket is provisioned or configured is the same irrespective of the level it's configured at, except for the access mechanism in Azure.

tip

W&B recommends that you use a Terraform module managed by W&B to provision a storage bucket along with the necessary access mechanism and related IAM permissions:

Provision the KMS Key​

W&B requires you to provision a KMS Key which is needed to encrypt and decrypt the data on the S3 bucket. The key usage type must be ENCRYPT_DECRYPT. Assign the following policy to the key:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid" : "Internal",
      "Effect" : "Allow",
      "Principal" : { "AWS" : "<Your_Account_Id>" },
      "Action" : "kms:*",
      "Resource" : "<aws_kms_key.key.arn>"
    },
    {
      "Sid" : "External",
      "Effect" : "Allow",
      "Principal" : { "AWS" : "<aws_principal_and_role_arn>" },
      "Action" : [
        "kms:Decrypt",
        "kms:Describe*",
        "kms:Encrypt",
        "kms:ReEncrypt*",
        "kms:GenerateDataKey*"
      ],
      "Resource" : "<aws_kms_key.key.arn>"
    }
  ]
}

Replace <Your_Account_Id> and <aws_kms_key.key.arn> accordingly.

If you are using SaaS Cloud or Dedicated Cloud, replace <aws_principal_and_role_arn> with the corresponding value:

This policy grants your AWS account full access to the key and also assigns the required permissions to the AWS account hosting the W&B Platform. Keep a record of the KMS Key ARN.

Provision the S3 Bucket​

Follow these steps to provision the S3 bucket in your AWS account:

  • Create the S3 bucket with a name of your choice. Optionally create a folder which you can configure as sub-path to store all W&B files.
  • Enable bucket versioning.
  • Enable server side encryption, using the KMS key from the previous step.
  • Configure CORS with the following policy:
[
    {
        "AllowedHeaders": [
            "*"
        ],
        "AllowedMethods": [
            "GET",
            "HEAD",
            "PUT"
        ],
        "AllowedOrigins": [
            "*"
        ],
        "ExposeHeaders": [
            "ETag"
        ],
        "MaxAgeSeconds": 3600
    }
]
  • Grant the required S3 permissions to the AWS account hosting the W&B Platform. These permissions are used to generate pre-signed URLs that AI workloads in your cloud infrastructure or user browsers utilize to access the bucket.
{
  "Version": "2012-10-17",
  "Id": "WandBAccess",
  "Statement": [
    {
      "Sid": "WAndBAccountAccess",
      "Effect": "Allow",
      "Principal": { "AWS": "<aws_principal_and_role_arn>" },
        "Action" : [
          "s3:GetObject*",
          "s3:GetEncryptionConfiguration",
          "s3:ListBucket",
          "s3:ListBucketMultipartUploads",
          "s3:ListBucketVersions",
          "s3:AbortMultipartUpload",
          "s3:DeleteObject",
          "s3:PutObject",
          "s3:GetBucketCORS",
          "s3:GetBucketLocation",
          "s3:GetBucketVersioning"
        ],
      "Resource": [
        "arn:aws:s3:::<wandb_bucket>",
        "arn:aws:s3:::<wandb_bucket>/*"
      ]
    }
  ]
}

Replace <wandb_bucket> accordingly and keep a record of the bucket name. If you are using Dedicated Cloud, share the bucket name with your W&B team in case of instance level BYOB. In case of team level BYOB on any deployment type, configure the bucket while creating the team.

If you are using SaaS Cloud or Dedicated Cloud, replace <aws_principal_and_role_arn> with the corresponding value.

For more details, see the AWS self-managed hosting guide.

Configure BYOB in W&B​

info

If you're connecting to a cloud-native storage bucket in another cloud or to an S3-compatible storage bucket like MinIO for team-level BYOB in your Dedicated Cloud or Self-Managed instance, refer to Cross-cloud or S3-compatible storage for team-level BYOB. In such cases, you must specify the storage bucket using the GORILLA_SUPPORTED_FILE_STORES environment variable for your W&B instance, before you configure it for a team using the instructions below.

Configure a storage bucket at the team level when you create a W&B Team:

  1. Provide a name for your team in the Team Name field.
  2. Select External storage for the Storage type option.
  3. Choose either New bucket from the dropdown or select an existing bucket.
tip

Multiple W&B Teams can use the same cloud storage bucket. To enable this, select an existing cloud storage bucket from the dropdown.

  1. From the Cloud provider dropdown, select your cloud provider.
  2. Provide the name of your storage bucket for the Name field. If you have a Dedicated Cloud or Self-Managed instance on Azure, provide the values for Account name and Container name fields.
  3. (Optional) Provide the bucket sub-path in the optional Path field. Do this if you would not like W&B to store any files in a folder at the root of the bucket.
  4. (Optional if using AWS bucket) Provide the ARN of your KMS encryption key for the KMS key ARN field.
  5. (Optional if using Azure bucket) Provide the values for the Tenant ID and the Managed Identity Client ID fields.
  6. (Optional on SaaS Cloud) Optionally invite team members when creating the team.
  7. Press the Create Team button.

An error or warning appears at the bottom of the page if there are issues accessing the bucket or the bucket has invalid settings.

On this page
Was this page helpful?👍👎