Databricks

Search…
Databricks
W&B integrates with Databricks by customizing the W&B Jupyter notebook experience in the Databricks environment.
Databricks Configuration
Install wandb in the cluster
Navigate to your cluster configuration, choose your cluster, click on Libraries, then on Install New, Choose PyPI and add the package wandb.
Authentication
In order to authenticate your W&B account you can add a databricks secret which your notebooks can query.
1
# install databricks cli
2
pip install databricks-cli
3
​
4
# Generate a token from databricks UI
5
databricks configure --token
6
​
7
# Create a scope with one of the two commands (depending if you have security features enabled on databricks):
8
# with security add-on
9
databricks secrets create-scope --scope wandb
10
# without security add-on
11
databricks secrets create-scope --scope wandb --initial-manage-principal users
12
​
13
# Add your api_key from: https://app.wandb.ai/authorize
14
databricks secrets put --scope wandb --key api_key
Copied!
Examples
Simple
1
import os
2
import wandb
3
​
4
api_key = dbutils.secrets.get("wandb", "api_key")
5
wandb.login(key=api_key)
6
​
7
wandb.init()
8
wandb.log({"foo": 1})
Copied!
Sweeps
Setup required (temporary) for notebooks attempting to use wandb.sweep() or wandb.agent():
1
import os
2
# These will not be necessary in the future
3
os.environ['WANDB_ENTITY'] = "my-entity"
4
os.environ['WANDB_PROJECT'] = "my-project-that-exists"
Copied!
We cover more details of how to run a sweep in a notebook here:
Running Sweeps in Jupyter
DeepChem
Docker
Last modified 11mo ago
Copy link
Contents
Databricks Configuration
Install wandb in the cluster