This is the multi-page printable view of this section. Click here to print.
W&B integrations make it fast and easy to set up experiment tracking and data versioning inside existing projects. Check out integrations for ML frameworks such as PyTorch, ML libraries such as Hugging Face, or cloud services such as Amazon SageMaker.
This guide provides best practices on how to integrate W&B into your Python library to get powerful Experiment Tracking, GPU and System Monitoring, Model Checkpointing, and more for your own library.
Below we cover best tips and best practices when the codebase you are working on is more complicated than a single Python training script or Jupyter notebook. The topics covered are:
Before you get started, decide whether or not to require W&B in your library’s dependencies:
Add the W&B Python library (wandb) to your dependencies file, for example, in your requirements.txt file:
torch==1.8.0
...
wandb==0.13.*
There are two ways to make the W&B SDK (wandb) optional:
A. Raise an error when a user tries to use wandb functionality without installing it manually and show an appropriate error message:
try:
import wandb
except ImportError:
raise ImportError(
"You are trying to use wandb which is not currently installed."
"Please install it using pip install wandb"
)
B. Add wandb as an optional dependency to your pyproject.toml file, if you are building a Python package:
[project]
name = "my_awesome_lib"
version = "0.1.0"
dependencies = [
"torch",
"sklearn"
]
[project.optional-dependencies]
dev = [
"wandb"
]
There are a few ways for your users to log in to W&B:
Log into W&B with a bash command in a terminal:
wandb login $MY_WANDB_KEY
If they’re in a Jupyter or Colab notebook, log into W&B like so:
import wandb
wandb.login()
Set a W&B environment variable for the API key:
export WANDB_API_KEY=$YOUR_API_KEY
or
os.environ['WANDB_API_KEY'] = "abc123..."
If a user is using wandb for the first time without following any of the steps mentioned above, they will automatically be prompted to log in when your script calls wandb.init.
A W&B Run is a unit of computation logged by W&B. Typically, you associate a single W&B Run per training experiment.
Initialize W&B and start a Run within your code with:
wandb.init()
Optionally, you can provide a name for their project, or let the user set it themselves with parameters such as wandb_project in your code along with the username or team name, such as wandb_entity, for the entity parameter:
wandb.init(project=wandb_project, entity=wandb_entity)
wandb.init?Your library should create W&B Run as early as possible because any output in your console, including error messages, is logged as part of the W&B Run. This makes debugging easier.
wandb As OptionalIf you want to make wandb optional when your users use your library, you can either:
wandb flag such as:trainer = my_trainer(..., use_wandb=True)
python train.py ... --use-wandb
wandb to be disabled in wandb.init:wandb.init(mode="disabled")
export WANDB_MODE=disabled
or
wandb disabled
wandb to be offline - note this will still run wandb, it just won’t try and communicate back to W&B over the internet:export WANDB_MODE=offline
or
os.environ['WANDB_MODE'] = 'offline'
wandb offline
With a wandb run config, you can provide metadata about your model, dataset, and so on when you create a W&B Run. You can use this information to compare different experiments and quickly understand the main differences.
Typical config parameters you can log include:
The following code snippet shows how to log a config:
config = {"batch_size": 32, ...}
wandb.init(..., config=config)
Use wandb.config.update to update the config. Updating your configuration dictionary is useful when parameters are obtained after the dictionary was defined. For example, you might want to add a model’s parameters after the model is instantiated.
wandb.config.update({"model_parameters": 3500})
For more information on how to define a config file, see Configure Experiments with wandb.config.
Create a dictionary where the key value is the name of the metric. Pass this dictionary object to wandb.log:
for epoch in range(NUM_EPOCHS):
for input, ground_truth in data:
prediction = model(input)
loss = loss_fn(prediction, ground_truth)
metrics = { "loss": loss }
wandb.log(metrics)
If you have a lot of metrics, you can have them automatically grouped in the UI by using prefixes in the metric name, such as train/... and val/.... This will create separate sections in your W&B Workspace for your training and validation metrics, or other metric types you’d like to separate:
metrics = {
"train/loss": 0.4,
"train/learning_rate": 0.4,
"val/loss": 0.5,
"val/accuracy": 0.7
}
wandb.log(metrics)
For more on wandb.log, see Log Data with wandb.log.
Sometimes you might need to perform multiple calls to wandb.log for the same training step. The wandb SDK has its own internal step counter that is incremented every time a wandb.log call is made. This means that there is a possibility that the wandb log counter is not aligned with the training step in your training loop.
To avoid this, we recommend that you specifically define your x-axis step. You can define the x-axis with wandb.define_metric and you only need to do this once, after wandb.init is called:
wandb.init(...)
wandb.define_metric("*", step_metric="global_step")
The glob pattern, *, means that every metric will use global_step as the x-axis in your charts. If you only want certain metrics to be logged against global_step, you can specify them instead:
wandb.define_metric("train/loss", step_metric="global_step")
Now that you’ve called wandb.define_metric, you just need to log your metrics as well as your step_metric, global_step, every time you call wandb.log:
for step, (input, ground_truth) in enumerate(data):
...
wandb.log({"global_step": step, "train/loss": 0.1})
wandb.log({"global_step": step, "eval/loss": 0.2})
If you do not have access to the independent step variable, for example “global_step” is not available during your validation loop, the previously logged value for “global_step” is automatically used by wandb. In this case, ensure you log an initial value for the metric so it has been defined when it’s needed.
In addition to metrics, you can log plots, histograms, tables, text, and media such as images, videos, audios, 3D, and more.
Some considerations when logging data include:
Refer to Log Data with wandb.log for a full guide on logging media, objects, plots, and more.
For frameworks supporting distributed environments, you can adapt any of the following workflows:
wandb there. Any required data coming from other processes must be routed to the main process first. (This workflow is encouraged).wandb in every process and auto-group them by giving them all the same unique group name.See Log Distributed Training Experiments for more details.
If your framework uses or produces models or datasets, you can log them for full traceability and have wandb automatically monitor your entire pipeline through W&B Artifacts.
When using Artifacts, it might be useful but not necessary to let your users define:
user/project/artifact.You can log Model Checkpoints to W&B. It is useful to leverage the unique wandb Run ID to name output Model Checkpoints to differentiate them between Runs. You can also add useful metadata. In addition, you can also add aliases to each model as shown below:
metadata = {"eval/accuracy": 0.8, "train/steps": 800}
artifact = wandb.Artifact(
name=f"model-{wandb.run.id}",
metadata=metadata,
type="model"
)
artifact.add_dir("output_model") # local directory where the model weights are stored
aliases = ["best", "epoch_10"]
wandb.log_artifact(artifact, aliases=aliases)
For information on how to create a custom alias, see Create a Custom Alias.
You can log output Artifacts at any frequency (for example, every epoch, every 500 steps, and so on) and they are automatically versioned.
You can log artifacts that are used as inputs to your training such as pre-trained models or datasets. The following snippet demonstrates how to log an Artifact and add it as an input to the ongoing Run as shown in the graph above.
artifact_input_data = wandb.Artifact(name="flowers", type="dataset")
artifact_input_data.add_file("flowers.npy")
wandb.use_artifact(artifact_input_data)
You re-use an Artifact (dataset, model, etc.) and wandb will download a copy locally (and cache it):
artifact = wandb.run.use_artifact("user/project/artifact:latest")
local_path = artifact.download("./tmp")
Artifacts can be found in the Artifacts section of W&B and can be referenced with aliases generated automatically (latest, v2, v3) or manually when logging (best_accuracy, etc.).
To download an Artifact without creating a wandb run (through wandb.init), for example in distributed environments or for simple inference, you can instead reference the artifact with the wandb API:
artifact = wandb.Api().artifact("user/project/artifact:latest")
local_path = artifact.download()
For more information, see Download and Use Artifacts.
If your library would like to leverage W&B hyper-parameter tuning, W&B Sweeps can also be added to your library.
You can also see what an advanced W&B integrations look like in the following integrations. Note most integrations will not be as complex as these:
Fine-tuning GPT-3.5 or GPT-4 models on Microsoft Azure using W&B tracks, analyzes, and improves model performance by automatically capturing metrics and facilitating systematic evaluation through W&B’s experiment tracking and evaluation tools.
During fine-tuning, W&B captures:
After fine-tuning, use W&B Weave to:
Catalyst is a PyTorch framework for deep learning R&D that focuses on reproducibility, rapid experimentation, and codebase reuse so you can create something new.
Catalyst includes a W&B integration for logging parameters, metrics, images, and other artifacts.
Check out their documentation of the integration, which includes examples using Python and Hydra.
Run an example colab to see Catalyst and W&B integration in action.
With Weights & Biases you can log your Cohere model’s fine-tuning metrics and configuration to analyze and understand the performance of your models and share the results with your colleagues.
This guide from Cohere has a full example of how to kick off a fine-tuning run and you can find the Cohere API docs here
To add Cohere fine-tuning logging to your W&B workspace:
Create a WandbConfig with your W&B API key, W&B entity and project name. You can find your W&B API key at https://wandb.ai/authorize
Pass this config to the FinetunedModel object along with your model name, dataset and hyperparameters to kick off your fine-tuning run.
from cohere.finetuning import WandbConfig, FinetunedModel
# create a config with your W&B details
wandb_ft_config = WandbConfig(
api_key="<wandb_api_key>",
entity="my-entity", # must be a valid enitity associated with the provided API key
project="cohere-ft",
)
... # set up your datasets and hyperparameters
# start a fine-tuning run on cohere
cmd_r_finetune = co.finetuning.create_finetuned_model(
request=FinetunedModel(
name="command-r-ft",
settings=Settings(
base_model=...
dataset_id=...
hyperparameters=...
wandb=wandb_ft_config # pass your W&B config here
),
),
)
View your model’s fine-tuning training and validation metrics and hyperparameters in the W&B project that you created.
Your W&B runs are automatically organized and can be filtered/sorted based on any configuration parameter such as job type, base model, learning rate and any other hyper-parameter.
In addition, you can rename your runs, add notes or create tags to group them.
W&B integrates with Databricks by customizing the W&B Jupyter notebook experience in the Databricks environment.
Install wandb in the cluster
Navigate to your cluster configuration, choose your cluster, click Libraries. Click Install New, choose PyPI, and add the package wandb.
Set up authentication
To authenticate your W&B account you can add a Databricks secret which your notebooks can query.
# install databricks cli
pip install databricks-cli
# Generate a token from databricks UI
databricks configure --token
# Create a scope with one of the two commands (depending if you have security features enabled on databricks):
# with security add-on
databricks secrets create-scope --scope wandb
# without security add-on
databricks secrets create-scope --scope wandb --initial-manage-principal users
# Add your api_key from: https://app.wandb.ai/authorize
databricks secrets put --scope wandb --key api_key
import os
import wandb
api_key = dbutils.secrets.get("wandb", "api_key")
wandb.login(key=api_key)
wandb.init()
wandb.log({"foo": 1})
Setup required (temporary) for notebooks attempting to use wandb.sweep() or wandb.agent():
import os
# These will not be necessary in the future
os.environ["WANDB_ENTITY"] = "my-entity"
os.environ["WANDB_PROJECT"] = "my-project-that-exists"
DeepChecks helps you validate your machine learning models and data, such as verifying your data’s integrity, inspecting its distributions, validating data splits, evaluating your model and comparing between different models, all with minimal effort.
Read more about DeepChecks and the wandb integration ->
To use DeepChecks with Weights & Biases you will first need to sign up for a Weights & Biases account here. With the Weights & Biases integration in DeepChecks you can quickly get started like so:
import wandb
wandb.login()
# import your check from deepchecks
from deepchecks.checks import ModelErrorAnalysis
# run your check
result = ModelErrorAnalysis()
# push that result to wandb
result.to_wandb()
You can also log an entire DeepChecks test suite to Weights & Biases
import wandb
wandb.login()
# import your full_suite tests from deepchecks
from deepchecks.suites import full_suite
# create and run a DeepChecks test suite
suite_result = full_suite().run(...)
# push thes results to wandb
# here you can pass any wandb.init configs and arguments you need
suite_result.to_wandb(project="my-suite-project", config={"suite-name": "full-suite"})
``This Report shows off the power of using DeepChecks and Weights & Biases
Any questions or issues about this Weights & Biases integration? Open an issue in the DeepChecks github repository and we’ll catch it and get you an answer :)
The DeepChem library provides open source tools that democratize the use of deep-learning in drug discovery, materials science, chemistry, and biology. This W&B integration adds simple and easy-to-use experiment tracking and model checkpointing while training models using DeepChem.
logger = WandbLogger(…)
model = TorchModel(…, wandb_logger=logger)
model.fit(…)
Explore the Using W&B with DeepChem: Molecular Graph Convolutional Networks article for an example charts generated using the W&B DeepChem integration.
If you’d rather dive straight into working code, check out this Google Colab.
Setup Weights & Biases for DeepChem models of type KerasModel or TorchModel.
Install the wandb library and log in
```
pip install wandb
wandb login
```
```python
!pip install wandb
import wandb
wandb.login()
```
Initialize and configure WandbLogger
from deepchem.models import WandbLogger
logger = WandbLogger(entity="my_entity", project="my_project")
Log your training and evaluation data to W&B
Training loss and evaluation metrics can be automatically logged to Weights & Biases. Optional evaluation can be enabled using the DeepChem ValidationCallback, the WandbLogger will detect ValidationCallback callback and log the metrics generated.
```python
from deepchem.models import TorchModel, ValidationCallback
vc = ValidationCallback(…) # optional
model = TorchModel(…, wandb_logger=logger)
model.fit(…, callbacks=[vc])
logger.finish()
```
```python
from deepchem.models import KerasModel, ValidationCallback
vc = ValidationCallback(…) # optional
model = KerasModel(…, wandb_logger=logger)
model.fit(…, callbacks=[vc])
logger.finish()
```
W&B can store a pointer to the Docker image that your code ran in, giving you the ability to restore a previous experiment to the exact environment it was run in. The wandb library looks for the WANDB_DOCKER environment variable to persist this state. We provide a few helpers that automatically set this state.
wandb docker is a command that starts a docker container, passes in wandb environment variables, mounts your code, and ensures wandb is installed. By default the command uses a docker image with TensorFlow, PyTorch, Keras, and Jupyter installed. You can use the same command to start your own docker image: wandb docker my/image:latest. The command mounts the current directory into the “/app” directory of the container, you can change this with the “–dir” flag.
The wandb docker-run command is provided for production workloads. It’s meant to be a drop in replacement for nvidia-docker. It’s a simple wrapper to the docker run command that adds your credentials and the WANDB_DOCKER environment variable to the call. If you do not pass the “–runtime” flag and nvidia-docker is available on the machine, this also ensures the runtime is set to nvidia.
If you run your training workloads in Kubernetes and the k8s API is exposed to your pod (which is the case by default). wandb will query the API for the digest of the docker image and automatically set the WANDB_DOCKER environment variable.
If a run was instrumented with the WANDB_DOCKER environment variable, calling wandb restore username/project:run_id will checkout a new branch restoring your code then launch the exact docker image used for training pre-populated with the original command.
If you’re using Farama Gymnasium we will automatically log videos of your environment generated by gymnasium.wrappers.Monitor. Just set the monitor_gym keyword argument to wandb.init to True.
Our gymnasium integration is very light. We simply look at the name of the video file being logged from gymnasium and name it after that or fall back to "videos" if we don’t find a match. If you want more control, you can always just manually log a video.
Check out this report to learn more on how to use Gymnasium with the CleanRL library.
If you’re using fastai to train your models, W&B has an easy integration using the WandbCallback. Explore the details in interactive docs with examples →
Sign up for a free account at https://wandb.ai/site and then log in to your wandb account.
Install the wandb library on your machine in a Python 3 environment using pip
log in to the wandb library on your machine.
Find your API key https://wandb.ai/authorize.
pip install wandb
wandb login
!pip install wandb
import wandb
wandb.login()
Add the WandbCallback to the learner or fit method:
import wandb
from fastai.callback.wandb import *
# start logging a wandb run
wandb.init(project="my_project")
# To log only during one training phase
learn.fit(..., cbs=WandbCallback())
# To log continuously for all training phases
learn = learner(..., cbs=WandbCallback())
WandbCallback accepts the following arguments:
| Args | Description |
|---|---|
| log | Whether to log the model’s: gradients , parameters, all or None (default). Losses & metrics are always logged. |
| log_preds | whether we want to log prediction samples (default to True). |
| log_preds_every_epoch | whether to log predictions every epoch or at the end (default to False) |
| log_model | whether we want to log our model (default to False). This also requires SaveModelCallback |
| model_name | The name of the file to save, overrides SaveModelCallback |
| log_dataset |
Note: subfolder “models” is always ignored. |
| dataset_name | name of logged dataset (default to folder name). |
| valid_dl | DataLoaders containing items used for prediction samples (default to random items from learn.dls.valid. |
| n_preds | number of logged predictions (default to 36). |
| seed | used for defining random samples. |
For custom workflows, you can manually log your datasets and models:
log_dataset(path, name=None, metadata={})log_model(path, name=None, metadata={})Note: any subfolder “models” will be ignored.
fastai supports distributed training by using the context manager distrib_ctx. W&B supports this automatically and enables you to track your Multi-GPU experiments out of the box.
Review this minimal example:
import wandb
from fastai.vision.all import *
from fastai.distributed import *
from fastai.callback.wandb import WandbCallback
wandb.require(experiment="service")
path = rank0_first(lambda: untar_data(URLs.PETS) / "images")
def train():
dls = ImageDataLoaders.from_name_func(
path,
get_image_files(path),
valid_pct=0.2,
label_func=lambda x: x[0].isupper(),
item_tfms=Resize(224),
)
wandb.init("fastai_ddp", entity="capecape")
cb = WandbCallback()
learn = vision_learner(dls, resnet34, metrics=error_rate, cbs=cb).to_fp16()
with learn.distrib_ctx(sync_bn=False):
learn.fit(1)
if __name__ == "__main__":
train()
Then, in your terminal you will execute:
$ torchrun --nproc_per_node 2 train.py
in this case, the machine has 2 GPUs.
You can now run distributed training directly inside a notebook.
import wandb
from fastai.vision.all import *
from accelerate import notebook_launcher
from fastai.distributed import *
from fastai.callback.wandb import WandbCallback
wandb.require(experiment="service")
path = untar_data(URLs.PETS) / "images"
def train():
dls = ImageDataLoaders.from_name_func(
path,
get_image_files(path),
valid_pct=0.2,
label_func=lambda x: x[0].isupper(),
item_tfms=Resize(224),
)
wandb.init("fastai_ddp", entity="capecape")
cb = WandbCallback()
learn = vision_learner(dls, resnet34, metrics=error_rate, cbs=cb).to_fp16()
with learn.distrib_ctx(in_notebook=True, sync_bn=False):
learn.fit(1)
notebook_launcher(train, num_processes=2)
In the examples above, wandb launches one run per process. At the end of the training, you will end up with two runs. This can sometimes be confusing, and you may want to log only on the main process. To do so, you will have to detect in which process you are manually and avoid creating runs (calling wandb.init in all other processes)
import wandb
from fastai.vision.all import *
from fastai.distributed import *
from fastai.callback.wandb import WandbCallback
wandb.require(experiment="service")
path = rank0_first(lambda: untar_data(URLs.PETS) / "images")
def train():
cb = []
dls = ImageDataLoaders.from_name_func(
path,
get_image_files(path),
valid_pct=0.2,
label_func=lambda x: x[0].isupper(),
item_tfms=Resize(224),
)
if rank_distrib() == 0:
run = wandb.init("fastai_ddp", entity="capecape")
cb = WandbCallback()
learn = vision_learner(dls, resnet34, metrics=error_rate, cbs=cb).to_fp16()
with learn.distrib_ctx(sync_bn=False):
learn.fit(1)
if __name__ == "__main__":
train()
in your terminal call:
$ torchrun --nproc_per_node 2 train.py
import wandb
from fastai.vision.all import *
from accelerate import notebook_launcher
from fastai.distributed import *
from fastai.callback.wandb import WandbCallback
wandb.require(experiment="service")
path = untar_data(URLs.PETS) / "images"
def train():
cb = []
dls = ImageDataLoaders.from_name_func(
path,
get_image_files(path),
valid_pct=0.2,
label_func=lambda x: x[0].isupper(),
item_tfms=Resize(224),
)
if rank_distrib() == 0:
run = wandb.init("fastai_ddp", entity="capecape")
cb = WandbCallback()
learn = vision_learner(dls, resnet34, metrics=error_rate, cbs=cb).to_fp16()
with learn.distrib_ctx(in_notebook=True, sync_bn=False):
learn.fit(1)
notebook_launcher(train, num_processes=2)
For scripts using fastai v1, we have a callback that can automatically log model topology, losses, metrics, weights, gradients, sample predictions and best trained model.
import wandb
from wandb.fastai import WandbCallback
wandb.init()
learn = cnn_learner(data, model, callback_fns=WandbCallback)
learn.fit(epochs)
Requested logged data is configurable through the callback constructor.
from functools import partial
learn = cnn_learner(
data, model, callback_fns=partial(WandbCallback, input_type="images")
)
It is also possible to use WandbCallback only when starting training. In this case it must be instantiated.
learn.fit(epochs, callbacks=WandbCallback(learn))
Custom parameters can also be given at that stage.
learn.fit(epochs, callbacks=WandbCallback(learn, input_type="images"))
We’ve created a few examples for you to see how the integration works:
Fastai v1
WandbCallback() class supports a number of options:
| Keyword argument | Default | Description |
|---|---|---|
| learn | N/A | the fast.ai learner to hook. |
| save_model | True | save the model if it’s improved at each step. It will also load best model at the end of training. |
| mode | auto | min, max, or auto: How to compare the training metric specified in monitor between steps. |
| monitor | None | training metric used to measure performance for saving the best model. None defaults to validation loss. |
| log | gradients | gradients, parameters, all, or None. Losses & metrics are always logged. |
| input_type | None | images or None. Used to display sample predictions. |
| validation_data | None | data used for sample predictions if input_type is set. |
| predictions | 36 | number of predictions to make if input_type is set and validation_data is None. |
| seed | 12345 | initialize random generator for sample predictions if input_type is set and validation_data is None. |
The Hugging Face Transformers library makes state-of-the-art NLP models like BERT and training techniques like mixed precision and gradient checkpointing easy to use. The W&B integration adds rich, flexible experiment tracking and model versioning to interactive centralized dashboards without compromising that ease of use.
os.environ["WANDB_PROJECT"] = "<my-amazing-project>" # name your W&B project
os.environ["WANDB_LOG_MODEL"] = "checkpoint" # log all model checkpoints
from transformers import TrainingArguments, Trainer
args = TrainingArguments(..., report_to="wandb") # turn on W&B logging
trainer = Trainer(..., args=args)
wandb library and log inSign up for a free account
Pip install the wandb library
To log in with your training script, you’ll need to sign in to you account at www.wandb.ai, then you will find your API key on the Authorize page.
If you are using Weights and Biases for the first time you might want to check out our quickstart
pip install wandb
wandb login
!pip install wandb
import wandb
wandb.login()
A W&B Project is where all of the charts, data, and models logged from related runs are stored. Naming your project helps you organize your work and keep all the information about a single project in one place.
To add a run to a project simply set the WANDB_PROJECT environment variable to the name of your project. The WandbCallback will pick up this project name environment variable and use it when setting up your run.
WANDB_PROJECT=amazon_sentiment_analysis
%env WANDB_PROJECT=amazon_sentiment_analysis
import os
os.environ["WANDB_PROJECT"]="amazon_sentiment_analysis"
Trainer.If a project name is not specified the project name defaults to huggingface.
This is the most important step when defining your Trainer training arguments, either inside your code or from the command line, is to set report_to to "wandb" in order enable logging with Weights & Biases.
The logging_steps argument in TrainingArguments will control how often training metrics are pushed to W&B during training. You can also give a name to the training run in W&B using the run_name argument.
That’s it. Now your models will log losses, evaluation metrics, model topology, and gradients to Weights & Biases while they train.
python run_glue.py \ # run your Python script
--report_to wandb \ # enable logging to W&B
--run_name bert-base-high-lr \ # name of the W&B run (optional)
# other command line arguments here
from transformers import TrainingArguments, Trainer
args = TrainingArguments(
# other args and kwargs here
report_to="wandb", # enable logging to W&B
run_name="bert-base-high-lr", # name of the W&B run (optional)
logging_steps=1, # how often to log to W&B
)
trainer = Trainer(
# other args and kwargs here
args=args, # your training args
)
trainer.train() # start training and logging to W&B
Trainer for the TensorFlow TFTrainer.Using Weights & Biases’ Artifacts, you can store up to 100GB of models and datasets for free and then use the Weights & Biases Model Registry to register models to prepare them for staging or deployment in your production environment.
Logging your Hugging Face model checkpoints to Artifacts can be done by setting the WANDB_LOG_MODEL environment variable to one of end or checkpoint or false:
checkpoint: a checkpoint will be uploaded every args.save_steps from the TrainingArguments.end: the model will be uploaded at the end of training.Use WANDB_LOG_MODEL along with load_best_model_at_end to upload the best model at the end of training.
import os
os.environ["WANDB_LOG_MODEL"] = "checkpoint"
WANDB_LOG_MODEL="checkpoint"
%env WANDB_LOG_MODEL="checkpoint"
Any Transformers Trainer you initialize from now on will upload models to your W&B project. The model checkpoints you log will be viewable through the Artifacts UI, and include the full model lineage (see an example model checkpoint in the UI here).
model-{run_id} when WANDB_LOG_MODEL is set to end or checkpoint-{run_id} when WANDB_LOG_MODEL is set to checkpoint.
However, If you pass a run_name in your TrainingArguments, the model will be saved as model-{run_name} or checkpoint-{run_name}.Once you have logged your checkpoints to Artifacts, you can then register your best model checkpoints and centralize them across your team using the Weights & Biases Model Registry. Here you can organize your best models by task, manage model lifecycle, facilitate easy tracking and auditing throughout the ML lifecyle, and automate downstream actions with webhooks or jobs.
See the Model Registry documentation for how to link a model Artifact to the Model Registry.
Visualing your model outputs during training or evaluation is often essential to really understand how your model is training.
By using the callbacks system in the Transformers Trainer, you can log additional helpful data to W&B such as your models’ text generation outputs or other predictions to W&B Tables.
See the Custom logging section below for a full guide on how to log evaluation outupts while training to log to a W&B Table like this:
If your training is encapsulated in a Python script, the W&B run will end when your script finishes.
If you are using a Jupyter or Google Colab notebook, you’ll need to tell us when you’re done with training by calling wandb.finish().
trainer.train() # start training and logging to W&B
# post-training analysis, testing, other logged code
wandb.finish()
Once you have logged your training results you can explore your results dynamically in the W&B Dashboard. It’s easy to compare across dozens of runs at once, zoom in on interesting findings, and coax insights out of complex data with flexible, interactive visualizations.
If load_best_model_at_end=True is set in the TrainingArguments that are passed to the Trainer, then W&B will save the best performing model checkpoint to Artifacts.
If you’d like to centralize all your best model versions across your team to organize them by ML task, stage them for production, bookmark them for further evaluation, or kick off downstream Model CI/CD processes then ensure you’re saving your model checkpoints to Artifacts. Once logged to Artifacts, these checkpoints can then be promoted to the Model Registry.
If you saved your model to W&B Artifacts with WANDB_LOG_MODEL, you can download your model weights for additional training or to run inference. You just load them back into the same Hugging Face architecture that you used before.
# Create a new run
with wandb.init(project="amazon_sentiment_analysis") as run:
# Pass the name and version of Artifact
my_model_name = "model-bert-base-high-lr:latest"
my_model_artifact = run.use_artifact(my_model_name)
# Download model weights to a folder and return the path
model_dir = my_model_artifact.download()
# Load your Hugging Face model from that folder
# using the same model class
model = AutoModelForSequenceClassification.from_pretrained(
model_dir, num_labels=num_labels
)
# Do additional training, or run inference
If you had set WANDB_LOG_MODEL='checkpoint' you can also resume training by you can using the model_dir as the model_name_or_path argument in your TrainingArguments and pass resume_from_checkpoint=True to Trainer.
last_run_id = "xxxxxxxx" # fetch the run_id from your wandb workspace
# resume the wandb run from the run_id
with wandb.init(
project=os.environ["WANDB_PROJECT"],
id=last_run_id,
resume="must",
) as run:
# Connect an Artifact to the run
my_checkpoint_name = f"checkpoint-{last_run_id}:latest"
my_checkpoint_artifact = run.use_artifact(my_model_name)
# Download checkpoint to a folder and return the path
checkpoint_dir = my_checkpoint_artifact.download()
# reinitialize your model and trainer
model = AutoModelForSequenceClassification.from_pretrained(
"<model_name>", num_labels=num_labels
)
# your awesome training arguments here.
training_args = TrainingArguments()
trainer = Trainer(model=model, args=training_args)
# make sure use the checkpoint dir to resume training from the checkpoint
trainer.train(resume_from_checkpoint=checkpoint_dir)
Logging to Weights & Biases via the Transformers Trainer is taken care of by the WandbCallback in the Transformers library. If you need to customize your Hugging Face logging you can modify this callback by subclassing WandbCallback and adding additional functionality that leverages additional methods from the Trainer class.
Below is the general pattern to add this new callback to the HF Trainer, and further down is a code-complete example to log evaluation outputs to a W&B Table:
# Instantiate the Trainer as normal
trainer = Trainer()
# Instantiate the new logging callback, passing it the Trainer object
evals_callback = WandbEvalsCallback(trainer, tokenizer, ...)
# Add the callback to the Trainer
trainer.add_callback(evals_callback)
# Begin Trainer training as normal
trainer.train()
The following section shows how to customize the WandbCallback to run model predictions and log evaluation samples to a W&B Table during training. We will every eval_steps using the on_evaluate method of the Trainer callback.
Here, we wrote a decode_predictions function to decode the predictions and labels from the model output using the tokenizer.
Then, we create a pandas DataFrame from the predictions and labels and add an epoch column to the DataFrame.
Finally, we create a wandb.Table from the DataFrame and log it to wandb.
Additionally, we can control the frequency of logging by logging the predictions every freq epochs.
Note: Unlike the regular WandbCallback this custom callback needs to be added to the trainer after the Trainer is instantiated and not during initialization of the Trainer.
This is because the Trainer instance is passed to the callback during initialization.
from transformers.integrations import WandbCallback
import pandas as pd
def decode_predictions(tokenizer, predictions):
labels = tokenizer.batch_decode(predictions.label_ids)
logits = predictions.predictions.argmax(axis=-1)
prediction_text = tokenizer.batch_decode(logits)
return {"labels": labels, "predictions": prediction_text}
class WandbPredictionProgressCallback(WandbCallback):
"""Custom WandbCallback to log model predictions during training.
This callback logs model predictions and labels to a wandb.Table at each
logging step during training. It allows to visualize the
model predictions as the training progresses.
Attributes:
trainer (Trainer): The Hugging Face Trainer instance.
tokenizer (AutoTokenizer): The tokenizer associated with the model.
sample_dataset (Dataset): A subset of the validation dataset
for generating predictions.
num_samples (int, optional): Number of samples to select from
the validation dataset for generating predictions. Defaults to 100.
freq (int, optional): Frequency of logging. Defaults to 2.
"""
def __init__(self, trainer, tokenizer, val_dataset, num_samples=100, freq=2):
"""Initializes the WandbPredictionProgressCallback instance.
Args:
trainer (Trainer): The Hugging Face Trainer instance.
tokenizer (AutoTokenizer): The tokenizer associated
with the model.
val_dataset (Dataset): The validation dataset.
num_samples (int, optional): Number of samples to select from
the validation dataset for generating predictions.
Defaults to 100.
freq (int, optional): Frequency of logging. Defaults to 2.
"""
super().__init__()
self.trainer = trainer
self.tokenizer = tokenizer
self.sample_dataset = val_dataset.select(range(num_samples))
self.freq = freq
def on_evaluate(self, args, state, control, **kwargs):
super().on_evaluate(args, state, control, **kwargs)
# control the frequency of logging by logging the predictions
# every `freq` epochs
if state.epoch % self.freq == 0:
# generate predictions
predictions = self.trainer.predict(self.sample_dataset)
# decode predictions and labels
predictions = decode_predictions(self.tokenizer, predictions)
# add predictions to a wandb.Table
predictions_df = pd.DataFrame(predictions)
predictions_df["epoch"] = state.epoch
records_table = self._wandb.Table(dataframe=predictions_df)
# log the table to wandb
self._wandb.log({"sample_predictions": records_table})
# First, instantiate the Trainer
trainer = Trainer(
model=model,
args=training_args,
train_dataset=lm_datasets["train"],
eval_dataset=lm_datasets["validation"],
)
# Instantiate the WandbPredictionProgressCallback
progress_callback = WandbPredictionProgressCallback(
trainer=trainer,
tokenizer=tokenizer,
val_dataset=lm_dataset["validation"],
num_samples=10,
freq=2,
)
# Add the callback to the trainer
trainer.add_callback(progress_callback)
For a more detailed example please refer to this colab
Further configuration of what is logged with Trainer is possible by setting environment variables. A full list of W&B environment variables can be found here.
| Environment Variable | Usage |
|---|---|
WANDB_PROJECT |
Give your project a name (huggingface by default) |
WANDB_LOG_MODEL |
Log the model checkpoint as a W&B Artifact (
|
WANDB_WATCH |
Set whether you’d like to log your models gradients, parameters or neither
|
WANDB_DISABLED |
Set to true to turn off logging entirely (false by default) |
WANDB_SILENT |
Set to true to silence the output printed by wandb (false by default) |
WANDB_WATCH=all
WANDB_SILENT=true
%env WANDB_WATCH=all
%env WANDB_SILENT=true
wandb.init?The WandbCallback that Trainer uses will call wandb.init under the hood when Trainer is initialized. You can alternatively set up your runs manually by calling wandb.init before theTrainer is initialized. This gives you full control over your W&B run configuration.
An example of what you might want to pass to init is below. For more details on how to use wandb.init, check out the reference documentation.
wandb.init(
project="amazon_sentiment_analysis",
name="bert-base-high-lr",
tags=["baseline", "high-lr"],
group="bert",
)
Below are 6 Transformers and W&B related articles you might enjoy
Read the full report here.
Read the full report here.
Read the full report here.
Read the full report here.
tf.keras.callbacks.EarlyStoppingcallback.Read the full report here.
We fine tune a DistilBERT transformer for sentiment analysis (binary classification) on a custom IMDB dataset.
Read the full report here.
For any issues, questions, or feature requests for the Hugging Face W&B integration, feel free to post in this thread on the Hugging Face forums or open an issue on the Hugging Face Transformers GitHub repo.
Hugging Face Diffusers is the go-to library for state-of-the-art pre-trained diffusion models for generating images, audio, and even 3D structures of molecules. The W&B integration adds rich, flexible experiment tracking, media visualization, pipeline architecture, and configuration management to interactive centralized dashboards without compromising that ease of use.
Log all the prompts, negative prompts, generated media, and configs associated with your experiment by simply including 2 lines of code. Here are the 2 lines of code to begin logging:
# import the autolog function
from wandb.integration.diffusers import autolog
# call the autolog before calling the pipeline
autolog(init=dict(project="diffusers_logging"))
 |
|---|
| An example of how the results of your experiment are logged. |
Install diffusers, transformers, accelerate, and wandb.
Command line:
pip install --upgrade diffusers transformers accelerate wandb
Notebook:
!pip install --upgrade diffusers transformers accelerate wandb
Use autolog to initialize a Weights & Biases run and automatically track the inputs and the outputs from all supported pipeline calls.
You can call the autolog() function with the init parameter, which accepts a dictionary of parameters required by wandb.init().
When you call autolog(), it initializes a Weights & Biases run and automatically tracks the inputs and the outputs from all supported pipeline calls.
wandb.Table.You can find a list of supported pipeline calls [here](https://github.com/wandb/wandb/blob/main/wandb/integration/diffusers/autologger.py#L12-L72). In case, you want to request a new feature of this integration or report a bug associated with it, please open an issue on [https://github.com/wandb/wandb/issues](https://github.com/wandb/wandb/issues).
Here is a brief end-to-end example of the autolog in action:
import torch
from diffusers import DiffusionPipeline
# import the autolog function
from wandb.integration.diffusers import autolog
# call the autolog before calling the pipeline
autolog(init=dict(project="diffusers_logging"))
# Initialize the diffusion pipeline
pipeline = DiffusionPipeline.from_pretrained(
"stabilityai/stable-diffusion-2-1", torch_dtype=torch.float16
).to("cuda")
# Define the prompts, negative prompts, and seed.
prompt = ["a photograph of an astronaut riding a horse", "a photograph of a dragon"]
negative_prompt = ["ugly, deformed", "ugly, deformed"]
generator = torch.Generator(device="cpu").manual_seed(10)
# call the pipeline to generate the images
images = pipeline(
prompt,
negative_prompt=negative_prompt,
num_images_per_prompt=2,
generator=generator,
)
import torch
from diffusers import DiffusionPipeline
import wandb
# import the autolog function
from wandb.integration.diffusers import autolog
# call the autolog before calling the pipeline
autolog(init=dict(project="diffusers_logging"))
# Initialize the diffusion pipeline
pipeline = DiffusionPipeline.from_pretrained(
"stabilityai/stable-diffusion-2-1", torch_dtype=torch.float16
).to("cuda")
# Define the prompts, negative prompts, and seed.
prompt = ["a photograph of an astronaut riding a horse", "a photograph of a dragon"]
negative_prompt = ["ugly, deformed", "ugly, deformed"]
generator = torch.Generator(device="cpu").manual_seed(10)
# call the pipeline to generate the images
images = pipeline(
prompt,
negative_prompt=negative_prompt,
num_images_per_prompt=2,
generator=generator,
)
# Finish the experiment
wandb.finish()
The results of a single experiment:
The results of multiple experiments:
The config of an experiment:
wandb.finish() when executing the code in IPython notebook environments after calling the pipeline. This is not necessary when executing python scripts.This section demonstrates the autolog with a typical Stable Diffusion XL + Refiner workflow, in which the latents generated by the StableDiffusionXLPipeline is refined by the corresponding refiner.
import torch
from diffusers import StableDiffusionXLImg2ImgPipeline, StableDiffusionXLPipeline
from wandb.integration.diffusers import autolog
# initialize the SDXL base pipeline
base_pipeline = StableDiffusionXLPipeline.from_pretrained(
"stabilityai/stable-diffusion-xl-base-1.0",
torch_dtype=torch.float16,
variant="fp16",
use_safetensors=True,
)
base_pipeline.enable_model_cpu_offload()
# initialize the SDXL refiner pipeline
refiner_pipeline = StableDiffusionXLImg2ImgPipeline.from_pretrained(
"stabilityai/stable-diffusion-xl-refiner-1.0",
text_encoder_2=base_pipeline.text_encoder_2,
vae=base_pipeline.vae,
torch_dtype=torch.float16,
use_safetensors=True,
variant="fp16",
)
refiner_pipeline.enable_model_cpu_offload()
prompt = "a photo of an astronaut riding a horse on mars"
negative_prompt = "static, frame, painting, illustration, sd character, low quality, low resolution, greyscale, monochrome, nose, cropped, lowres, jpeg artifacts, deformed iris, deformed pupils, bad eyes, semi-realistic worst quality, bad lips, deformed mouth, deformed face, deformed fingers, deformed toes standing still, posing"
# Make the experiment reproducible by controlling randomness.
# The seed would be automatically logged to WandB.
seed = 42
generator_base = torch.Generator(device="cuda").manual_seed(seed)
generator_refiner = torch.Generator(device="cuda").manual_seed(seed)
# Call WandB Autolog for Diffusers. This would automatically log
# the prompts, generated images, pipeline architecture and all
# associated experiment configs to Weights & Biases, thus making your
# image generation experiments easy to reproduce, share and analyze.
autolog(init=dict(project="sdxl"))
# Call the base pipeline to generate the latents
image = base_pipeline(
prompt=prompt,
negative_prompt=negative_prompt,
output_type="latent",
generator=generator_base,
).images[0]
# Call the refiner pipeline to generate the refined image
image = refiner_pipeline(
prompt=prompt,
negative_prompt=negative_prompt,
image=image[None, :],
generator=generator_refiner,
).images[0]
import torch
from diffusers import StableDiffusionXLImg2ImgPipeline, StableDiffusionXLPipeline
import wandb
from wandb.integration.diffusers import autolog
# initialize the SDXL base pipeline
base_pipeline = StableDiffusionXLPipeline.from_pretrained(
"stabilityai/stable-diffusion-xl-base-1.0",
torch_dtype=torch.float16,
variant="fp16",
use_safetensors=True,
)
base_pipeline.enable_model_cpu_offload()
# initialize the SDXL refiner pipeline
refiner_pipeline = StableDiffusionXLImg2ImgPipeline.from_pretrained(
"stabilityai/stable-diffusion-xl-refiner-1.0",
text_encoder_2=base_pipeline.text_encoder_2,
vae=base_pipeline.vae,
torch_dtype=torch.float16,
use_safetensors=True,
variant="fp16",
)
refiner_pipeline.enable_model_cpu_offload()
prompt = "a photo of an astronaut riding a horse on mars"
negative_prompt = "static, frame, painting, illustration, sd character, low quality, low resolution, greyscale, monochrome, nose, cropped, lowres, jpeg artifacts, deformed iris, deformed pupils, bad eyes, semi-realistic worst quality, bad lips, deformed mouth, deformed face, deformed fingers, deformed toes standing still, posing"
# Make the experiment reproducible by controlling randomness.
# The seed would be automatically logged to WandB.
seed = 42
generator_base = torch.Generator(device="cuda").manual_seed(seed)
generator_refiner = torch.Generator(device="cuda").manual_seed(seed)
# Call WandB Autolog for Diffusers. This would automatically log
# the prompts, generated images, pipeline architecture and all
# associated experiment configs to Weights & Biases, thus making your
# image generation experiments easy to reproduce, share and analyze.
autolog(init=dict(project="sdxl"))
# Call the base pipeline to generate the latents
image = base_pipeline(
prompt=prompt,
negative_prompt=negative_prompt,
output_type="latent",
generator=generator_base,
).images[0]
# Call the refiner pipeline to generate the refined image
image = refiner_pipeline(
prompt=prompt,
negative_prompt=negative_prompt,
image=image[None, :],
generator=generator_refiner,
).images[0]
# Finish the experiment
wandb.finish()
Hugging Face AutoTrain is a no-code tool for training state-of-the-art models for Natural Language Processing (NLP) tasks, for Computer Vision (CV) tasks, and for Speech tasks and even for Tabular tasks.
Weights & Biases is directly integrated into Hugging Face AutoTrain, providing experiment tracking and config management. It’s as easy as using a single parameter in the CLI command for your experiments
Install autotrain-advanced and wandb.
pip install --upgrade autotrain-advanced wandb
!pip install --upgrade autotrain-advanced wandb
To demonstrate these changes, this page fine-tines an LLM on a math dataset to achieve SoTA result in pass@1 on the GSM8k Benchmarks.
Hugging Face AutoTrain expects your CSV custom dataset to have a specific format to work properly.
Your training file must contain a text column, which the training uses. For best results, the text column’s data must conform to the ### Human: Question?### Assistant: Answer. format. Review a great example in timdettmers/openassistant-guanaco.
However, the MetaMathQA dataset includes the columns query, response, and type. First, pre-process this dataset. Remove the type column and combine the content of the query and response columns into a new text column in the ### Human: Query?### Assistant: Response. format. Training uses the resulting dataset, rishiraj/guanaco-style-metamath.
autotrainYou can start training using the autotrain advanced from the command line or a notebook. Use the --log argument, or use --log wandb to log your results to a W&B run.
autotrain llm \
--train \
--model HuggingFaceH4/zephyr-7b-alpha \
--project-name zephyr-math \
--log wandb \
--data-path data/ \
--text-column text \
--lr 2e-5 \
--batch-size 4 \
--epochs 3 \
--block-size 1024 \
--warmup-ratio 0.03 \
--lora-r 16 \
--lora-alpha 32 \
--lora-dropout 0.05 \
--weight-decay 0.0 \
--gradient-accumulation 4 \
--logging_steps 10 \
--fp16 \
--use-peft \
--use-int4 \
--merge-adapter \
--push-to-hub \
--token <huggingface-token> \
--repo-id <huggingface-repository-address>
# Set hyperparameters
learning_rate = 2e-5
num_epochs = 3
batch_size = 4
block_size = 1024
trainer = "sft"
warmup_ratio = 0.03
weight_decay = 0.
gradient_accumulation = 4
lora_r = 16
lora_alpha = 32
lora_dropout = 0.05
logging_steps = 10
# Run training
!autotrain llm \
--train \
--model "HuggingFaceH4/zephyr-7b-alpha" \
--project-name "zephyr-math" \
--log "wandb" \
--data-path data/ \
--text-column text \
--lr str(learning_rate) \
--batch-size str(batch_size) \
--epochs str(num_epochs) \
--block-size str(block_size) \
--warmup-ratio str(warmup_ratio) \
--lora-r str(lora_r) \
--lora-alpha str(lora_alpha) \
--lora-dropout str(lora_dropout) \
--weight-decay str(weight_decay) \
--gradient-accumulation str(gradient_accumulation) \
--logging-steps str(logging_steps) \
--fp16 \
--use-peft \
--use-int4 \
--merge-adapter \
--push-to-hub \
--token str(hf_token) \
--repo-id "rishiraj/zephyr-math"
Hugging Face Accelerate is a library that enables the same PyTorch code to run across any distributed configuration, to simplify model training and inference at scale.
Accelerate includes a Weights & Biases Tracker which we show how to use below. You can also read more about Accelerate Trackers in their docs here
To get started with Accelerate and Weights & Biases you can follow the pseudocode below:
from accelerate import Accelerator
# Tell the Accelerator object to log with wandb
accelerator = Accelerator(log_with="wandb")
# Initialise your wandb run, passing wandb parameters and any config information
accelerator.init_trackers(
project_name="my_project",
config={"dropout": 0.1, "learning_rate": 1e-2}
init_kwargs={"wandb": {"entity": "my-wandb-team"}}
)
...
# Log to wandb by calling `accelerator.log`, `step` is optional
accelerator.log({"train_loss": 1.12, "valid_loss": 0.8}, step=global_step)
# Make sure that the wandb tracker finishes correctly
accelerator.end_training()
Explaining more, you need to:
log_with="wandb" when initialising the Accelerator classinit_trackers method and pass it:project_namewandb.init via a nested dict to init_kwargsconfig.log method to log to Weigths & Biases; the step argument is optional.end_training when finished trainingTo access the W&B tracker, use the Accelerator.get_tracker() method. Pass in the string corresponding to a tracker’s .name attribute, which returns the tracker on the main process.
wandb_tracker = accelerator.get_tracker("wandb")
From there you can interact with wandb’s run object like normal:
wandb_tracker.log_artifact(some_artifact_to_log)
Trackers built in Accelerate will automatically execute on the correct process, so if a tracker is only meant to be ran on the main process it will do so automatically.
If you want to truly remove Accelerate’s wrapping entirely, you can achieve the same outcome with:
wandb_tracker = accelerator.get_tracker("wandb", unwrap=True)
with accelerator.on_main_process:
wandb_tracker.log_artifact(some_artifact_to_log)
Below is an Accelerate article you may enjoy
Read the full report here.
Hydra is an open-source Python framework that simplifies the development of research and other complex applications. The key feature is the ability to dynamically create a hierarchical configuration by composition and override it through config files and the command line.
You can continue to use Hydra for configuration management while taking advantage of the power of W&B.
Track your metrics as normal with wandb.init and wandb.log . Here, wandb.entity and wandb.project are defined within a hydra configuration file.
import wandb
@hydra.main(config_path="configs/", config_name="defaults")
def run_experiment(cfg):
run = wandb.init(entity=cfg.wandb.entity, project=cfg.wandb.project)
wandb.log({"loss": loss})
Hydra uses omegaconf as the default way to interface with configuration dictionaries. OmegaConf’s dictionary are not a subclass of primitive dictionaries so directly passing Hydra’s Config to wandb.config leads to unexpected results on the dashboard. It’s necessary to convert omegaconf.DictConfig to the primitive dict type before passing to wandb.config.
@hydra.main(config_path="configs/", config_name="defaults")
def run_experiment(cfg):
wandb.config = omegaconf.OmegaConf.to_container(
cfg, resolve=True, throw_on_missing=True
)
wandb.init(entity=cfg.wandb.entity, project=cfg.wandb.project)
wandb.log({"loss": loss})
model = Model(**wandb.config.model.configs)
If your process hangs when started, this may be caused by this known issue. To solve this, try to changing wandb’s multiprocessing protocol either by adding an extra settings parameter to `wandb.init` as:
wandb.init(settings=wandb.Settings(start_method="thread"))
or by setting a global environment variable from your shell:
$ export WANDB_START_METHOD=thread
W&B Sweeps is a highly scalable hyperparameter search platform, which provides interesting insights and visualization about W&B experiments with minimal requirements code real-estate. Sweeps integrates seamlessly with Hydra projects with no-coding requirements. The only thing needed is a configuration file describing the various parameters to sweep over as normal.
A simple example sweep.yaml file would be:
program: main.py
method: bayes
metric:
goal: maximize
name: test/accuracy
parameters:
dataset:
values: [mnist, cifar10]
command:
- ${env}
- python
- ${program}
- ${args_no_hyphens}
Invoke the sweep:
wandb sweep sweep.yaml` \
W&B automatically creates a sweep inside your project and returns a wandb agent command for you to run on each machine you want to run your sweep.
Hydra supports passing extra parameters through the command line which aren’t present in the default configuration file, by using a + before command. For example, you can pass an extra parameter with some value by simply calling:
$ python program.py +experiment=some_experiment
You cannot sweep over such + configurations similar to what one does while configuring Hydra Experiments. To work around this, you can initialize the experiment parameter with a default empty file and use W&B Sweep to override those empty configs on each call. For more information, read this W&B Report.
W&B has three callbacks for Keras, available from wandb v0.13.4. For the legacy WandbCallback scroll down.
WandbMetricsLogger : Use this callback for Experiment Tracking. It logs your training and validation metrics along with system metrics to Weights and Biases.
WandbModelCheckpoint : Use this callback to log your model checkpoints to Weight and Biases Artifacts.
WandbEvalCallback: This base callback logs model predictions to Weights and Biases Tables for interactive visualization.
These new callbacks:
WandbCallback) for everything.WandbMetricsLoggerWandbMetricsLogger automatically logs Keras’ logs dictionary that callback methods such as on_epoch_end, on_batch_end etc, take as an argument.
This tracks:
model.compile.import wandb
from wandb.integration.keras import WandbMetricsLogger
# Initialize a new W&B run
wandb.init(config={"bs": 12})
# Pass the WandbMetricsLogger to model.fit
model.fit(
X_train, y_train, validation_data=(X_test, y_test), callbacks=[WandbMetricsLogger()]
)
WandbMetricsLogger reference| Parameter | Description |
|---|---|
log_freq |
(epoch, batch, or an int): if epoch, logs metrics at the end of each epoch. If batch, logs metrics at the end of each batch. If an int, logs metrics at the end of that many batches. Defaults to epoch. |
initial_global_step |
(int): Use this argument to correctly log the learning rate when you resume training from some initial_epoch, and a learning rate scheduler is used. This can be computed as step_size * initial_step. Defaults to 0. |
WandbModelCheckpointUse WandbModelCheckpoint callback to save the Keras model (SavedModel format) or model weights periodically and uploads them to W&B as a wandb.Artifact for model versioning.
This callback is subclassed from tf.keras.callbacks.ModelCheckpoint ,thus the checkpointing logic is taken care of by the parent callback.
This callback saves:
SavedModel format or in .h5 format.Use this callback in conjunction with WandbMetricsLogger.
import wandb
from wandb.integration.keras import WandbMetricsLogger, WandbModelCheckpoint
# Initialize a new W&B run
wandb.init(config={"bs": 12})
# Pass the WandbModelCheckpoint to model.fit
model.fit(
X_train,
y_train,
validation_data=(X_test, y_test),
callbacks=[
WandbMetricsLogger(),
WandbModelCheckpoint("models"),
],
)
WandbModelCheckpoint reference| Parameter | Description |
|---|---|
filepath |
(str): path to save the mode file. |
monitor |
(str): The metric name to monitor. |
verbose |
(int): Verbosity mode, 0 or 1. Mode 0 is silent, and mode 1 displays messages when the callback takes an action. |
save_best_only |
(Boolean): if save_best_only=True, it only saves the latest model or the model it considers the best, according to the defined by the monitor and mode attributes. |
save_weights_only |
(Boolean): if True, saves only the model’s weights. |
mode |
(auto, min, or max): For val_acc, set it to max, for val_loss, set it to min, and so on |
save_freq |
(“epoch” or int): When using ‘epoch’, the callback saves the model after each epoch. When using an integer, the callback saves the model at end of this many batches. Note that when monitoring validation metrics such as val_acc or val_loss, save_freq must be set to “epoch” as those metrics are only available at the end of an epoch. |
options |
(str): Optional tf.train.CheckpointOptions object if save_weights_only is true or optional tf.saved_model.SaveOptions object if save_weights_only is false. |
initial_value_threshold |
(float): Floating point initial “best” value of the metric to be monitored. |
By default (save_freq="epoch"), the callback creates a checkpoint and uploads it as an artifact after each epoch. To create a checkpoint after a specific number of batches, set save_freq to an integer. To checkpoint after N epochs, compute the cardinality of the train dataloader and pass it to save_freq:
WandbModelCheckpoint(
filepath="models/",
save_freq=int((trainloader.cardinality()*N).numpy())
)
While checkpointing on TPUs you might encounter UnimplementedError: File system scheme '[local]' not implemented error message. This happens because the model directory (filepath) must use a cloud storage bucket path (gs://bucket-name/...), and this bucket must be accessible from the TPU server. We can however, use the local path for checkpointing which in turn is uploaded as an Artifacts.
checkpoint_options = tf.saved_model.SaveOptions(experimental_io_device="/job:localhost")
WandbModelCheckpoint(
filepath="models/,
options=checkpoint_options,
)
WandbEvalCallbackThe WandbEvalCallback is an abstract base class to build Keras callbacks primarily for model prediction and, secondarily, dataset visualization.
This abstract callback is agnostic with respect to the dataset and the task. To use this, inherit from this base WandbEvalCallback callback class and implement the add_ground_truth and add_model_prediction methods.
The WandbEvalCallback is a utility class that provides methods to:
wandb.Table instances.wandb.Artifact.on_train_begin.on_epoch_end.The following example uses WandbClfEvalCallback for an image classification task. This example callback logs the validation data (data_table) to W&B, performs inference, and logs the prediction (pred_table) to W&B at the end of every epoch.
import wandb
from wandb.integration.keras import WandbMetricsLogger, WandbEvalCallback
# Implement your model prediction visualization callback
class WandbClfEvalCallback(WandbEvalCallback):
def __init__(
self, validation_data, data_table_columns, pred_table_columns, num_samples=100
):
super().__init__(data_table_columns, pred_table_columns)
self.x = validation_data[0]
self.y = validation_data[1]
def add_ground_truth(self, logs=None):
for idx, (image, label) in enumerate(zip(self.x, self.y)):
self.data_table.add_data(idx, wandb.Image(image), label)
def add_model_predictions(self, epoch, logs=None):
preds = self.model.predict(self.x, verbose=0)
preds = tf.argmax(preds, axis=-1)
table_idxs = self.data_table_ref.get_index()
for idx in table_idxs:
pred = preds[idx]
self.pred_table.add_data(
epoch,
self.data_table_ref.data[idx][0],
self.data_table_ref.data[idx][1],
self.data_table_ref.data[idx][2],
pred,
)
# ...
# Initialize a new W&B run
wandb.init(config={"hyper": "parameter"})
# Add the Callbacks to Model.fit
model.fit(
X_train,
y_train,
validation_data=(X_test, y_test),
callbacks=[
WandbMetricsLogger(),
WandbClfEvalCallback(
validation_data=(X_test, y_test),
data_table_columns=["idx", "image", "label"],
pred_table_columns=["epoch", "idx", "image", "label", "pred"],
),
],
)
WandbEvalCallback reference| Parameter | Description |
|---|---|
data_table_columns |
(list) List of column names for the data_table |
pred_table_columns |
(list) List of column names for the pred_table |
We log the data_table to W&B when the on_train_begin method is invoked. Once it’s uploaded as a W&B Artifact, we get a reference to this table which can be accessed using data_table_ref class variable. The data_table_ref is a 2D list that can be indexed like self.data_table_ref[idx][n], where idx is the row number while n is the column number. Let’s see the usage in the example below.
You can override the on_train_begin or on_epoch_end methods to have more fine-grained control. If you want to log the samples after N batches, you can implement on_train_batch_end method.
WandbEvalCallback and something needs to be clarified or fixed, please let us know by opening an issue.WandbCallback [legacy]Use the W&B library WandbCallback Class to automatically save all the metrics and the loss values tracked in model.fit.
import wandb
from wandb.integration.keras import WandbCallback
wandb.init(config={"hyper": "parameter"})
... # code to set up your model in Keras
# Pass the callback to model.fit
model.fit(
X_train, y_train, validation_data=(X_test, y_test), callbacks=[WandbCallback()]
)
You can watch the short video Get Started with Keras and Weights & Biases in Less Than a Minute.
For a more detailed video, watch Integrate Weights & Biases with Keras. You can review the Colab Jupyter Notebook.
The WandbCallback class supports a wide variety of logging configuration options: specifying a metric to monitor, tracking of weights and gradients, logging of predictions on training_data and validation_data, and more.
Check out the reference documentation for the keras.WandbCallback for full details.
The WandbCallback
keras_model.compile().monitor and mode attributes. This defaults to the epoch with the minimum val_loss. WandbCallback by default saves the model associated with the best epoch.WandbCallback reference| Arguments | |
|---|---|
monitor |
(str) name of metric to monitor. Defaults to val_loss. |
mode |
(str) one of {auto, min, max}. min - save model when monitor is minimized max - save model when monitor is maximized auto - try to guess when to save the model (default). |
save_model |
True - save a model when monitor beats all previous epochs False - don’t save models |
save_graph |
(boolean) if True save model graph to wandb (default to True). |
save_weights_only |
(boolean) if True, saves only the model’s weights(model.save_weights(filepath)). Otherwise, saves the full model). |
log_weights |
(boolean) if True save histograms of the model’s layer’s weights. |
log_gradients |
(boolean) if True log histograms of the training gradients |
training_data |
(tuple) Same format (X,y) as passed to model.fit. This is needed for calculating gradients - this is mandatory if log_gradients is True. |
validation_data |
(tuple) Same format (X,y) as passed to model.fit. A set of data for wandb to visualize. If you set this field, every epoch, wandb makes a small number of predictions and saves the results for later visualization. |
generator |
(generator) a generator that returns validation data for wandb to visualize. This generator should return tuples (X,y). Either validate_data or generator should be set for wandb to visualize specific data examples. |
validation_steps |
(int) if validation_data is a generator, how many steps to run the generator for the full validation set. |
labels |
(list) If you are visualizing your data with wandb this list of labels converts numeric output to understandable string if you are building a classifier with multiple classes. For a binary classifier, you can pass in a list of two labels [label for false, label for true]. If validate_data and generator are both false, this does nothing. |
predictions |
(int) the number of predictions to make for visualization each epoch, max is 100. |
input_type |
(string) type of the model input to help visualization. can be one of: (image, images, segmentation_mask). |
output_type |
(string) type of the model output to help visualziation. can be one of: (image, images, segmentation_mask). |
log_evaluation |
(boolean) if True, save a Table containing validation data and the model’s predictions at each epoch. See validation_indexes, validation_row_processor, and output_row_processor for additional details. |
class_colors |
([float, float, float]) if the input or output is a segmentation mask, an array containing an rgb tuple (range 0-1) for each class. |
log_batch_frequency |
(integer) if None, callback logs every epoch. If set to integer, callback logs training metrics every log_batch_frequency batches. |
log_best_prefix |
(string) if None, saves no extra summary metrics. If set to a string, prepends the monitored metric and epoch with the prefix and saves the results as summary metrics. |
validation_indexes |
([wandb.data_types._TableLinkMixin]) an ordered list of index keys to associate with each validation example. If log_evaluation is True and you provide validation_indexes, does not create a Table of validation data. Instead, associates each prediction with the row represented by the TableLinkMixin. To obtain a list of row keys, use Table.get_index() . |
validation_row_processor |
(Callable) a function to apply to the validation data, commonly used to visualize the data. The function receives an ndx (int) and a row (dict). If your model has a single input, then row["input"] contains the input data for the row. Otherwise, it contains the names of the input slots. If your fit function takes a single target, then row["target"] contains the target data for the row. Otherwise, it contains the names of the output slots. For example, if your input data is a single array, to visualize the data as an Image, provide lambda ndx, row: {"img": wandb.Image(row["input"])} as the processor. Ignored if log_evaluation is False or validation_indexes are present. |
output_row_processor |
(Callable) same as validation_row_processor, but applied to the model’s output. row["output"] contains the results of the model output. |
infer_missing_processors |
(Boolean) Determines whether to infer validation_row_processor and output_row_processor if they are missing. Defaults to True. If you provide labels, W&B attempts to infer classification-type processors where appropriate. |
log_evaluation_frequency |
(int) Determines how often to log evaluation results. Defaults to 0 to log only at the end of training. Set to 1 to log every epoch, 2 to log every other epoch, and so on. Has no effect when log_evaluation is False. |
Keras multiprocessing with wandb?When setting use_multiprocessing=True, this error may occur:
Error("You must call wandb.init() before wandb.config.batch_size")
To work around it:
Sequence class construction, add: wandb.init(group='...').main, make sure you’re using if __name__ == "__main__": and put the rest of your script logic inside it.Kubeflow Pipelines (kfp) is a platform for building and deploying portable, scalable machine learning (ML) workflows based on Docker containers.
This integration lets users apply decorators to kfp python functional components to automatically log parameters and artifacts to W&B.
This feature was enabled in wandb==0.12.11 and requires kfp<2.0.0
!pip install kfp wandb
import wandb
wandb.login()
pip install kfp wandb
wandb login
Add the @wandb_log decorator and create your components as usual. This will automatically log the input/outputs parameters and artifacts to W&B each time you run your pipeline.
from kfp import components
from wandb.integration.kfp import wandb_log
@wandb_log
def add(a: float, b: float) -> float:
return a + b
add = components.create_component_from_func(add)
You may need to explicitly pass environment variables to your containers. For two-way linking, you should also set the environment variables WANDB_KUBEFLOW_URL to the base URL of your Kubeflow Pipelines instance. For example, https://kubeflow.mysite.com.
import os
from kubernetes.client.models import V1EnvVar
def add_wandb_env_variables(op):
env = {
"WANDB_API_KEY": os.getenv("WANDB_API_KEY"),
"WANDB_BASE_URL": os.getenv("WANDB_BASE_URL"),
}
for name, value in env.items():
op = op.add_env_variable(V1EnvVar(name, value))
return op
@dsl.pipeline(name="example-pipeline")
def example_pipeline(param1: str, param2: int):
conf = dsl.get_pipeline_conf()
conf.add_op_transformer(add_wandb_env_variables)
Click on any Run in the Kubeflow Pipelines UI that has been logged with W&B.
Input/Output and ML Metadata tabs.Visualizations tab.
The web app UI has the same content as the Visualizations tab in Kubeflow Pipelines, but with more space. Learn more about the web app UI here.
Here’s a mapping of Kubeflow Pipelines concepts to W&B
| Kubeflow Pipelines | W&B | Location in W&B |
|---|---|---|
| Input Scalar | config |
Overview tab |
| Output Scalar | summary |
Overview tab |
| Input Artifact | Input Artifact | Artifacts tab |
| Output Artifact | Output Artifact | Artifacts tab |
If you want finer control of logging, you can sprinkle in wandb.log and wandb.log_artifact calls in the component.
wandb.log_artifacts callsIn this example below, we are training a model. The @wandb_log decorator will automatically track the relevant inputs and outputs. If you want to log the training process, you can explicitly add that logging like so:
@wandb_log
def train_model(
train_dataloader_path: components.InputPath("dataloader"),
test_dataloader_path: components.InputPath("dataloader"),
model_path: components.OutputPath("pytorch_model"),
):
...
for epoch in epochs:
for batch_idx, (data, target) in enumerate(train_dataloader):
...
if batch_idx % log_interval == 0:
wandb.log(
{"epoch": epoch, "step": batch_idx * len(data), "loss": loss.item()}
)
...
wandb.log_artifact(model_artifact)
If you’re using a framework integration we support, you can also pass in the callback directly:
@wandb_log
def train_model(
train_dataloader_path: components.InputPath("dataloader"),
test_dataloader_path: components.InputPath("dataloader"),
model_path: components.OutputPath("pytorch_model"),
):
from pytorch_lightning.loggers import WandbLogger
from pytorch_lightning import Trainer
trainer = Trainer(logger=WandbLogger())
... # do training
LangChain is a framework for developing applications powered by language models.
To use the Weights & Biases LangChain integration please see our W&B Prompts Quickstart
Or try our Google Colab Jupyter notebook
The wandb library includes a special callback for LightGBM. It’s also easy to use the generic logging features of Weights & Biases to track large experiments, like hyperparameter sweeps.
from wandb.integration.lightgbm import wandb_callback, log_summary
import lightgbm as lgb
# Log metrics to W&B
gbm = lgb.train(..., callbacks=[wandb_callback()])
# Log feature importance plot and upload model checkpoint to W&B
log_summary(gbm, save_model_checkpoint=True)
Attaining the maximum performance out of models requires tuning hyperparameters, like tree depth and learning rate. Weights & Biases includes Sweeps, a powerful toolkit for configuring, orchestrating, and analyzing large hyperparameter testing experiments.
To learn more about these tools and see an example of how to use Sweeps with XGBoost, check out this interactive Colab notebook.
Metaflow is a framework created by Netflix for creating and running ML workflows.
This integration lets users apply decorators to Metaflow steps and flows to automatically log parameters and artifacts to W&B.
!pip install -Uqqq metaflow fastcore wandb
import wandb
wandb.login()
pip install -Uqqq metaflow fastcore wandb
wandb login
Decorating a step turns logging off or on for certain types within that step.
In this example, all datasets and models in start will be logged
from wandb.integration.metaflow import wandb_log
class WandbExampleFlow(FlowSpec):
@wandb_log(datasets=True, models=True, settings=wandb.Settings(...))
@step
def start(self):
self.raw_df = pd.read_csv(...). # pd.DataFrame -> upload as dataset
self.model_file = torch.load(...) # nn.Module -> upload as model
self.next(self.transform)
Decorating a flow is equivalent to decorating all the constituent steps with a default.
In this case, all steps in WandbExampleFlow default to logging datasets and models by default, just like decorating each step with @wandb_log(datasets=True, models=True)
from wandb.integration.metaflow import wandb_log
@wandb_log(datasets=True, models=True) # decorate all @step
class WandbExampleFlow(FlowSpec):
@step
def start(self):
self.raw_df = pd.read_csv(...). # pd.DataFrame -> upload as dataset
self.model_file = torch.load(...) # nn.Module -> upload as model
self.next(self.transform)
Decorating the flow is equivalent to decorating all steps with a default. That means if you later decorate a Step with another @wandb_log, it overrides the flow-level decoration.
In this example:
start and mid log both datasets and models.end logs neither datasets nor models.from wandb.integration.metaflow import wandb_log
@wandb_log(datasets=True, models=True) # same as decorating start and mid
class WandbExampleFlow(FlowSpec):
# this step will log datasets and models
@step
def start(self):
self.raw_df = pd.read_csv(...). # pd.DataFrame -> upload as dataset
self.model_file = torch.load(...) # nn.Module -> upload as model
self.next(self.mid)
# this step will also log datasets and models
@step
def mid(self):
self.raw_df = pd.read_csv(...). # pd.DataFrame -> upload as dataset
self.model_file = torch.load(...) # nn.Module -> upload as model
self.next(self.end)
# this step is overwritten and will NOT log datasets OR models
@wandb_log(datasets=False, models=False)
@step
def end(self):
self.raw_df = pd.read_csv(...).
self.model_file = torch.load(...)
You can access the information we’ve captured in three ways: inside the original Python process being logged using the wandb client library, with the web app UI, or programmatically using our Public API. Parameters are saved to W&B’s config and can be found in the Overview tab. datasets, models, and others are saved to W&B Artifacts and can be found in the Artifacts tab. Base python types are saved to W&B’s summary dict and can be found in the Overview tab. See our guide to the Public API for details on using the API to get this information programmatically from outside .
| Data | Client library | UI |
|---|---|---|
Parameter(...) |
wandb.config |
Overview tab, Config |
datasets, models, others |
wandb.use_artifact("{var_name}:latest") |
Artifacts tab |
Base Python types (dict, list, str, etc.) |
wandb.summary |
Overview tab, Summary |
wandb_log kwargs| kwarg | Options |
|---|---|
datasets |
|
models |
|
others |
|
settings |
By default, if:
|
wandb_log only logs instance variables. Local variables are NEVER logged. This is useful to avoid logging unnecessary data.
We currently support these types:
| Logging Setting | Type |
|---|---|
| default (always on) |
|
datasets |
|
models |
|
others |
|
| Kind of Variable | behavior | Example | Data Type |
|---|---|---|---|
| Instance | Auto-logged | self.accuracy |
float |
| Instance | Logged if datasets=True |
self.df |
pd.DataFrame |
| Instance | Not logged if datasets=False |
self.df |
pd.DataFrame |
| Local | Never logged | accuracy |
float |
| Local | Never logged | df |
pd.DataFrame |
Yes. If you have an artifact that is an output of step A and an input to step B, we automatically construct the lineage DAG for you.
For an example of this behavior, please see this notebook and its corresponding W&B Artifacts page
MMEngine by OpenMMLab is a foundational library for training deep learning models based on PyTorch. MMEngine implements a next-generation training architecture for the OpenMMLab algorithm library, providing a unified execution foundation for over 30 algorithm libraries within OpenMMLab. Its core components include the training engine, evaluation engine, and module management.
Weights and Biases is directly integrated into MMEngine through a dedicated WandbVisBackend that can be used to
Install openmim and wandb.
pip install -q -U openmim wandb
!pip install -q -U openmim wandb
Next, install mmengine and mmcv using mim.
mim install -q mmengine mmcv
!mim install -q mmengine mmcv
WandbVisBackend with MMEngine RunnerThis section demonstrates a typical workflow using WandbVisBackend using mmengine.runner.Runner.
Define a visualizer from a visualization config.
from mmengine.visualization import Visualizer
# define the visualization configs
visualization_cfg = dict(
name="wandb_visualizer",
vis_backends=[
dict(
type='WandbVisBackend',
init_kwargs=dict(project="mmengine"),
)
],
save_dir="runs/wandb"
)
# get the visualizer from the visualization configs
visualizer = Visualizer.get_instance(**visualization_cfg)
You pass a dictionary of arguments for [W&B run initialization](/ref/python/init) input parameters to `init_kwargs`.
Initialize a runner with the visualizer, and call runner.train().
from mmengine.runner import Runner
# build the mmengine Runner which is a training helper for PyTorch
runner = Runner(
model,
work_dir='runs/gan/',
train_dataloader=train_dataloader,
train_cfg=train_cfg,
optim_wrapper=opt_wrapper_dict,
visualizer=visualizer, # pass the visualizer
)
# start training
runner.train()
WandbVisBackend with OpenMMLab computer vision librariesThe WandbVisBackend can also be used easily to track experiments with OpenMMLab computer vision libraries such as MMDetection.
# inherit base configs from the default runtime configs
_base_ = ["../_base_/default_runtime.py"]
# Assign the `WandbVisBackend` config dictionary to the
# `vis_backends` of the `visualizer` from the base configs
_base_.visualizer.vis_backends = [
dict(
type='WandbVisBackend',
init_kwargs={
'project': 'mmdet',
'entity': 'geekyrakshit'
},
),
]
The WandbLogger class in Meta AI’s MMF library will enable Weights & Biases to log the training/validation metrics, system (GPU and CPU) metrics, model checkpoints and configuration parameters.
The following features are currently supported by the WandbLogger in MMF:
The following options are available in MMF config to enable and customize the wandb logging:
training:
wandb:
enabled: true
# An entity is a username or team name where you're sending runs.
# By default it will log the run to your user account.
entity: null
# Project name to be used while logging the experiment with wandb
project: mmf
# Experiment/ run name to be used while logging the experiment
# under the project with wandb. The default experiment name
# is: ${training.experiment_name}
name: ${training.experiment_name}
# Turn on model checkpointing, saving checkpoints to W&B Artifacts
log_model_checkpoint: true
# Additional argument values that you want to pass to wandb.init().
# Check out the documentation at /ref/python/init
# to see what arguments are available, such as:
# job_type: 'train'
# tags: ['tag1', 'tag2']
env:
# To change the path to the directory where wandb metadata would be
# stored (Default: env.log_dir):
wandb_logdir: ${env:MMF_WANDB_LOGDIR,}
Composer is a library for training neural networks better, faster, and cheaper. It contains many state-of-the-art methods for accelerating neural network training and improving generalization, along with an optional Trainer API that makes composing many different enhancements easy.
W&B provides a lightweight wrapper for logging your ML experiments. But you don’t need to combine the two yourself: W&B is incorporated directly into the Composer library via the WandBLogger.
from composer import Trainer
from composer.loggers import WandBLogger
trainer = Trainer(..., logger=WandBLogger())
WandBLoggerThe Composer library uses WandBLogger class in the Trainer to log metrics to Weights and Biases. It is a simple as instantiating the logger and passing it to the Trainer
wandb_logger = WandBLogger(project="gpt-5", log_artifacts=True)
trainer = Trainer(logger=wandb_logger)
Below the parameters for WandbLogger, see the Composer documentation for a full list and description
| Parameter | Description |
|---|---|
project |
W&B project name (str, optional) |
group |
W&B group name (str, optional) |
name |
W&B run name. If not specified, the State.run_name is used (str, optional) |
entity |
W&B entity name, such as your username or W&B Team name (str, optional) |
tags |
W&B tags (List[str], optional) |
log_artifacts |
Whether to log checkpoints to wandb, default: false (bool, optional) |
rank_zero_only |
Whether to log only on the rank-zero process. When logging artifacts, it is highly recommended to log on all ranks. Artifacts from ranks ≥1 are not stored, which may discard pertinent information. For example, when using Deepspeed ZeRO, it would be impossible to restore from checkpoints without artifacts from all ranks, default: True (bool, optional) |
init_kwargs |
Params to pass to wandb.init such as your wandb config etc See here for the full list wandb.init accepts |
A typical usage would be:
init_kwargs = {"notes":"Testing higher learning rate in this experiment",
"config":{"arch":"Llama",
"use_mixed_precision":True
}
}
wandb_logger = WandBLogger(log_artifacts=True, init_kwargs=init_kwargs)
You can use Composer’s Callbacks system to control when you log to Weights & Biases via the WandBLogger, in this example a sample of the validation images and predictions is logged:
import wandb
from composer import Callback, State, Logger
class LogPredictions(Callback):
def __init__(self, num_samples=100, seed=1234):
super().__init__()
self.num_samples = num_samples
self.data = []
def eval_batch_end(self, state: State, logger: Logger):
"""Compute predictions per batch and stores them on self.data"""
if state.timer.epoch == state.max_duration: #on last val epoch
if len(self.data) < self.num_samples:
n = self.num_samples
x, y = state.batch_pair
outputs = state.outputs.argmax(-1)
data = [[wandb.Image(x_i), y_i, y_pred] for x_i, y_i, y_pred in list(zip(x[:n], y[:n], outputs[:n]))]
self.data += data
def eval_end(self, state: State, logger: Logger):
"Create a wandb.Table and logs it"
columns = ['image', 'ground truth', 'prediction']
table = wandb.Table(columns=columns, data=self.data[:self.num_samples])
wandb.log({'sample_table':table}, step=int(state.timer.batch))
...
trainer = Trainer(
...
loggers=[WandBLogger()],
callbacks=[LogPredictions()]
)
Use the W&B OpenAI API integration to log requests, responses, token counts and model metadata for all OpenAI models, including fine-tuned models.
Log your API inputs and outputs you can quickly evaluate the performance of difference prompts, compare different model settings (such as temperature), and track other usage metrics such as token usage.
The W&B autolog integration works with OpenAI version 0.28.1 and below.
To install OpenAI Python API version 0.28.1, run:
pip install openai==0.28.1
First, import autolog from wandb.integration.openai and initialise it.
import os
import openai
from wandb.integration.openai import autolog
autolog({"project": "gpt5"})
You can optionally pass a dictionary with argument that wandb.init() accepts to autolog. This includes a project name, team name, entity, and more. For more information about wandb.init, see the API Reference Guide.
Each call you make to the OpenAI API is now logged to W&B automatically.
os.environ["OPENAI_API_KEY"] = "XXX"
chat_request_kwargs = dict(
model="gpt-3.5-turbo",
messages=[
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "Who won the world series in 2020?"},
{"role": "assistant", "content": "The Los Angeles Dodgers"},
{"role": "user", "content": "Where was it played?"},
],
)
response = openai.ChatCompletion.create(**chat_request_kwargs)
Click on the W&B run link generated by autolog in step 1. This redirects you to your project workspace in the W&B App.
Select a run you created to view the trace table, trace timeline and the model architecture of the OpenAI LLM used.
W&B recommends that you call disable() to close all W&B processes when you are finished using the OpenAI API.
autolog.disable()
Now your inputs and completions will be logged to W&B, ready for analysis or to be shared with colleagues.
Log your OpenAI GPT-3.5 or GPT-4 model’s fine-tuning metrics and configuration to W&B. Utilize the W&B ecosystem to track your fine-tuning experiments, models, and datasets and share your results with your colleagues.
See the Weights and Biases Integration section in the OpenAI documentation for supplemental information on how to integrate W&B with OpenAI for fine-tuning.
The W&B OpenAI fine-tuning integration works with OpenAI version 1.0 and above. See the PyPI documentation for the latest version of the OpenAI Python API library.
To install OpenAI Python API, run:
pip install openai
If you already have OpenAI Python API installed, you can update it with:
pip install -U openai
Integrate W&B with OpenAI’s fine-tuning API to log your fine-tuning metrics and configuration to W&B. To do this, use the WandbLogger class from the wandb.integration.openai.fine_tuning module.
from wandb.integration.openai.fine_tuning import WandbLogger
# Finetuning logic
WandbLogger.sync(fine_tune_job_id=FINETUNE_JOB_ID)
Sync your results from your script
from wandb.integration.openai.fine_tuning import WandbLogger
# one line command
WandbLogger.sync()
# passing optional parameters
WandbLogger.sync(
fine_tune_job_id=None,
num_fine_tunes=None,
project="OpenAI-Fine-Tune",
entity=None,
overwrite=False,
model_artifact_name="model-metadata",
model_artifact_type="model",
**kwargs_wandb_init
)
| Argument | Description |
|---|---|
| fine_tune_job_id | This is the OpenAI Fine-Tune ID which you get when you create your fine-tune job using client.fine_tuning.jobs.create. If this argument is None (default), all the OpenAI fine-tune jobs that haven’t already been synced will be synced to W&B. |
| openai_client | Pass an initialized OpenAI client to sync. If no client is provided, one is initialized by the logger itself. By default it is None. |
| num_fine_tunes | If no ID is provided, then all the unsynced fine-tunes will be logged to W&B. This argument allows you to select the number of recent fine-tunes to sync. If num_fine_tunes is 5, it selects the 5 most recent fine-tunes. |
| project | Weights and Biases project name where your fine-tune metrics, models, data, etc. will be logged. By default, the project name is “OpenAI-Fine-Tune.” |
| entity | W&B Username or team name where you’re sending runs. By default, your default entity is used, which is usually your username. |
| overwrite | Forces logging and overwrite existing wandb run of the same fine-tune job. By default this is False. |
| wait_for_job_success | Once an OpenAI fine-tuning job is started it usually takes a bit of time. To ensure that your metrics are logged to W&B as soon as the fine-tune job is finished, this setting will check every 60 seconds for the status of the fine-tune job to change to succeeded. Once the fine-tune job is detected as being successful, the metrics will be synced automatically to W&B. Set to True by default. |
| model_artifact_name | The name of the model artifact that is logged. Defaults to "model-metadata". |
| model_artifact_type | The type of the model artifact that is logged. Defaults to "model". |
| **kwargs_wandb_init | Aany additional argument passed directly to wandb.init() |
The training and validation data that you upload to OpenAI for fine-tuning are automatically logged as W&B Artifacts for easier version control. Below is an view of the training file in Artifacts. Here you can see the W&B run that logged this file, when it was logged, what version of the dataset this is, the metadata, and DAG lineage from the training data to the trained model.
The datasets are visualized as W&B Tables, which allows you to explore, search, and interact with the dataset. Check out the training samples visualized using W&B Tables below.
OpenAI gives you an id of the fine-tuned model. Since we don’t have access to the model weights, the WandbLogger creates a model_metadata.json file with all the details (hyperparameters, data file ids, etc.) of the model along with the `fine_tuned_model`` id and is logged as a W&B Artifact.
This model (metadata) artifact can further be linked to a model in the W&B Model Registry.
Log your fine-tune jobs to your team account with:
WandbLogger.sync(entity="YOUR_TEAM_NAME")
Your W&B runs are automatically organized and can be filtered/sorted based on any configuration parameter such as job type, base model, learning rate, training filename and any other hyper-parameter.
In addition, you can rename your runs, add notes or create tags to group them.
Once you’re satisfied, you can save your workspace and use it to create report, importing data from your runs and saved artifacts (training/validation files).
Fine-tuned model ID is logged to W&B as artifacts (model_metadata.json) as well config.
import wandb
ft_artifact = wandb.run.use_artifact("ENTITY/PROJECT/model_metadata:VERSION")
artifact_dir = artifact.download()
where VERSION is either:
v2ft-xxxxxxxxxlatest or manuallyYou can then access fine_tuned_model id by reading the downloaded model_metadata.json file.
If a fine-tune was not logged to W&B successfully, you can use the overwrite=True and pass the fine-tune job id:
WandbLogger.sync(
fine_tune_job_id="FINE_TUNE_JOB_ID",
overwrite=True,
)
The training and validation data are logged automatically to W&B as artifacts. The metadata including the ID for the fine-tuned model is also logged as artifacts.
You can always control the pipeline using low level wandb APIs like wandb.Artifact, wandb.log, etc. This will allow complete traceability of your data and models.
“The team that has been maintaining Gym since 2021 has moved all future development to Gymnasium, a drop in replacement for Gym (import gymnasium as gym), and Gym will not be receiving any future updates.” (Source)
Since Gym is no longer an actively maintained project, try out our integration with Gymnasium.
If you’re using OpenAI Gym, Weights & Biases automatically logs videos of your environment generated by gym.wrappers.Monitor. Just set the monitor_gym keyword argument to wandb.init to True or call wandb.gym.monitor().
Our gym integration is very light. We simply look at the name of the video file being logged from gym and name it after that or fall back to "videos" if we don’t find a match. If you want more control, you can always just manually log a video.
The OpenRL Benchmark by CleanRL uses this integration for its OpenAI Gym examples. You can find source code (including the specific code used for specific runs) that demonstrates how to use gym with
PaddleDetection is an end-to-end object-detection development kit based on PaddlePaddle. It detects various mainstream objects, segments instances, and tracks and detects keypoints using configurable modules such as network components, data augmentations, and losses.
PaddleDetection now includes a built-in W&B integration which logs all your training and validation metrics, as well as your model checkpoints and their corresponding metadata.
The PaddleDetection WandbLogger logs your training and evaluation metrics to Weights & Biases as well as your model checkpoints while training.
Read a W&B blog post which illustrates how to integrate a YOLOX model with PaddleDetection on a subset of the COCO2017 dataset.
Sign up for a free Weights & Biases account, then pip install the wandb library. To login, you’ll need to be signed in to you account at www.wandb.ai. Once signed in you will find your API key on the Authorize page.
pip install wandb
wandb login
!pip install wandb
wandb.login()
WandbLogger in your training scriptTo use wandb via arguments to train.py in PaddleDetection:
--use_wandb flag-o (you only need to pass this once)wandb- . For example any argument to be passed to wandb.init would get the wandb- prefixpython tools/train.py
-c config.yml \
--use_wandb \
-o \
wandb-project=MyDetector \
wandb-entity=MyTeam \
wandb-save_dir=./logs
config.yml fileYou can also activate wandb via the config file. Add the wandb arguments to the config.yml file under the wandb header like so:
wandb:
project: MyProject
entity: MyTeam
save_dir: ./logs
Once you run your train.py file with Weights & Biases turned on, a link will be generated to bring you to your W&B dashboard:
If you have any feedback or issues about the Weights & Biases integration please open an issue on the PaddleDetection GitHub or email support@wandb.com.
PaddleOCR aims to create multilingual, awesome, leading, and practical OCR tools that help users train better models and apply them into practice implemented in PaddlePaddle. PaddleOCR support a variety of cutting-edge algorithms related to OCR, and developed industrial solution. PaddleOCR now comes with a Weights & Biases integration for logging training and evaluation metrics along with model checkpoints with corresponding metadata.
Read here to see how to train a model with PaddleOCR on the ICDAR2015 dataset. This also comes with a Google Colab and the corresponding live W&B dashboard is available here. There is also a Chinese version of this blog here: W&B对您的OCR模型进行训练和调试
Sign up for a free account, then from the command line install the wandb library in a Python 3 environment. To login, you’ll need to be signed in to you account at www.wandb.ai, then you will find your API key on the Authorize page.
pip install wandb
wandb login
!pip install wandb
wandb.login()
config.yml filePaddleOCR requires configuration variables to be provided using a yaml file. Adding the following snippet at the end of the configuration yaml file will automatically log all training and validation metrics to a W&B dashboard along with model checkpoints:
Global:
use_wandb: True
Any additional, optional arguments that you might like to pass to wandb.init can also be added under the wandb header in the yaml file:
wandb:
project: CoolOCR # (optional) this is the wandb project name
entity: my_team # (optional) if you're using a wandb team, you can pass the team name here
name: MyOCRModel # (optional) this is the name of the wandb run
config.yml file to train.pyThe yaml file is then provided as an argument to the training script available in the PaddleOCR repository.
python tools/train.py -c config.yml
Once you run your train.py file with Weights & Biases turned on, a link will be generated to bring you to your W&B dashboard:
If you have any feedback or issues about the Weights & Biases integration please open an issue on the PaddleOCR GitHub or email support@wandb.com.
Prodigy is an annotation tool for creating training and evaluation data for machine learning models, error analysis, data inspection & cleaning. W&B Tables allow you to log, visualize, analyze, and share datasets (and more!) inside W&B.
The W&B integration with Prodigy adds simple and easy-to-use functionality to upload your Prodigy-annotated dataset directly to W&B for use with Tables.
Run a few lines of code, like these:
import wandb
from wandb.integration.prodigy import upload_dataset
with wandb.init(project="prodigy"):
upload_dataset("news_headlines_ner")
and get visual, interactive, shareable tables like this one:
Use wandb.integration.prodigy.upload_dataset to upload your annotated prodigy dataset directly from the local Prodigy database to W&B in our Table format. For more information on Prodigy, including installation & setup, please refer to the Prodigy documentation.
W&B will automatically try to convert images and named entity fields to wandb.Image and wandb.Htmlrespectively. Extra columns may be added to the resulting table to include these visualizations.
Explore the Visualizing Prodigy Datasets Using W&B Tables for example visualizations generated with W&B Prodigy integration.
W&B also has an integration with spaCy, see the docs here.
PyTorch is one of the most popular frameworks for deep learning in Python, especially among researchers. W&B provides first class support for PyTorch, from logging gradients to profiling your code on the CPU and GPU.
Try our integration out in a Colab notebook.
You can also see our example repo for scripts, including one on hyperparameter optimization using Hyperband on Fashion MNIST, plus the W&B Dashboard it generates.
wandb.watchTo automatically log gradients, you can call wandb.watch and pass in your PyTorch model.
import wandb
wandb.init(config=args)
model = ... # set up your model
# Magic
wandb.watch(model, log_freq=100)
model.train()
for batch_idx, (data, target) in enumerate(train_loader):
output = model(data)
loss = F.nll_loss(output, target)
loss.backward()
optimizer.step()
if batch_idx % args.log_interval == 0:
wandb.log({"loss": loss})
If you need to track multiple models in the same script, you can call wandb.watch on each model separately. Reference documentation for this function is here.
wandb.log is called after a forward and backward pass.You can pass PyTorch Tensors with image data into wandb.Image and utilities from torchvision will be used to convert them to images automatically:
images_t = ... # generate or load images as PyTorch Tensors
wandb.log({"examples": [wandb.Image(im) for im in images_t]})
For more on logging rich media to W&B in PyTorch and other frameworks, check out our media logging guide.
If you also want to include information alongside media, like your model’s predictions or derived metrics, use a wandb.Table.
my_table = wandb.Table()
my_table.add_column("image", images_t)
my_table.add_column("label", labels)
my_table.add_column("class_prediction", predictions_t)
# Log your Table to W&B
wandb.log({"mnist_predictions": my_table})
For more on logging and visualizing datasets and models, check out our guide to W&B Tables.
W&B integrates directly with PyTorch Kineto’s Tensorboard plugin to provide tools for profiling PyTorch code, inspecting the details of CPU and GPU communication, and identifying bottlenecks and optimizations.
profile_dir = "path/to/run/tbprofile/"
profiler = torch.profiler.profile(
schedule=schedule, # see the profiler docs for details on scheduling
on_trace_ready=torch.profiler.tensorboard_trace_handler(profile_dir),
with_stack=True,
)
with profiler:
... # run the code you want to profile here
# see the profiler docs for detailed usage information
# create a wandb Artifact
profile_art = wandb.Artifact("trace", type="profile")
# add the pt.trace.json files to the Artifact
profile_art.add_file(glob.glob(profile_dir + ".pt.trace.json"))
# log the artifact
profile_art.save()
See and run working example code in this Colab.
PyTorch Geometric or PyG is one of the most popular libraries for geometric deep learning and W&B works extremely well with it for visualizing graphs and tracking experiments.
After you have installed pytorch geometric, install the wandb library and login
pip install wandb
wandb login
!pip install wandb
import wandb
wandb.login()
You can save details about the input graphs including number of edges, number of nodes and more. W&B supports logging plotly charts and HTML panels so any visualizations you create for your graph can then also be logged to W&B.
The following snippet shows how you could do that with PyVis and HTML.
from pyvis.network import Network
Import wandb
wandb.init(project=’graph_vis’)
net = Network(height="750px", width="100%", bgcolor="#222222", font_color="white")
# Add the edges from the PyG graph to the PyVis network
for e in tqdm(g.edge_index.T):
src = e[0].item()
dst = e[1].item()
net.add_node(dst)
net.add_node(src)
net.add_edge(src, dst, value=0.1)
# Save the PyVis visualisation to a HTML file
net.show("graph.html")
wandb.log({"eda/graph": wandb.Html("graph.html")})
wandb.finish()
To use plotly to create a graph visualization, first you need to convert the PyG graph to a networkx object. Following this you will need to create Plotly scatter plots for both nodes and edges. The snippet below can be used for this task.
def create_vis(graph):
G = to_networkx(graph)
pos = nx.spring_layout(G)
edge_x = []
edge_y = []
for edge in G.edges():
x0, y0 = pos[edge[0]]
x1, y1 = pos[edge[1]]
edge_x.append(x0)
edge_x.append(x1)
edge_x.append(None)
edge_y.append(y0)
edge_y.append(y1)
edge_y.append(None)
edge_trace = go.Scatter(
x=edge_x, y=edge_y,
line=dict(width=0.5, color='#888'),
hoverinfo='none',
mode='lines'
)
node_x = []
node_y = []
for node in G.nodes():
x, y = pos[node]
node_x.append(x)
node_y.append(y)
node_trace = go.Scatter(
x=node_x, y=node_y,
mode='markers',
hoverinfo='text',
line_width=2
)
fig = go.Figure(data=[edge_trace, node_trace], layout=go.Layout())
return fig
wandb.init(project=’visualize_graph’)
wandb.log({‘graph’: wandb.Plotly(create_vis(graph))})
wandb.finish()
You can use W&B to track your experiments and related metrics, such as loss functions, accuracy, and more. Add the following line to your training loop:
wandb.log({
‘train/loss’: training_loss,
‘train/acc’: training_acc,
‘val/loss’: validation_loss,
‘val/acc’: validation_acc
})
torchtune is a PyTorch-based library designed to streamline the authoring, fine-tuning, and experimentation processes for large language models (LLMs). Additionally, torchtune has built-in support for logging with W&B, enhancing tracking and visualization of training processes.
Check the W&B blog post on Fine-tuning Mistral 7B using torchtune.
Override command line arguments at launch:
tune run lora_finetune_single_device --config llama3/8B_lora_single_device \
metric_logger._component_=torchtune.utils.metric_logging.WandBLogger \
metric_logger.project="llama3_lora" \
log_every_n_steps=5
Enable W&B logging on the recipe’s config:
# inside llama3/8B_lora_single_device.yaml
metric_logger:
_component_: torchtune.utils.metric_logging.WandBLogger
project: llama3_lora
log_every_n_steps: 5
Enable W&B logging on the recipe’s config file by modifying the metric_logger section. Change the _component_ to torchtune.utils.metric_logging.WandBLogger class. You can also pass a project name and log_every_n_steps to customize the logging behavior.
You can also pass any other kwargs as you would to the wandb.init method. For example, if you are working on a team, you can pass the entity argument to the WandBLogger class to specify the team name.
# inside llama3/8B_lora_single_device.yaml
metric_logger:
_component_: torchtune.utils.metric_logging.WandBLogger
project: llama3_lora
entity: my_project
job_type: lora_finetune_single_device
group: my_awesome_experiments
log_every_n_steps: 5
tune run lora_finetune_single_device --config llama3/8B_lora_single_device \
metric_logger._component_=torchtune.utils.metric_logging.WandBLogger \
metric_logger.project="llama3_lora" \
metric_logger.entity="my_project" \
metric_logger.job_type="lora_finetune_single_device" \
metric_logger.group="my_awesome_experiments" \
log_every_n_steps=5
You can explore the W&B dashboard to see the logged metrics. By default W&B logs all of the hyperparameters from the config file and the launch overrides.
W&B captures the resolved config on the Overview tab. W&B also stores the config in YAML format on the Files tab.
Each recipe has its own training loop. Check each individual recipe to see its logged metrics, which include these by default:
| Metric | Description |
|---|---|
loss |
The loss of the model |
lr |
The learning rate |
tokens_per_second |
The tokens per second of the model |
grad_norm |
The gradient norm of the model |
global_step |
Corresponds to the current step in the training loop. Takes into account gradient accumulation, basically every time an optimizer step is taken, the model is updated, the gradients are accumulated and the model is updated once every gradient_accumulation_steps |
global_step is not the same as the number of training steps. It corresponds to the current step in the training loop. Takes into account gradient accumulation, basically every time an optimizer step is taken the global_step is incremented by 1. For example, if the dataloader has 10 batches, gradient accumulation steps is 2 and run for 3 epochs, the optimizer will step 15 times, in this case global_step will range from 1 to 15.The streamlined design of torchtune allows to easily add custom metrics or modify the existing ones. It suffices to modify the corresponding recipe file, for example, computing one could log current_epoch as a percentage of the total number of epochs as following:
# inside `train.py` function in the recipe file
self._metric_logger.log_dict(
{"current_epoch": self.epochs * self.global_step / self._steps_per_epoch},
step=self.global_step,
)
self._metric_logger.* function.The torchtune library supports various checkpoint formats. Depending on the origin of the model you are using, you should switch to the appropriate checkpointer class.
If you want to save the model checkpoints to W&B Artifacts, the simplest solution is to override the save_checkpoint functions inside the corresponding recipe.
Here is an example of how you can override the save_checkpoint function to save the model checkpoints to W&B Artifacts.
def save_checkpoint(self, epoch: int) -> None:
...
## Let's save the checkpoint to W&B
## depending on the Checkpointer Class the file will be named differently
## Here is an example for the full_finetune case
checkpoint_file = Path.joinpath(
self._checkpointer._output_dir, f"torchtune_model_{epoch}"
).with_suffix(".pt")
wandb_artifact = wandb.Artifact(
name=f"torchtune_model_{epoch}",
type="model",
# description of the model checkpoint
description="Model checkpoint",
# you can add whatever metadata you want as a dict
metadata={
utils.SEED_KEY: self.seed,
utils.EPOCHS_KEY: self.epochs_run,
utils.TOTAL_EPOCHS_KEY: self.total_epochs,
utils.MAX_STEPS_KEY: self.max_steps_per_epoch,
},
)
wandb_artifact.add_file(checkpoint_file)
wandb.log_artifact(wandb_artifact)
Ignite supports Weights & Biases handler to log metrics, model/optimizer parameters, gradients during training and validation. It can also be used to log model checkpoints to the Weights & Biases cloud. This class is also a wrapper for the wandb module. This means that you can call any wandb function using this wrapper. See examples on how to save model parameters and gradients.
from argparse import ArgumentParser
import wandb
import torch
from torch import nn
from torch.optim import SGD
from torch.utils.data import DataLoader
import torch.nn.functional as F
from torchvision.transforms import Compose, ToTensor, Normalize
from torchvision.datasets import MNIST
from ignite.engine import Events, create_supervised_trainer, create_supervised_evaluator
from ignite.metrics import Accuracy, Loss
from tqdm import tqdm
class Net(nn.Module):
def __init__(self):
super(Net, self).__init__()
self.conv1 = nn.Conv2d(1, 10, kernel_size=5)
self.conv2 = nn.Conv2d(10, 20, kernel_size=5)
self.conv2_drop = nn.Dropout2d()
self.fc1 = nn.Linear(320, 50)
self.fc2 = nn.Linear(50, 10)
def forward(self, x):
x = F.relu(F.max_pool2d(self.conv1(x), 2))
x = F.relu(F.max_pool2d(self.conv2_drop(self.conv2(x)), 2))
x = x.view(-1, 320)
x = F.relu(self.fc1(x))
x = F.dropout(x, training=self.training)
x = self.fc2(x)
return F.log_softmax(x, dim=-1)
def get_data_loaders(train_batch_size, val_batch_size):
data_transform = Compose([ToTensor(), Normalize((0.1307,), (0.3081,))])
train_loader = DataLoader(MNIST(download=True, root=".", transform=data_transform, train=True),
batch_size=train_batch_size, shuffle=True)
val_loader = DataLoader(MNIST(download=False, root=".", transform=data_transform, train=False),
batch_size=val_batch_size, shuffle=False)
return train_loader, val_loader
Using WandBLogger in ignite is a modular process. First, you create a WandBLogger object. Next, you attach it to a trainer or evaluator to automatically log the metrics. This example:
from ignite.contrib.handlers.wandb_logger import *
def run(train_batch_size, val_batch_size, epochs, lr, momentum, log_interval):
train_loader, val_loader = get_data_loaders(train_batch_size, val_batch_size)
model = Net()
device = 'cpu'
if torch.cuda.is_available():
device = 'cuda'
optimizer = SGD(model.parameters(), lr=lr, momentum=momentum)
trainer = create_supervised_trainer(model, optimizer, F.nll_loss, device=device)
evaluator = create_supervised_evaluator(model,
metrics={'accuracy': Accuracy(),
'nll': Loss(F.nll_loss)},
device=device)
desc = "ITERATION - loss: {:.2f}"
pbar = tqdm(
initial=0, leave=False, total=len(train_loader),
desc=desc.format(0)
)
#WandBlogger Object Creation
wandb_logger = WandBLogger(
project="pytorch-ignite-integration",
name="cnn-mnist",
config={"max_epochs": epochs,"batch_size":train_batch_size},
tags=["pytorch-ignite", "mninst"]
)
wandb_logger.attach_output_handler(
trainer,
event_name=Events.ITERATION_COMPLETED,
tag="training",
output_transform=lambda loss: {"loss": loss}
)
wandb_logger.attach_output_handler(
evaluator,
event_name=Events.EPOCH_COMPLETED,
tag="training",
metric_names=["nll", "accuracy"],
global_step_transform=lambda *_: trainer.state.iteration,
)
wandb_logger.attach_opt_params_handler(
trainer,
event_name=Events.ITERATION_STARTED,
optimizer=optimizer,
param_name='lr' # optional
)
wandb_logger.watch(model)
You can optionally utilize ignite EVENTS to log the metrics directly to the terminal
@trainer.on(Events.ITERATION_COMPLETED(every=log_interval))
def log_training_loss(engine):
pbar.desc = desc.format(engine.state.output)
pbar.update(log_interval)
@trainer.on(Events.EPOCH_COMPLETED)
def log_training_results(engine):
pbar.refresh()
evaluator.run(train_loader)
metrics = evaluator.state.metrics
avg_accuracy = metrics['accuracy']
avg_nll = metrics['nll']
tqdm.write(
"Training Results - Epoch: {} Avg accuracy: {:.2f} Avg loss: {:.2f}"
.format(engine.state.epoch, avg_accuracy, avg_nll)
)
@trainer.on(Events.EPOCH_COMPLETED)
def log_validation_results(engine):
evaluator.run(val_loader)
metrics = evaluator.state.metrics
avg_accuracy = metrics['accuracy']
avg_nll = metrics['nll']
tqdm.write(
"Validation Results - Epoch: {} Avg accuracy: {:.2f} Avg loss: {:.2f}"
.format(engine.state.epoch, avg_accuracy, avg_nll))
pbar.n = pbar.last_print_n = 0
trainer.run(train_loader, max_epochs=epochs)
pbar.close()
if __name__ == "__main__":
parser = ArgumentParser()
parser.add_argument('--batch_size', type=int, default=64,
help='input batch size for training (default: 64)')
parser.add_argument('--val_batch_size', type=int, default=1000,
help='input batch size for validation (default: 1000)')
parser.add_argument('--epochs', type=int, default=10,
help='number of epochs to train (default: 10)')
parser.add_argument('--lr', type=float, default=0.01,
help='learning rate (default: 0.01)')
parser.add_argument('--momentum', type=float, default=0.5,
help='SGD momentum (default: 0.5)')
parser.add_argument('--log_interval', type=int, default=10,
help='how many batches to wait before logging training status')
args = parser.parse_args()
run(args.batch_size, args.val_batch_size, args.epochs, args.lr, args.momentum, args.log_interval)
This code generates these visualizations::
Refer to the Ignite Docs for more details.
PyTorch Lightning provides a lightweight wrapper for organizing your PyTorch code and easily adding advanced features such as distributed training and 16-bit precision. W&B provides a lightweight wrapper for logging your ML experiments. But you don’t need to combine the two yourself: Weights & Biases is incorporated directly into the PyTorch Lightning library via the WandbLogger.
from lightning.pytorch.loggers import WandbLogger
from lightning.pytorch import Trainer
wandb_logger = WandbLogger(log_model="all")
trainer = Trainer(logger=wandb_logger)
Using wandb.log(): The WandbLogger logs to W&B using the Trainer’s global_step. If you make additional calls to wandb.log directly in your code, do not use the step argument in wandb.log().
Instead, log the Trainer’s global_step like your other metrics:
wandb.log({"accuracy":0.99, "trainer/global_step": step})
import lightning as L
from wandb.integration.lightning.fabric import WandbLogger
wandb_logger = WandbLogger(log_model="all")
fabric = L.Fabric(loggers=[wandb_logger])
fabric.launch()
fabric.log_dict({"important_metric": important_metric})
Sign up for a free account.
Run example commands to:
pip to install the wandb library.In your browser, find your API key on the Authorize page.
If you are using Weights and Biases for the first time you might want to check out our quickstart
pip install wandb
wandb login
!pip install wandb
import wandb
wandb.login()
WandbLoggerPyTorch Lightning has multiple WandbLogger classes to log metrics and model weights, media, and more.
To integrate with Lightning, instantiate the WandbLogger and pass it to Lightning’s Trainer or Fabric.
trainer = Trainer(logger=wandb_logger)
fabric = L.Fabric(loggers=[wandb_logger])
fabric.launch()
fabric.log_dict({
"important_metric": important_metric
})
Below are some of the most used parameters in WandbLogger. Review the PyTorch Lightning documentation for details about all logger arguments.
| Parameter | Description |
|---|---|
project |
Define what wandb Project to log to |
name |
Give a name to your wandb run |
log_model |
Log all models if log_model="all" or at end of training if log_model=True |
save_dir |
Path where data is saved |
class LitModule(LightningModule):
def __init__(self, *args, **kwarg):
self.save_hyperparameters()
wandb_logger.log_hyperparams(
{
"hyperparameter_1": hyperparameter_1,
"hyperparameter_2": hyperparameter_2,
}
)
# add one parameter
wandb_logger.experiment.config["key"] = value
# add multiple parameters
wandb_logger.experiment.config.update({key1: val1, key2: val2})
# use directly wandb module
wandb.config["key"] = value
wandb.config.update()
You can pass your model object to wandblogger.watch() to monitor your models’s gradients and parameters as you train. See the PyTorch Lightning WandbLogger documentation
You can log your metrics to W&B when using the WandbLogger by calling self.log('my_metric_name', metric_vale) within your LightningModule, such as in your training_step or validation_step methods.
The code snippet below shows how to define your LightningModule to log your metrics and your LightningModule hyperparameters. This example uses the torchmetrics library to calculate your metrics
import torch
from torch.nn import Linear, CrossEntropyLoss, functional as F
from torch.optim import Adam
from torchmetrics.functional import accuracy
from lightning.pytorch import LightningModule
class My_LitModule(LightningModule):
def __init__(self, n_classes=10, n_layer_1=128, n_layer_2=256, lr=1e-3):
"""method used to define the model parameters"""
super().__init__()
# mnist images are (1, 28, 28) (channels, width, height)
self.layer_1 = Linear(28 * 28, n_layer_1)
self.layer_2 = Linear(n_layer_1, n_layer_2)
self.layer_3 = Linear(n_layer_2, n_classes)
self.loss = CrossEntropyLoss()
self.lr = lr
# save hyper-parameters to self.hparams (auto-logged by W&B)
self.save_hyperparameters()
def forward(self, x):
"""method used for inference input -> output"""
# (b, 1, 28, 28) -> (b, 1*28*28)
batch_size, channels, width, height = x.size()
x = x.view(batch_size, -1)
# let's do 3 x (linear + relu)
x = F.relu(self.layer_1(x))
x = F.relu(self.layer_2(x))
x = self.layer_3(x)
return x
def training_step(self, batch, batch_idx):
"""needs to return a loss from a single batch"""
_, loss, acc = self._get_preds_loss_accuracy(batch)
# Log loss and metric
self.log("train_loss", loss)
self.log("train_accuracy", acc)
return loss
def validation_step(self, batch, batch_idx):
"""used for logging metrics"""
preds, loss, acc = self._get_preds_loss_accuracy(batch)
# Log loss and metric
self.log("val_loss", loss)
self.log("val_accuracy", acc)
return preds
def configure_optimizers(self):
"""defines model optimizer"""
return Adam(self.parameters(), lr=self.lr)
def _get_preds_loss_accuracy(self, batch):
"""convenience function since train/valid/test steps are similar"""
x, y = batch
logits = self(x)
preds = torch.argmax(logits, dim=1)
loss = self.loss(logits, y)
acc = accuracy(preds, y)
return preds, loss, acc
import lightning as L
import torch
import torchvision as tv
from wandb.integration.lightning.fabric import WandbLogger
import wandb
fabric = L.Fabric(loggers=[wandb_logger])
fabric.launch()
model = tv.models.resnet18()
optimizer = torch.optim.SGD(model.parameters(), lr=lr)
model, optimizer = fabric.setup(model, optimizer)
train_dataloader = fabric.setup_dataloaders(
torch.utils.data.DataLoader(train_dataset, batch_size=batch_size)
)
model.train()
for epoch in range(num_epochs):
for batch in train_dataloader:
optimizer.zero_grad()
loss = model(batch)
loss.backward()
optimizer.step()
fabric.log_dict({"loss": loss})
Using wandb’s define_metric function you can define whether you’d like your W&B summary metric to display the min, max, mean or best value for that metric. If define_metric _ isn’t used, then the last value logged with appear in your summary metrics. See the define_metric reference docs here and the guide here for more.
To tell W&B to keep track of the max validation accuracy in the W&B summary metric, call wandb.define_metric only once, at the beginning of training:
class My_LitModule(LightningModule):
...
def validation_step(self, batch, batch_idx):
if trainer.global_step == 0:
wandb.define_metric("val_accuracy", summary="max")
preds, loss, acc = self._get_preds_loss_accuracy(batch)
# Log loss and metric
self.log("val_loss", loss)
self.log("val_accuracy", acc)
return preds
wandb.define_metric("val_accuracy", summary="max")
fabric = L.Fabric(loggers=[wandb_logger])
fabric.launch()
fabric.log_dict({"val_accuracy": val_accuracy})
To save model checkpoints as W&B Artifacts,
use the Lightning ModelCheckpoint callback and set the log_model argument in the WandbLogger.
trainer = Trainer(logger=wandb_logger, callbacks=[checkpoint_callback])
fabric = L.Fabric(loggers=[wandb_logger], callbacks=[checkpoint_callback])
The latest and best aliases are automatically set to easily retrieve a model checkpoint from a W&B Artifact:
# reference can be retrieved in artifacts panel
# "VERSION" can be a version (ex: "v2") or an alias ("latest or "best")
checkpoint_reference = "USER/PROJECT/MODEL-RUN_ID:VERSION"
# download checkpoint locally (if not already cached)
wandb_logger.download_artifact(checkpoint_reference, artifact_type="model")
# download checkpoint locally (if not already cached)
run = wandb.init(project="MNIST")
artifact = run.use_artifact(checkpoint_reference, type="model")
artifact_dir = artifact.download()
# load checkpoint
model = LitModule.load_from_checkpoint(Path(artifact_dir) / "model.ckpt")
# Request the raw checkpoint
full_checkpoint = fabric.load(Path(artifact_dir) / "model.ckpt")
model.load_state_dict(full_checkpoint["model"])
optimizer.load_state_dict(full_checkpoint["optimizer"])
The model checkpoints you log are viewable through the W&B Artifacts UI, and include the full model lineage (see an example model checkpoint in the UI here).
To bookmark your best model checkpoints and centralize them across your team, you can link them to the W&B Model Registry.
Here you can organize your best models by task, manage model lifecycle, facilitate easy tracking and auditing throughout the ML lifecyle, and automate downstream actions with webhooks or jobs.
The WandbLogger has log_image, log_text and log_table methods for logging media.
You can also directly call wandb.log or trainer.logger.experiment.log to log other media types such as Audio, Molecules, Point Clouds, 3D Objects and more.
# using tensors, numpy arrays or PIL images
wandb_logger.log_image(key="samples", images=[img1, img2])
# adding captions
wandb_logger.log_image(key="samples", images=[img1, img2], caption=["tree", "person"])
# using file path
wandb_logger.log_image(key="samples", images=["img_1.jpg", "img_2.jpg"])
# using .log in the trainer
trainer.logger.experiment.log(
{"samples": [wandb.Image(img, caption=caption) for (img, caption) in my_images]},
step=current_trainer_global_step,
)
# data should be a list of lists
columns = ["input", "label", "prediction"]
my_data = [["cheese", "english", "english"], ["fromage", "french", "spanish"]]
# using columns and data
wandb_logger.log_text(key="my_samples", columns=columns, data=my_data)
# using a pandas DataFrame
wandb_logger.log_text(key="my_samples", dataframe=my_dataframe)
# log a W&B Table that has a text caption, an image and audio
columns = ["caption", "image", "sound"]
# data should be a list of lists
my_data = [
["cheese", wandb.Image(img_1), wandb.Audio(snd_1)],
["wine", wandb.Image(img_2), wandb.Audio(snd_2)],
]
# log the Table
wandb_logger.log_table(key="my_samples", columns=columns, data=data)
You can use Lightning’s Callbacks system to control when you log to Weights & Biases via the WandbLogger, in this example we log a sample of our validation images and predictions:
import torch
import wandb
import lightning.pytorch as pl
from lightning.pytorch.loggers import WandbLogger
# or
# from wandb.integration.lightning.fabric import WandbLogger
class LogPredictionSamplesCallback(Callback):
def on_validation_batch_end(
self, trainer, pl_module, outputs, batch, batch_idx, dataloader_idx
):
"""Called when the validation batch ends."""
# `outputs` comes from `LightningModule.validation_step`
# which corresponds to our model predictions in this case
# Let's log 20 sample image predictions from the first batch
if batch_idx == 0:
n = 20
x, y = batch
images = [img for img in x[:n]]
captions = [
f"Ground Truth: {y_i} - Prediction: {y_pred}"
for y_i, y_pred in zip(y[:n], outputs[:n])
]
# Option 1: log images with `WandbLogger.log_image`
wandb_logger.log_image(key="sample_images", images=images, caption=captions)
# Option 2: log images and predictions as a W&B Table
columns = ["image", "ground truth", "prediction"]
data = [
[wandb.Image(x_i), y_i, y_pred] or x_i,
y_i,
y_pred in list(zip(x[:n], y[:n], outputs[:n])),
]
wandb_logger.log_table(key="sample_table", columns=columns, data=data)
trainer = pl.Trainer(callbacks=[LogPredictionSamplesCallback()])
PyTorch Lightning has Multi-GPU support through their DDP Interface. However, PyTorch Lightning’s design requires you to be careful about how you instantiate our GPUs.
Lightning assumes that each GPU (or Rank) in your training loop must be instantiated in exactly the same way - with the same initial conditions. However, only rank 0 process gets access to the wandb.run object, and for non-zero rank processes: wandb.run = None. This could cause your non-zero processes to fail. Such a situation can put you in a deadlock because rank 0 process will wait for the non-zero rank processes to join, which have already crashed.
For this reason, be careful about how we set up your training code. The recommended way to set it up would be to have your code be independent of the wandb.run object.
class MNISTClassifier(pl.LightningModule):
def __init__(self):
super(MNISTClassifier, self).__init__()
self.model = nn.Sequential(
nn.Flatten(),
nn.Linear(28 * 28, 128),
nn.ReLU(),
nn.Linear(128, 10),
)
self.loss = nn.CrossEntropyLoss()
def forward(self, x):
return self.model(x)
def training_step(self, batch, batch_idx):
x, y = batch
y_hat = self.forward(x)
loss = self.loss(y_hat, y)
self.log("train/loss", loss)
return {"train_loss": loss}
def validation_step(self, batch, batch_idx):
x, y = batch
y_hat = self.forward(x)
loss = self.loss(y_hat, y)
self.log("val/loss", loss)
return {"val_loss": loss}
def configure_optimizers(self):
return torch.optim.Adam(self.parameters(), lr=0.001)
def main():
# Setting all the random seeds to the same value.
# This is important in a distributed training setting.
# Each rank will get its own set of initial weights.
# If they don't match up, the gradients will not match either,
# leading to training that may not converge.
pl.seed_everything(1)
train_loader = DataLoader(train_dataset, batch_size=64, shuffle=True, num_workers=4)
val_loader = DataLoader(val_dataset, batch_size=64, shuffle=False, num_workers=4)
model = MNISTClassifier()
wandb_logger = WandbLogger(project="<project_name>")
callbacks = [
ModelCheckpoint(
dirpath="checkpoints",
every_n_train_steps=100,
),
]
trainer = pl.Trainer(
max_epochs=3, gpus=2, logger=wandb_logger, strategy="ddp", callbacks=callbacks
)
trainer.fit(model, train_loader, val_loader)
You can follow along in a video tutorial with a Colab here.
The core integration is based on the Lightning loggers API, which lets you write much of your logging code in a framework-agnostic way. Loggers are passed to the Lightning Trainer and are triggered based on that API’s rich hook-and-callback system. This keeps your research code well-separated from engineering and logging code.
We’ll save your model checkpoints to W&B, where you can view them or download them for use in future runs. We’ll also capture system metrics, like GPU usage and network I/O, environment information, like hardware and OS information, code state (including git commit and diff patch, notebook contents and session history), and anything printed to the standard out.
wandb.run in my training setup?You need to expand the scope of the variable you need to access yourself. In other words, make sure that the initial conditions are the same on all processes.
if os.environ.get("LOCAL_RANK", None) is None:
os.environ["WANDB_DIR"] = wandb.run.dir
If they are, you can use os.environ["WANDB_DIR"] to set up the model checkpoints directory. This way, any non-zero rank process can access wandb.run.dir.
W&B integrates with Ray by offering two lightweight integrations.
WandbLoggerCallback function automatically logs metrics reported to Tune to the Wandb API.setup_wandb() function, which can be used with the function API, automatically initializes the Wandb API with Tune’s training information. You can use the Wandb API as usual. such as by using wandb.log() to log your training process.from ray.air.integrations.wandb import WandbLoggerCallback
Wandb configuration is done by passing a wandb key to the config parameter of tune.run() (see example below).
The content of the wandb config entry is passed to wandb.init() as keyword arguments. The exception are the following settings, which are used to configure the WandbLoggerCallback itself:
project (str): Name of the Wandb project. Mandatory.
api_key_file (str): Path to file containing the Wandb API KEY.
api_key (str): Wandb API Key. Alternative to setting api_key_file.
excludes (list): List of metrics to exclude from the log.
log_config (bool): Whether to log the config parameter of the results dictionary. Defaults to False.
upload_checkpoints (bool): If True, model checkpoints are uploaded as artifacts. Defaults to False.
from ray import tune, train
from ray.air.integrations.wandb import WandbLoggerCallback
def train_fc(config):
for i in range(10):
train.report({"mean_accuracy": (i + config["alpha"]) / 10})
tuner = tune.Tuner(
train_fc,
param_space={
"alpha": tune.grid_search([0.1, 0.2, 0.3]),
"beta": tune.uniform(0.5, 1.0),
},
run_config=train.RunConfig(
callbacks=[
WandbLoggerCallback(
project="<your-project>", api_key="<your-api-key>", log_config=True
)
]
),
)
results = tuner.fit()
from ray.air.integrations.wandb import setup_wandb
This utility function helps initialize Wandb for use with Ray Tune. For basic usage, call setup_wandb() in your training function:
from ray.air.integrations.wandb import setup_wandb
def train_fn(config):
# Initialize wandb
wandb = setup_wandb(config)
for i in range(10):
loss = config["a"] + config["b"]
wandb.log({"loss": loss})
tune.report(loss=loss)
tuner = tune.Tuner(
train_fn,
param_space={
# define search space here
"a": tune.choice([1, 2, 3]),
"b": tune.choice([4, 5, 6]),
# wandb configuration
"wandb": {"project": "Optimization_Project", "api_key_file": "/path/to/file"},
},
)
results = tuner.fit()
We’ve created a few examples for you to see how the integration works:
W&B integrates with Amazon SageMaker, automatically reading hyperparameters, grouping distributed runs, and resuming runs from checkpoints.
W&B looks for a file named secrets.env relative to the training script and loads them into the environment when wandb.init() is called. You can generate a secrets.env file by calling wandb.sagemaker_auth(path="source_dir") in the script you use to launch your experiments. Be sure to add this file to your .gitignore!
If you’re using one of SageMakers preconfigured estimators you need to add a requirements.txt to your source directory that includes wandb
wandb
If you’re using an estimator that’s running Python 2, you’ll need to install psutil directly from this wheel before installing wandb:
https://wheels.galaxyproject.org/packages/psutil-5.4.8-cp27-cp27mu-manylinux1_x86_64.whl
wandb
Review a complete example on GitHub, and read more on our blog.
You can also read the tutorial on deploying a sentiment analyzer using SageMaker and W&B.
The W&B sweep agent will only behave as expected in a SageMaker job if our SageMaker integration is turned off. You can turn off the SageMaker integration in your runs by modifying your invocation of wandb.init as follows:
wandb.init(..., settings=wandb.Settings(sagemaker_disable=True))
You can use wandb to visualize and compare your scikit-learn models’ performance with just a few lines of code. Try an example →
To get started:
Sign up for a free account.
Use this code to install the wandb library and sign in to your wandb account.
Find your API key on the Authorize page.
If you are using Weights and Biases for the first time,check out a quickstart
pip install wandb
wandb login
!pip install wandb
wandb.login()
import wandb
wandb.init(project="visualize-sklearn")
y_pred = clf.predict(X_test)
accuracy = sklearn.metrics.accuracy_score(y_true, y_pred)
# If logging metrics over time, then use wandb.log
wandb.log({"accuracy": accuracy})
# OR to log a final metric at the end of training you can also use wandb.summary
wandb.summary["accuracy"] = accuracy
import wandb
wandb.init(project="visualize-sklearn")
After training a model and making predictions you can then generate plots in wandb to analyze your predictions. See the Supported Plots section below for a full list of supported charts
# Visualize single plot
wandb.sklearn.plot_confusion_matrix(y_true, y_pred, labels)
W&B has functions such as plot_classifier that will plot several relevant plots:
# Visualize all classifier plots
wandb.sklearn.plot_classifier(
clf,
X_train,
X_test,
y_train,
y_test,
y_pred,
y_probas,
labels,
model_name="SVC",
feature_names=None,
)
# All regression plots
wandb.sklearn.plot_regressor(reg, X_train, X_test, y_train, y_test, model_name="Ridge")
# All clustering plots
wandb.sklearn.plot_clusterer(
kmeans, X_train, cluster_labels, labels=None, model_name="KMeans"
)
Plots created on Matplotlib can also be logged on W&B dashboard. To do that, it is first required to install plotly.
pip install plotly
Finally, the plots can be logged on W&B’s dashboard as follows:
import matplotlib.pyplot as plt
import wandb
wandb.init(project="visualize-sklearn")
# do all the plt.plot(), plt.scatter(), etc. here.
# ...
# instead of doing plt.show() do:
wandb.log({"plot": plt})
Trains model on datasets of varying lengths and generates a plot of cross validated scores vs dataset size, for both training and test sets.
wandb.sklearn.plot_learning_curve(model, X, y)
ROC curves plot true positive rate (y-axis) vs false positive rate (x-axis). The ideal score is a TPR = 1 and FPR = 0, which is the point on the top left. Typically we calculate the area under the ROC curve (AUC-ROC), and the greater the AUC-ROC the better.
wandb.sklearn.plot_roc(y_true, y_probas, labels)
Plots the distribution of target classes in training and test sets. Useful for detecting imbalanced classes and ensuring that one class doesn’t have a disproportionate influence on the model.
wandb.sklearn.plot_class_proportions(y_train, y_test, ['dog', 'cat', 'owl'])
Computes the tradeoff between precision and recall for different thresholds. A high area under the curve represents both high recall and high precision, where high precision relates to a low false positive rate, and high recall relates to a low false negative rate.
High scores for both show that the classifier is returning accurate results (high precision), as well as returning a majority of all positive results (high recall). PR curve is useful when the classes are very imbalanced.
wandb.sklearn.plot_precision_recall(y_true, y_probas, labels)
Evaluates and plots the importance of each feature for the classification task. Only works with classifiers that have a feature_importances_ attribute, like trees.
wandb.sklearn.plot_feature_importances(model, ['width', 'height, 'length'])
Plots how well calibrated the predicted probabilities of a classifier are and how to calibrate an uncalibrated classifier. Compares estimated predicted probabilities by a baseline logistic regression model, the model passed as an argument, and by both its isotonic calibration and sigmoid calibrations.
The closer the calibration curves are to a diagonal the better. A transposed sigmoid like curve represents an overfitted classifier, while a sigmoid like curve represents an underfitted classifier. By training isotonic and sigmoid calibrations of the model and comparing their curves we can figure out whether the model is over or underfitting and if so which calibration (sigmoid or isotonic) might help fix this.
For more details, check out sklearn’s docs.
wandb.sklearn.plot_calibration_curve(clf, X, y, 'RandomForestClassifier')
Computes the confusion matrix to evaluate the accuracy of a classification. It’s useful for assessing the quality of model predictions and finding patterns in the predictions the model gets wrong. The diagonal represents the predictions the model got right, such as where the actual label is equal to the predicted label.
wandb.sklearn.plot_confusion_matrix(y_true, y_pred, labels)
mse, mae, and r2 score.f1, accuracy, precision, and recall.wandb.sklearn.plot_summary_metrics(model, X_train, y_train, X_test, y_test)
Measures and plots the percentage of variance explained as a function of the number of clusters, along with training times. Useful in picking the optimal number of clusters.
wandb.sklearn.plot_elbow_curve(model, X_train)
Measures & plots how close each point in one cluster is to points in the neighboring clusters. The thickness of the clusters corresponds to the cluster size. The vertical line represents the average silhouette score of all the points.
Silhouette coefficients near +1 indicate that the sample is far away from the neighboring clusters. A value of 0 indicates that the sample is on or very close to the decision boundary between two neighboring clusters and negative values indicate that those samples might have been assigned to the wrong cluster.
In general we want all silhouette cluster scores to be above average (past the red line) and as close to 1 as possible. We also prefer cluster sizes that reflect the underlying patterns in the data.
wandb.sklearn.plot_silhouette(model, X_train, ['spam', 'not spam'])
Measures a datapoint’s influence on regression model via cook’s distance. Instances with heavily skewed influences could potentially be outliers. Useful for outlier detection.
wandb.sklearn.plot_outlier_candidates(model, X, y)
Measures and plots the predicted target values (y-axis) vs the difference between actual and predicted target values (x-axis), as well as the distribution of the residual error.
Generally, the residuals of a well-fit model should be randomly distributed because good models will account for most phenomena in a data set, except for random error.
wandb.sklearn.plot_residuals(model, X, y)
model (regressor): Takes in a fitted classifier.
X (arr): Training set features.
y (arr): Training set labels.
If you have any questions, we’d love to answer them in our slack community.
This library is based on the Transformers library by Hugging Face. Simple Transformers lets you quickly train and evaluate Transformer models. Only 3 lines of code are needed to initialize a model, train the model, and evaluate a model. It supports Sequence Classification, Token Classification (NER),Question Answering,Language Model Fine-Tuning, Language Model Training, Language Generation, T5 Model, Seq2Seq Tasks , Multi-Modal Classification and Conversational AI.
To use Weights and Biases for visualizing model training. To use this, set a project name for W&B in the wandb_project attribute of the args dictionary. This logs all hyperparameter values, training losses, and evaluation metrics to the given project.
model = ClassificationModel('roberta', 'roberta-base', args={'wandb_project': 'project-name'})
Any additional arguments that go into wandb.init can be passed as wandb_kwargs.
The library is designed to have a separate class for every NLP task. The classes that provide similar functionality are grouped together.
simpletransformers.classification - Includes all Classification models.
ClassificationModelMultiLabelClassificationModelsimpletransformers.ner - Includes all Named Entity Recognition models.
NERModelsimpletransformers.question_answering - Includes all Question Answering models.
QuestionAnsweringModelHere are some minimal examples
model = MultiLabelClassificationModel("distilbert","distilbert-base-uncased",num_labels=6,
args={"reprocess_input_data": True, "overwrite_output_dir": True, "num_train_epochs":epochs,'learning_rate':learning_rate,
'wandb_project': "simpletransformers"},
)
# Train the model
model.train_model(train_df)
# Evaluate the model
result, model_outputs, wrong_predictions = model.eval_model(eval_df)
train_args = {
'learning_rate': wandb.config.learning_rate,
'num_train_epochs': 2,
'max_seq_length': 128,
'doc_stride': 64,
'overwrite_output_dir': True,
'reprocess_input_data': False,
'train_batch_size': 2,
'fp16': False,
'wandb_project': "simpletransformers"
}
model = QuestionAnsweringModel('distilbert', 'distilbert-base-cased', args=train_args)
model.train_model(train_data)
SimpleTransformers provides classes as well as training scripts for all common natural language tasks. Here is the complete list of global arguments that are supported by the library, with their default arguments.
global_args = {
"adam_epsilon": 1e-8,
"best_model_dir": "outputs/best_model",
"cache_dir": "cache_dir/",
"config": {},
"do_lower_case": False,
"early_stopping_consider_epochs": False,
"early_stopping_delta": 0,
"early_stopping_metric": "eval_loss",
"early_stopping_metric_minimize": True,
"early_stopping_patience": 3,
"encoding": None,
"eval_batch_size": 8,
"evaluate_during_training": False,
"evaluate_during_training_silent": True,
"evaluate_during_training_steps": 2000,
"evaluate_during_training_verbose": False,
"fp16": True,
"fp16_opt_level": "O1",
"gradient_accumulation_steps": 1,
"learning_rate": 4e-5,
"local_rank": -1,
"logging_steps": 50,
"manual_seed": None,
"max_grad_norm": 1.0,
"max_seq_length": 128,
"multiprocessing_chunksize": 500,
"n_gpu": 1,
"no_cache": False,
"no_save": False,
"num_train_epochs": 1,
"output_dir": "outputs/",
"overwrite_output_dir": False,
"process_count": cpu_count() - 2 if cpu_count() > 2 else 1,
"reprocess_input_data": True,
"save_best_model": True,
"save_eval_checkpoints": True,
"save_model_every_epoch": True,
"save_steps": 2000,
"save_optimizer_and_scheduler": True,
"silent": False,
"tensorboard_dir": None,
"train_batch_size": 8,
"use_cached_eval_features": False,
"use_early_stopping": False,
"use_multiprocessing": True,
"wandb_kwargs": {},
"wandb_project": None,
"warmup_ratio": 0.06,
"warmup_steps": 0,
"weight_decay": 0,
}
Refer to simpletransformers on github for more detailed documentation.
Checkout this Weights and Biases report that covers training transformers on some the most popular GLUE benchmark datasets. Try it out yourself on colab.
You can use Weights & Biases with Skorch to automatically log the model with the best performance, along with all model performance metrics, the model topology and compute resources after each epoch. Every file saved in wandb_run.dir is automatically logged to W&B servers.
See example run.
| Parameter | Type | Description |
|---|---|---|
wandb_run |
wandb.wandb_run. Run |
wandb run used to log data. |
save_model |
bool (default=True) | Whether to save a checkpoint of the best model and upload it to your Run on W&B servers. |
keys_ignored |
str or list of str (default=None) | Key or list of keys that should not be logged to tensorboard. Note that in addition to the keys provided by the user, keys such as those starting with event_ or ending on _best are ignored by default. |
We’ve created a few examples for you to see how the integration works:
# Install wandb
... pip install wandb
import wandb
from skorch.callbacks import WandbLogger
# Create a wandb Run
wandb_run = wandb.init()
# Alternative: Create a wandb Run without a W&B account
wandb_run = wandb.init(anonymous="allow")
# Log hyper-parameters (optional)
wandb_run.config.update({"learning rate": 1e-3, "batch size": 32})
net = NeuralNet(..., callbacks=[WandbLogger(wandb_run)])
net.fit(X, y)
| Method | Description |
|---|---|
initialize() |
(Re-)Set the initial state of the callback. |
on_batch_begin(net[, X, y, training]) |
Called at the beginning of each batch. |
on_batch_end(net[, X, y, training]) |
Called at the end of each batch. |
on_epoch_begin(net[, dataset_train, …]) |
Called at the beginning of each epoch. |
on_epoch_end(net, **kwargs) |
Log values from the last history step and save best model |
on_grad_computed(net, named_parameters[, X, …]) |
Called once per batch after gradients have been computed but before an update step was performed. |
on_train_begin(net, **kwargs) |
Log model topology and add a hook for gradients |
on_train_end(net[, X, y]) |
Called at the end of training. |
spaCy is a popular “industrial-strength” NLP library: fast, accurate models with a minimum of fuss. As of spaCy v3, Weights and Biases can now be used with spacy train to track your spaCy model’s training metrics as well as to save and version your models and datasets. And all it takes is a few added lines in your configuration.
wandb library and log inpip install wandb
wandb login
!pip install wandb
import wandb
wandb.login()
WandbLogger to your spaCy config filespaCy config files are used to specify all aspects of training, not just logging – GPU allocation, optimizer choice, dataset paths, and more. Minimally, under [training.logger] you need to provide the key @loggers with the value "spacy.WandbLogger.v3", plus a project_name.
[training.logger]
@loggers = "spacy.WandbLogger.v3"
project_name = "my_spacy_project"
remove_config_values = ["paths.train", "paths.dev", "corpora.train.path", "corpora.dev.path"]
log_dataset_dir = "./corpus"
model_log_interval = 1000
| Name | Description |
|---|---|
project_name |
str. The name of the W&B Project. The project will be created automatically if it doesn’t exist yet. |
remove_config_values |
List[str] . A list of values to exclude from the config before it is uploaded to W&B. [] by default. |
model_log_interval |
Optional int. None by default. If set, model versioning with Artifactswill be enabled. Pass in the number of steps to wait between logging model checkpoints. None by default. |
log_dataset_dir |
Optional str. If passed a path, the dataset will be uploaded as an Artifact at the beginning of training. None by default. |
entity |
Optional str . If passed, the run will be created in the specified entity |
run_name |
Optional str . If specified, the run will be created with the specified name. |
Once you have added the WandbLogger to your spaCy training config you can run spacy train as usual.
python -m spacy train \
config.cfg \
--output ./output \
--paths.train ./train \
--paths.dev ./dev
!python -m spacy train \
config.cfg \
--output ./output \
--paths.train ./train \
--paths.dev ./dev
When training begins, a link to your training run’s W&B page will be output which will take you to this run’s experiment tracking dashboard in the Weights & Biases web UI.
Stable Baselines 3 (SB3) is a set of reliable implementations of reinforcement learning algorithms in PyTorch. W&B’s SB3 integration:
Review an example of a SB3 training run with W&B.
from wandb.integration.sb3 import WandbCallback
model.learn(..., callback=WandbCallback())
| Argument | Usage |
|---|---|
verbose |
The verbosity of sb3 output |
model_save_path |
Path to the folder where the model will be saved, The default value is `None` so the model is not logged |
model_save_freq |
Frequency to save the model |
gradient_save_freq |
Frequency to log gradient. The default value is 0 so the gradients are not logged |
The W&B SB3 integration uses the logs output from TensorBoard to log your metrics
import gym
from stable_baselines3 import PPO
from stable_baselines3.common.monitor import Monitor
from stable_baselines3.common.vec_env import DummyVecEnv, VecVideoRecorder
import wandb
from wandb.integration.sb3 import WandbCallback
config = {
"policy_type": "MlpPolicy",
"total_timesteps": 25000,
"env_name": "CartPole-v1",
}
run = wandb.init(
project="sb3",
config=config,
sync_tensorboard=True, # auto-upload sb3's tensorboard metrics
monitor_gym=True, # auto-upload the videos of agents playing the game
save_code=True, # optional
)
def make_env():
env = gym.make(config["env_name"])
env = Monitor(env) # record stats such as returns
return env
env = DummyVecEnv([make_env])
env = VecVideoRecorder(
env,
f"videos/{run.id}",
record_video_trigger=lambda x: x % 2000 == 0,
video_length=200,
)
model = PPO(config["policy_type"], env, verbose=1, tensorboard_log=f"runs/{run.id}")
model.learn(
total_timesteps=config["total_timesteps"],
callback=WandbCallback(
gradient_save_freq=100,
model_save_path=f"models/{run.id}",
verbose=2,
),
)
run.finish()
Upload your TensorBoard logs to the cloud, quickly share your results among colleagues and classmates and keep your analysis in one centralized location.
import wandb
# Start a wandb run with `sync_tensorboard=True`
wandb.init(project="my-project", sync_tensorboard=True)
# Your training code using TensorBoard
...
# [Optional]Finish the wandb run to upload the tensorboard logs to W&B (if running in Notebook)
wandb.finish()
Review an example.
Once your run finishes, you can access your TensorBoard event files in W&B and you can visualize your metrics in native W&B charts, together with additional useful information like the system’s CPU or GPU utilization, the git state, the terminal command the run used, and more.
If you need to log additional custom metrics that aren’t being logged to TensorBoard, you can call wandb.log in your code wandb.log({"custom": 0.8})
Setting the step argument in wandb.log is turned off when syncing Tensorboard. If you’d like to set a different step count, you can log the metrics with a step metric as:
wandb.log({"custom": 0.8, "global_step": global_step})
wandb?If you want more control over how TensorBoard is patched you can call wandb.tensorboard.patch instead of passing sync_tensorboard=True to wandb.init.
import wandb
wandb.tensorboard.patch(root_logdir="<logging_directory>")
wandb.init()
# Finish the wandb run to upload the tensorboard logs to W&B (if running in Notebook)
wandb.finish()
You can pass tensorboard_x=False to this method to ensure vanilla TensorBoard is patched, if you’re using TensorBoard > 1.14 with PyTorch you can pass pytorch=True to ensure it’s patched. Both of these options have smart defaults depending on what versions of these libraries have been imported.
By default, we also sync the tfevents files and any .pbtxt files. This enables us to launch a TensorBoard instance on your behalf. You will see a TensorBoard tab on the run page. This behavior can be turned off by passing save=False to wandb.tensorboard.patch
import wandb
wandb.init()
wandb.tensorboard.patch(save=False, tensorboard_x=True)
# If running in a notebook, finish the wandb run to upload the tensorboard logs to W&B
wandb.finish()
wandb.init or wandb.tensorboard.patch before calling tf.summary.create_file_writer or constructing a SummaryWriter via torch.utils.tensorboard.If you have existing tfevents files stored locally and you would like to import them into W&B, you can run wandb sync log_dir, where log_dir is a local directory containing the tfevents files.
If running your code in a Jupyter or Colab notebook, make sure to call wandb.finish() and the end of your training. This will finish the wandb run and upload the tensorboard logs to W&B so they can be visualized. This is not necessary when running a .py script as wandb finishes automatically when a script finishes.
To run shell commands in a notebook environment, you must prepend a !, as in !wandb sync directoryname.
If you use PyTorch’s TensorBoard integration, you may need to manually upload the PyTorch Profiler JSON file.
wandb.save(glob.glob(f"runs/*.pt.trace.json")[0], base_path=f"runs")
If you’re already using TensorBoard, it’s easy to integrate with wandb.
import tensorflow as tf
import wandb
wandb.init(config=tf.flags.FLAGS, sync_tensorboard=True)
If you need to log additional custom metrics that aren’t being logged to TensorBoard, you can call wandb.log in your code wandb.log({"custom": 0.8})
Setting the step argument in wandb.log is turned off when syncing Tensorboard. If you’d like to set a different step count, you can log the metrics with a step metric as:
wandb.log({"custom": 0.8, "global_step":global_step}, step=global_step)
If you want more control over what gets logged, wandb also provides a hook for TensorFlow estimators. It will log all tf.summary values in the graph.
import tensorflow as tf
import wandb
wandb.init(config=tf.FLAGS)
estimator.train(hooks=[wandb.tensorflow.WandbHook(steps_per_log=1000)])
The simplest way to log metrics in TensorFlow is by logging tf.summary with the TensorFlow logger:
import wandb
with tf.Session() as sess:
# ...
wandb.tensorflow.log(tf.summary.merge_all())
With TensorFlow 2, the recommended way of training a model with a custom loop is via using tf.GradientTape. You can read more about it here. If you want to incorporate wandb to log metrics in your custom TensorFlow training loops you can follow this snippet:
with tf.GradientTape() as tape:
# Get the probabilities
predictions = model(features)
# Calculate the loss
loss = loss_func(labels, predictions)
# Log your metrics
wandb.log("loss": loss.numpy())
# Get the gradients
gradients = tape.gradient(loss, model.trainable_variables)
# Update the weights
optimizer.apply_gradients(zip(gradients, model.trainable_variables))
A full example is available here.
When the cofounders started working on W&B, they were inspired to build a tool for the frustrated TensorBoard users at OpenAI. Here are a few things we’ve focused on improving:
Get started with a free account
We’ve created a few examples for you to see how the integration works:
For those running machine learning experiments in the Julia programming language, a community contributor has created an unofficial set of Julia bindings called wandb.jl that you can use.
You can find examples in the documentation on the wandb.jl repository. Their “Getting Started” example is here:
using Wandb, Dates, Logging
# Start a new run, tracking hyperparameters in config
lg = WandbLogger(project = "Wandb.jl",
name = "wandbjl-demo-$(now())",
config = Dict("learning_rate" => 0.01,
"dropout" => 0.2,
"architecture" => "CNN",
"dataset" => "CIFAR-100"))
# Use LoggingExtras.jl to log to multiple loggers together
global_logger(lg)
# Simulating the training or evaluation loop
for x ∈ 1:50
acc = log(1 + x + rand() * get_config(lg, "learning_rate") + rand() + get_config(lg, "dropout"))
loss = 10 - log(1 + x + rand() + x * get_config(lg, "learning_rate") + rand() + get_config(lg, "dropout"))
# Log metrics from your script to W&B
@info "metrics" accuracy=acc loss=loss
end
# Finish the run
close(lg)
The wandb library has a WandbCallback callback for logging metrics, configs and saved boosters from training with XGBoost. Here you can see a live Weights & Biases dashboard with outputs from the XGBoost WandbCallback.
Logging XGBoost metrics, configs and booster models to Weights & Biases is as easy as passing the WandbCallback to XGBoost:
from wandb.integration.xgboost import WandbCallback
import xgboost as XGBClassifier
...
# Start a wandb run
run = wandb.init()
# Pass WandbCallback to the model
bst = XGBClassifier()
bst.fit(X_train, y_train, callbacks=[WandbCallback(log_model=True)])
# Close your wandb run
run.finish()
You can open this notebook for a comprehensive look at logging with XGBoost and Weights & Biases
WandbCallback referencePassing WandbCallback to a XGBoost model will:
log_model = True)log_feature_importance=True (default).wandb.summary when define_metric=True (default).log_model: (boolean) if True save and upload the model to Weights & Biases Artifacts
log_feature_importance: (boolean) if True log a feature importance bar plot
importance_type: (str) one of {weight, gain, cover, total_gain, total_cover} for tree model. weight for linear model.
define_metric: (boolean) if True (default) capture model performance at the best step, instead of the last step, of training in your wandb.summary.
You can review the source code for WandbCallback.
For additional examples, check out the repository of examples on GitHub.
Attaining the maximum performance out of models requires tuning hyperparameters, like tree depth and learning rate. Weights & Biases includes Sweeps, a powerful toolkit for configuring, orchestrating, and analyzing large hyperparameter testing experiments.
You can also try this XGBoost & Sweeps Python script.
Ultralytics’ YOLOv5 (“You Only Look Once”) model family enables real-time object detection with convolutional neural networks without all the agonizing pain.
Weights & Biases is directly integrated into YOLOv5, providing experiment metric tracking, model and dataset versioning, rich model prediction visualization, and more. It’s as easy as running a single pip install before you run your YOLO experiments.
Simply by installing wandb, you’ll activate the built-in W&B logging features: system metrics, model metrics, and media logged to interactive Dashboards.
pip install wandb
git clone https://github.com/ultralytics/yolov5.git
python yolov5/train.py # train a small network on a small dataset
Just follow the links printed to the standard out by wandb.
By passing a few simple command line arguments to YOLO, you can take advantage of even more W&B features.
--save_period will turn on model versioning. At the end of every save_period epochs, the model weights will be saved to W&B. The best-performing model on the validation set will be tagged automatically.--upload_dataset flag will also upload the dataset for data versioning.--bbox_interval will turn on data visualization. At the end of every bbox_interval epochs, the outputs of the model on the validation set will be uploaded to W&B.python yolov5/train.py --epochs 20 --save_period 1
python yolov5/train.py --epochs 20 --save_period 1 \
--upload_dataset --bbox_interval 1
Here’s what that looks like.
Ultralytics is the home for cutting-edge, state-of-the-art computer vision models for tasks like image classification, object detection, image segmentation, and pose estimation. Not only it hosts YOLOv8, the latest iteration in the YOLO series of real-time object detection models, but other powerful computer vision models such as SAM (Segment Anything Model), RT-DETR, YOLO-NAS, etc. Besides providing implementations of these models, Ultralytics also provides us with out-of-the-box workflows for training, fine-tuning, and applying these models using an easy-to-use API.
Install ultralytics and wandb.
```shell
pip install --upgrade ultralytics==8.0.238 wandb
# or
# conda install ultralytics
```
```bash
!pip install --upgrade ultralytics==8.0.238 wandb
```
The development team has tested the integration with ultralyticsv8.0.238 and below. To report any issues with the integration, create a GitHub issue with the tag yolov8.
This section demonstrates a typical workflow of using an Ultralytics model for training, fine-tuning, and validation and performing experiment tracking, model-checkpointing, and visualization of the model’s performance using W&B.
You can also check out about the integration in this report: Supercharging Ultralytics with W&B
To use the W&B integration with Ultralytics, import the wandb.integration.ultralytics.add_wandb_callback function.
import wandb
from wandb.integration.ultralytics import add_wandb_callback
from ultralytics import YOLO
Initialize the YOLO model of your choice, and invoke the add_wandb_callback function on it before performing inference with the model. This ensures that when you perform training, fine-tuning, validation, or inference, it automatically saves the experiment logs and the images, overlaid with both ground-truth and the respective prediction results using the interactive overlays for computer vision tasks on W&B along with additional insights in a wandb.Table.
# Initialize YOLO Model
model = YOLO("yolov8n.pt")
# Add W&B callback for Ultralytics
add_wandb_callback(model, enable_model_checkpointing=True)
# Train/fine-tune your model
# At the end of each epoch, predictions on validation batches are logged
# to a W&B table with insightful and interactive overlays for
# computer vision tasks
model.train(project="ultralytics", data="coco128.yaml", epochs=5, imgsz=640)
# Finish the W&B run
wandb.finish()
Here’s how experiments tracked using W&B for an Ultralytics training or fine-tuning workflow looks like:
YOLO Fine-tuning Experiments
Here’s how epoch-wise validation results are visualized using a W&B Table:
WandB Validation Visualization Table
This section demonstrates a typical workflow of using an Ultralytics model for inference and visualizing the results using W&B.
You can try out the code in Google Colab: Open in Colab.
You can also check out about the integration in this report: Supercharging Ultralytics with W&B
In order to use the W&B integration with Ultralytics, we need to import the wandb.integration.ultralytics.add_wandb_callback function.
import wandb
from wandb.integration.ultralytics import add_wandb_callback
from ultralytics.engine.model import YOLO
Download a few images to test the integration on. You can use still images, videos, or camera sources. For more information on inference sources, check out the Ultralytics docs.
!wget https://raw.githubusercontent.com/wandb/examples/ultralytics/colabs/ultralytics/assets/img1.png
!wget https://raw.githubusercontent.com/wandb/examples/ultralytics/colabs/ultralytics/assets/img2.png
!wget https://raw.githubusercontent.com/wandb/examples/ultralytics/colabs/ultralytics/assets/img4.png
!wget https://raw.githubusercontent.com/wandb/examples/ultralytics/colabs/ultralytics/assets/img5.png
Next, initialize a W&B run using wandb.init.
# Initialize W&B run
wandb.init(project="ultralytics", job_type="inference")
Next, initialize your desired YOLO model and invoke the add_wandb_callback function on it before you perform inference with the model. This ensures that when you perform inference, it automatically logs the images overlaid with your interactive overlays for computer vision tasks along with additional insights in a wandb.Table.
# Initialize YOLO Model
model = YOLO("yolov8n.pt")
# Add W&B callback for Ultralytics
add_wandb_callback(model, enable_model_checkpointing=True)
# Perform prediction which automatically logs to a W&B Table
# with interactive overlays for bounding boxes, segmentation masks
model(
[
"./assets/img1.jpeg",
"./assets/img3.png",
"./assets/img4.jpeg",
"./assets/img5.jpeg",
]
)
# Finish the W&B run
wandb.finish()
You do not need to explicitly initialize a run using wandb.init() in case of a training or fine-tuning workflow. However, if the code involves only prediction, you must explicitly create a run.
Here’s how the interactive bbox overlay looks:
WandB Image Overlay
You can fine more information on the W&B image overlays here.
YOLOX is an anchor-free version of YOLO with strong performance for object detection. You can use the YOLOX Weights & Biases integration to turn on logging of metrics related to training, validation, and the system, and you can interactively validate predictions with a single command-line argument.
To use YOLOX with Weights & Biases you will first need to sign up for a Weights & Biases account here.
Then just use the --logger wandb command line argument to turn on logging with wandb. Optionally you can also pass all of the arguments that wandb.init would expect, just prepend wandb- to the start of each argument
num_eval_imges controls the number of validation set images and predictions that are logged to Weights & Biases tables for model evaluation.
# login to wandb
wandb login
# call your yolox training script with the `wandb` logger argument
python tools/train.py .... --logger wandb \
wandb-project <project-name> \
wandb-entity <entity>
wandb-name <run-name> \
wandb-id <run-id> \
wandb-save_dir <save-dir> \
wandb-num_eval_imges <num-images> \
wandb-log_checkpoints <bool>
Example dashboard with YOLOX training and validation metrics ->
Any questions or issues about this Weights & Biases integration? Open an issue in the YOLOX repository.