Log Data & Media
Call wandb.log(dict) to log a dictionary of metrics or custom objects to a step. Each time you log, we increment the step by default, letting you view metrics over time.
wandb.log({'accuracy': 0.9, 'epoch': 5})
Compare the best accuracy: To compare the best value of a metric across runs, set the summary value for that metric. By default, summary is set to the last value you logged for each key. This is useful in the table in the UI, where you can sort and filter runs based on their summary metrics — so you could compare runs in a table or bar chart based on their best accuracy, instead of final accuracy. For example, you could set summary like so:
wandb.run.summary["accuracy"] = best_accuracyMultiple metrics on one chart: Log multiple metrics in the same call to wandb.log(), like this:
wandb.log({'acc': 0.9, 'loss': 0.1})and they will both be available to plot against in the UICustom x-axis: Add a custom x-axis to the same log call to visualize your metrics against a different axis in the W&B dashboard. For example:
wandb.log({'acc': 0.9, 'custom_step': 3})
View the reference docs, generated from the wandb Python library.
We support images, video, audio, custom graphs, and more. Log rich media to explore your results and visualize comparisons between your runs.
wandb.log({"gradients": wandb.Histogram(numpy_array_or_sequence)})wandb.run.summary.update({"gradients": wandb.Histogram(np_histogram=np.histogram(data))})
If a sequence is provided as the first argument, we will bin the histogram automatically. You can also pass what is returned from np.histogram to the np_histogram keyword argument to do your own binning. The maximum number of bins supported is 512. You can use the optional num_bins keyword argument when passing a sequence to override the default of 64 bins.
If histograms are in your summary they will appear as sparklines on the individual run pages. If they are in your history, we plot a heatmap of bins over time.
wandb.log({"examples": [wandb.Image(numpy_array_or_pil, caption="Label")]})
If a numpy array is supplied we assume it's gray scale if the last dimension is 1, RGB if it's 3, and RGBA if it's 4. If the array contains floats we convert them to ints between 0 and 255. You can specify a mode manually or just supply a PIL.Image. It's recommended to log fewer than 50 images per step.
If you have images with masks for semantic segmentation, you can log the masks and toggle them on and off in the UI. To log multiple masks, log a mask dictionary with multiple keys. Here's an example:
mask_data: a 2D numpy array containing an integer class label for each pixel
class_labels: a dictionary mapping the numbers from mask_data to readable labels
mask_data = np.array([[1, 2, 2, ... , 2, 2, 1], ...])class_labels = {1: "tree",2: "car",3: "road"}mask_img = wandb.Image(image, masks={"predictions": {"mask_data": mask_data,"class_labels": class_labels},"groud_truth": {...},...})
Log bounding boxes with images, and use filters and toggles to dynamically visualize different sets of boxes in the UI.
class_id_to_label = {1: "car",2: "road",3: "building",....}img = wandb.Image(image, boxes={"predictions": {"box_data": [{"position": {"minX": 0.1,"maxX": 0.2,"minY": 0.3,"maxY": 0.4,},"class_id" : 2,"box_caption": "minMax(pixel)","scores" : {"acc": 0.1,"loss": 1.2},},# Log as many boxes as needed...],"class_labels": class_id_to_label},"ground_truth": {# Log each group of boxes with a unique key name...}})wandb.log({"driving_scene": img})
Optional Parameters
class_labels An optional argument mapping your class_ids to string values. By default we will generate class_labels class_0, class_1, etc...
Boxes - Each box passed into box_data can be defined with different coordinate systems.
position
Option 1:
{minX, maxX, minY, maxY}Provide a set of coordinates defining the upper and lower bounds of each box dimension.Option 2:
{middle, width, height}Provide a set of coordinates specifying the middle coordinates as[x,y], andwidth, andheightas scalars
domain Change the domain of your position values based on your data representation
percentage(Default) A relative value representing the percent of the image as distancepixelAn absolute pixel value
wandb.log({"examples": [wandb.Audio(numpy_array, caption="Nice", sample_rate=32)]})
The maximum number of audio clips that can be logged per step is 100.
wandb.log({"video": wandb.Video(numpy_array_or_path_to_video, fps=4, format="gif")})
If a numpy array is supplied we assume the dimensions are: time, channels, width, height. By default we create a 4 fps gif image (ffmpeg and the moviepy python library is required when passing numpy objects). Supported formats are "gif", "mp4", "webm", and "ogg". If you pass a string to wandb.Video we assert the file exists and is a supported format before uploading to wandb. Passing a BytesIO object will create a tempfile with the specified format as the extension.
On the W&B runs page, you will see your videos in the Media section.
Use wandb.Table() to log text in tables to show up in the UI. By default, the column headers are ["Input", "Output", "Expected"]. To ensure optimal UI performance, the default maximum number of rows is set to 10,000. However, users can explicitly override the maximum with wandb.Table.MAX_ROWS = {DESIRED_MAX}.
# Method 1data = [["I love my phone", "1", "1"],["My phone sucks", "0", "-1"]]wandb.log({"examples": wandb.Table(data=data, columns=["Text", "Predicted Label", "True Label"])})# Method 2table = wandb.Table(columns=["Text", "Predicted Label", "True Label"])table.add_data("I love my phone", "1", "1")table.add_data("My phone sucks", "0", "-1")wandb.log({"examples": table})
You can also pass a pandas DataFrame object.
table = wandb.Table(dataframe=my_dataframe)
wandb.log({"custom_file": wandb.Html(open("some.html"))})wandb.log({"custom_string": wandb.Html('<a href="https://mysite">Link</a>')})
Custom html can be logged at any key, this exposes an HTML panel on the run page. By default we inject default styles, you can disable default styles by passing inject=False.
wandb.log({"custom_file": wandb.Html(open("some.html"), inject=False)})
wandb.log({"protein": wandb.Molecule(open("6lu7.pdb"))}
Log molecular data in any of 10 file types:
'pdb', 'pqr', 'mmcif', 'mcif', 'cif', 'sdf', 'sd', 'gro', 'mol2', 'mmtf'
When your run finishes, you'll be able to interact with 3D visualizations of your molecules in the UI.
These presets have builtin wandb.plot methods that make it fast to log charts directly from your script and see the exact visualizations you're looking for in the UI.
wandb.plot.line()
Log a custom line plot—a list of connected and ordered points (x,y) on arbitrary axes x and y.
data = [[x, y] for (x, y) in zip(x_values, y_values)]table = wandb.Table(data=data, columns = ["x", "y"])wandb.log({"my_custom_plot_id" : wandb.plot.line(table, "x", "y",title="Custom Y vs X Line Plot")})
You can use this to log curves on any two dimensions. Note that if you're plotting two lists of values against each other, the number of values in the lists must match exactly (i.e. each point must have an x and a y).
wandb.plot.scatter()
Log a custom scatter plot—a list of points (x, y) on a pair of arbitrary axes x and y.
data = [[x, y] for (x, y) in zip(class_x_scores, class_y_scores)]table = wandb.Table(data=data, columns = ["class_x", "class_y"])wandb.log({"my_custom_id" : wandb.plot.scatter(table,"class_x", "class_y")})
You can use this to log scatter points on any two dimensions. Note that if you're plotting two lists of values against each other, the number of values in the lists must match exactly (i.e. each point must have an x and a y).
wandb.plot.bar()
Log a custom bar chart—a list of labeled values as bars—natively in a few lines:
data = [[label, val] for (label, val) in zip(labels, values)]table = wandb.Table(data=data, columns = ["label", "value"])wandb.log({"my_bar_chart_id" : wandb.plot.bar(table, "label","value", title="Custom Bar Chart")
You can use this to log arbitrary bar charts. Note that the number of labels and values in the lists must match exactly (i.e. each data point must have both).
wandb.plot.histogram()
Log a custom histogram—sort a list of values into bins by count/frequency of occurrence—natively in a few lines. Let's say I have a list of prediction confidence scores (scores) and want to visualize their distribution:
data = [[s] for s in scores]table = wandb.Table(data=data, columns=["scores"])wandb.log({'my_histogram': wandb.plot.histogram(table, "scores",title="Histogram")})
You can use this to log arbitrary histograms. Note that data is a list of lists, intended to support a 2D array of rows and columns.
wandb.plot.pr_curve()
Log a Precision-Recall curve in one line:
wandb.log({"pr" : wandb.plot.pr_curve(ground_truth, predictions,labels=None, classes_to_plot=None)})
You can log this whenever your code has access to:
a model's predicted scores (
predictions) on a set of examplesthe corresponding ground truth labels (
ground_truth) for those examples(optionally) a list of the labels/class names (
labels=["cat", "dog", "bird"...]if label index 0 means cat, 1 = dog, 2 = bird, etc.)(optionally) a subset (still in list format) of the labels to visualize in the plot
wandb.plot.roc_curve()
Log an ROC curve in one line:
wandb.log({"roc" : wandb.plot.roc_curve(ground_truth,predictions, labels=None, classes_to_plot=None)})
You can log this whenever your code has access to:
a model's predicted scores (
predictions) on a set of examplesthe corresponding ground truth labels (
ground_truth) for those examples(optionally) a list of the labels/ class names (
labels=["cat", "dog", "bird"...]if label index 0 means cat, 1 = dog, 2 = bird, etc.)(optionally) a subset (still in list format) of these labels to visualize on the plot
wandb.plot.confusion_matrix()
Log a multi-class confusion matrix in one line:
wandb.log({"conf_mat" : wandb.plot.confusion_matrix(probs=None,y_true=ground_truth,preds=predictions,class_names=class_names})
You can log this wherever your code has access to:
a model's predicted labels on a set of examples (
preds) or the normalized probability scores (probs). The probabilities must have the shape (number of examples, number of classes). You can supply either probabilities or predictions but not both.the corresponding ground truth labels for those examples (
y_true)a full list of the labels/class names as strings (
class_names, e.g.class_names=["cat", "dog", "bird"]if index 0 is cat, 1=dog, 2=bird, etc)
wandb.plot.line_series()
Plot multiple lines, or multiple different lists of x-y coordinate pairs, on one shared set of x-y axes:
wandb.log({"my_custom_id" : wandb.plot.line_series(xs=[0, 1, 2, 3, 4],ys=[[10, 20, 30, 40, 50], [0.5, 11, 72, 3, 41]],keys=["metric Y", "metric Z"],title="Two Random Metrics",xname="x units")})
Note that the number of x and y points must match exactly. You can supply one list of x values to match multiple lists of y values, or a separate list of x values for each list of y values.
Tweak a builtin Custom Chart preset, or create a new preset, then save the chart. Use the chart ID to log data to that custom preset directly from your script.
# Create a table with the columns to plottable = wandb.Table(data=data, columns=["step", "height"])# Map from the table's columns to the chart's fieldsfields = {"x": "step","value": "height"}# Use the table to populate the new custom chart preset# To use your own saved chart preset, change the vega_spec_namemy_custom_chart = wandb.plot_table(vega_spec_name="carey/new_chart",data_table=table,fields=fields,)
import matplotlib.pyplot as pltplt.plot([1, 2, 3, 4])plt.ylabel('some interesting numbers')wandb.log({"chart": plt})
You can pass a matplotlib pyplot or figure object to wandb.log(). By default we'll convert the plot into a Plotly plot. If you want to explicitly log the plot as an image, you can pass the plot into wandb.Image. We also accept directly logging Plotly charts.
Log files in the formats obj, gltf, or glb, and we will render them in the UI when your run finishes.
wandb.log({"generated_samples":[wandb.Object3D(open("sample.obj")),wandb.Object3D(open("sample.gltf")),wandb.Object3D(open("sample.glb"))]})
Log 3D point clouds and Lidar scenes with bounding boxes. Pass in a numpy array containing coordinates and colors for the points to render. In the UI, we truncate to 300,000 points.
point_cloud = np.array([[0, 0, 0, COLOR...], ...])wandb.log({"point_cloud": wandb.Object3D(point_cloud)})
Three different shapes of numpy arrays are supported for flexible color schemes.
[[x, y, z], ...]nx3[[x, y, z, c], ...]nx4| c is a categoryin the range[1, 14](Useful for segmentation)[[x, y, z, r, g, b], ...]nx6 | r,g,bare values in the range[0,255]for red, green, and blue color channels.
Here's an example of logging code below:
pointsis a numpy array with the same format as the simple point cloud renderer shown above.boxesis a numpy array of python dictionaries with three attributes:corners- a list of eight cornerslabel- a string representing the label to be rendered on the box (Optional)color- rgb values representing the color of the box
typeis a string representing the scene type to render. Currently the only supported value islidar/beta
# Log points and boxes in W&Bwandb.log({"point_scene": wandb.Object3D({"type": "lidar/beta","points": np.array([[0.4, 1, 1.3],[1, 1, 1],[1.2, 1, 1.2]]),"boxes": np.array([{"corners": [[0,0,0],[0,1,0],[0,0,1],[1,0,0],[1,1,0],[0,1,1],[1,0,1],[1,1,1]],"label": "Box","color": [123,321,111],},{"corners": [[0,0,0],[0,2,0],[0,0,2],[2,0,0],[2,2,0],[0,2,2],[2,0,2],[2,2,2]],"label": "Box-2","color": [111,321,0],}]),"vectors": [{"start": [0,0,0], "end": [0.1,0.2,0.5]}]})})
If you want to plot your metrics against different x-axes, you can log the step as a metric, like wandb.log({'loss': 0.1, 'epoch': 1, 'batch': 3}). In the UI you can switch between x-axes in the chart settings.
If you want to log to a single history step from lots of different places in your code you can pass a step index to wandb.log() as follows:
wandb.log({'loss': 0.2}, step=step)
As long as you keep passing the same value for step, W&B will collect the keys and values from each call in one unified dictionary. As soon you call wandb.log() with a different value for step than the previous one, W&B will write all the collected keys and values to the history, and start collection over again. Note that this means you should only use this with consecutive values for step: 0, 1, 2, .... This feature doesn't let you write to absolutely any history step that you'd like, only the "current" one and the "next" one.
You can also set commit=False in wandb.log to accumulate metrics, just be sure to call wandb.log without the commit flag to persist the metrics.
wandb.log({'loss': 0.2}, commit=False)# Somewhere else when I'm ready to report this step:wandb.log({'accuracy': 0.8})
The summary statistics are used to track single metrics per model. If a summary metric is modified, only the updated state is saved. We automatically set summary to the last history row added unless you modify it manually. If you change a summary metric, we only persist the last value it was set to.
wandb.init(config=args)best_accuracy = 0for epoch in range(1, args.epochs + 1):test_loss, test_accuracy = test()if (test_accuracy > best_accuracy):wandb.run.summary["best_accuracy"] = test_accuracybest_accuracy = test_accuracy
You may want to store evaluation metrics in a runs summary after training has completed. Summary can handle numpy arrays, pytorch tensors or tensorflow tensors. When a value is one of these types we persist the entire tensor in a binary file and store high level metrics in the summary object such as min, mean, variance, 95% percentile, etc.
api = wandb.Api()run = api.run("username/project/run_id")run.summary["tensor"] = np.random.random(1000)run.summary.update()
The history object is used to track metrics logged by wandb.log. You can access a mutable dictionary of metrics via run.history.row. The row will be saved and a new row created when run.history.add or wandb.log is called.
wandb.init(config=flags.FLAGS)# Start tensorflow trainingwith tf.Session() as sess:sess.run(init)for step in range(1, run.config.num_steps+1):batch_x, batch_y = mnist.train.next_batch(run.config.batch_size)# Run optimization op (backprop)sess.run(train_op, feed_dict={X: batch_x, Y: batch_y})# Calculate batch loss and accuracyloss, acc = sess.run([loss_op, accuracy], feed_dict={X: batch_x, Y: batch_y})wandb.log({'acc': acc, 'loss':loss}) # log accuracy and loss
# Start pytorch trainingwandb.init(config=args)for epoch in range(1, args.epochs + 1):train_loss = train(epoch)test_loss, test_accuracy = test()torch.save(model.state_dict(), 'model')wandb.log({"loss": train_loss, "val_loss": test_loss})
Each time you log images from a step, we save them to show in the UI. Pin the image panel, and use the step slider to look at images from different steps. This makes it easy to compare how a model's output changes over training.
wandb.log({'epoch': epoch, 'val_acc': 0.94})
If you'd like to log certain metrics in every batch and standardize plots, you can log x axis values that you want to plot with your metrics. Then in the custom plots, click edit and select the custom x-axis.
wandb.log({'batch': 1, 'loss': 0.3})
wandb.Image converts numpy arrays or instances of PILImage to PNG's by default.
wandb.log({"example": wandb.Image(...)})# Or multiple imageswandb.log({"example": [wandb.Image(...) for img in images]})
To save a JPEG you can pass a path to a file:
im = PIL.fromarray(...)rgb_im = im.convert('RGB')rgb_im.save('myimage.jpg')wandb.log({"example": wandb.Image("myimage.jpg")})
wandb.log({"example": wandb.Video("myvideo.mp4")})
Now you can view videos in the media browser. Go to your project workspace, run workspace, or report and click "Add visualization" to add a rich media panel.
By default, we increment the global step every time you call wandb.log. If you'd like, you can log your own monotonically increasing step and then select it as a custom x-axis on your graphs.
For example, if you have training and validation steps you'd like to align, pass us your own step counter: wandb.log({"acc":1, "global_step":1}). Then in the graphs choose "global_step" as the x-axis.
wandb.log({"acc":1, "batch":10}, step=epoch) would enable you to choose "batch" as an x axis in addition to the default step axis
You can hold control and use the mouse to move around inside the space
If you're seeing "No visualization data logged yet" that means that we haven't gotten the first wandb.log call from your script yet. This could be because your run takes a long time to finish a step. If you're logging at the end of each epoch, you could log a few times per epoch to see data stream in more quickly.
If you're logging different types under the same key, we have to split them out in the database. This means you'll see multiple entries of the same metric name in a dropdown in the UI. The types we group by are number, string, bool, other (mostly arrays), and any wandb type (histogram, images, etc). Please send only one type to each key to avoid this behavior.
Sampling
The more points you send us, the longer it will take to load your graphs in the UI. If you have more than 1000 points on a line, we sample down to 1000 points on the backend before we send your browser the data. This sampling is nondeterministic, so if you refresh the page you'll see a different set of sampled points.
If you'd like all the original data, you can use our data API to pull down unsampled data.
Guidelines
We recommend that you try to log less than 10,000 points per metric. If you have more than 500 columns of config and summary metrics, we'll only show 500 in the table. If you log more than 1 million points in a line, it will take us while to load the page.
We store metrics in a case-insensitive fashion, so make sure you don't have two metrics with the same name like "My-Metric" and "my-metric".
"I want to integrate W&B in my project, but I don't want to upload any images"
Our integration doesn't automatically upload images— you specify any files you'd like to upload explicitly. Here's a quick example I made for PyTorch where I explicitly log images: http://bit.ly/pytorch-mnist-colab
wandb.log({"Examples": example_images,"Test Accuracy": 100. * correct / len(test_loader.dataset),"Test Loss": test_loss})
