Use these configuration fields to customize your sweep. There are two ways to specify your configuration:
Python data structure: best for running a sweep from a Jupyter Notebook
Top-level key | Meaning |
name | The name of the sweep, displayed in the W&B UI |
description | Text description of the sweep (notes) |
program | Training script to run (required) |
metric | Specify the metric to optimize (used by some search strategies and stopping criteria) |
method | Specify the search strategy (required) |
early_terminate | Specify the stopping criteria (optional, defaults to no early stopping) |
parameters | Specify parameters bounds to search (required) |
project | Specify the project for this sweep |
entity | Specify the entity for this sweep |
command | Specify command line for how the training script should be run |
Specify the metric to optimize. This metric should be logged explicitly to W&B by your training script. For example, if you want to minimize the validation loss of your model:
# [model training code that returns validation loss as valid_loss]wandb.log({"val_loss" : valid_loss})
| Meaning |
name | Name of the metric to optimize |
goal |
|
target | Goal value for the metric you're optimizing. When any run in the sweep achieves that target value, the sweep's state will be set to finished. This means all agents with active runs will finish those jobs, but no new runs will be launched in the sweep. |
⚠️ Can't optimize nested metrics
The metric you're optimizing to be at the top level of the config.
This will NOT work:
Sweep configuration
metric:
name: my_metric.nested
Code
nested_metrics = {"nested": 4} wandb.log({"my_metric", nested_metrics}
Workaround: log the metric at the top level
Sweep configuration
metric:
name: my_metric_nested
Codenested_metrics = {"nested": 4} wandb.log{{"my_metric", nested_metric} wandb.log({"my_metric_nested", nested_metric["nested"]})
Examples
metric:name: val_lossgoal: maximize
metric:name: val_loss
metric:name: val_lossgoal: maximizetarget: 0.1
Specify the search strategy with the method
key in the sweep configuration.
| Meaning |
grid | Grid search iterates over all possible combinations of parameter values. |
random | Random search chooses random sets of values. |
bayes | Bayesian Optimization uses a gaussian process to model the function and then chooses parameters to optimize probability of improvement. This strategy requires a metric key to be specified. |
Examples
method: random
method: grid
method: bayesmetric:name: val_lossgoal: minimize
Early termination is an optional feature that speeds up hyperparameter search by stopping poorly-performing runs. When the early stopping is triggered, the agent stops the current run and gets the next set of hyperparameters to try.
| Meaning |
type | specify the stopping algorithm |
We support the following stopping algorithm(s):
| Meaning |
hyperband | Use the hyperband method |
Hyperband stopping evaluates whether a program should be stopped or permitted to continue at one or more brackets during the execution of the program. Brackets are configured at static iterations for a specified metric
(where an iteration is the number of times a metric has been logged — if the metric is logged every epoch, then there are epoch iterations).
In order to specify the bracket schedule, eithermin_iter
or max_iter
needs to be defined.
| Meaning |
min_iter | specify the iteration for the first bracket |
max_iter | specify the maximum number of iterations for the program |
s | specify the total number of brackets (required for |
eta | specify the bracket multiplier schedule (default: 3) |
Examples
early_terminate:type: hyperbandmin_iter: 3
Brackets: 3, 9 (3*eta), 27 (9 * eta), 81 (27 * eta)
early_terminate:type: hyperbandmax_iter: 27s: 2
Brackets: 9 (27/eta), 3 (9/eta)
Describe the hyperparameters to explore. For each hyperparameter, specify the name and the possible values as a list of constants (for any method) or specify a distribution (for random
or bayes
).
Values | Meaning |
values: [(type1), (type2), ...] | Specifies all valid values for this hyperparameter. Compatible with |
value: (type) | Specifies the single valid value for this hyperparameter. Compatible with |
distribution: (distribution) | Selects a distribution from the distribution table below. If not specified, will default to |
min: (float) max: (float) | Maximum and minimum valid values for |
min: (int) max: (int) | Maximum and minimum values for |
mu: (float) | Mean parameter for |
sigma: (float) | Standard deviation parameter for |
q: (float) | Quantization step size for quantized hyperparameters |
Example
parameter_name:value: 1.618
parameter_name:values:- 8- 6- 7- 5- 3- 0- 9
parameter_name:distribution: normalmu: 100sigma: 10
Name | Meaning |
constant | Constant distribution. Must specify |
categorical | Categorical distribution. Must specify |
int_uniform | Discrete uniform distribution on integers. Must specify |
uniform | Continuous uniform distribution. Must specify |
q_uniform | Quantized uniform distribution. Returns |
log_uniform | Log-uniform distribution. Returns a value between |
q_log_uniform | Quantized log uniform. Returns |
normal | Normal distribution. Return value is normally-distributed with mean |
q_normal | Quantized normal distribution. Returns |
log_normal | Log normal distribution. Returns a value |
q_log_normal | Quantized log normal distribution. Returns |
Example
parameter_name:distribution: constantvalue: 2.71828
parameter_name:distribution: categoricalvalues:- elu- celu- gelu- selu- relu- prelu- lrelu- rrelu- relu6
parameter_name:distribution: uniformmin: 0max: 1
parameter_name:distribution: q_uniformmin: 0max: 256q: 1
The sweep agent constructs a command line in the following format by default:
/usr/bin/env python train.py --param1=value1 --param2=value2
On Windows machines the /usr/bin/env will be omitted. On UNIX systems it ensures the right python interpreter is chosen based on the environment.
This command line can be modified by specifying a command
key in the configuration file.
By default the command is defined as:
command:- ${env}- ${interpreter}- ${program}- ${args}
Command Macro | Expansion |
${env} | /usr/bin/env on UNIX systems, omitted on Windows |
${interpreter| | Expands to "python". |
${program} | Training script specified by the sweep configuration |
${args} | Expanded arguments in the form --param1=value1 --param2=value2 |
${args_no_hyphens} | Expanded arguments in the form param1=value1 param2=value2 |
${args_json} | Arguments encoded as JSON |
${args_json_file} | The path to a file containing the args encoded as JSON |
Examples:
In order to hardcode the python interpreter you can can specify the interpreter explicitly:
command:- ${env}- python3- ${program}- ${args}
Add extra command line arguments not specified by sweep configuration parameters:
command:- ${env}- ${interpreter}- ${program}- "-config"- your-training-config- ${args}
If your program does not use argument parsing you can avoid passing arguments all together and take advantage of wandb.init()
picking up sweep parameters automatically:
command:- ${env}- ${interpreter}- ${program}
You can change the command to pass arguments they way tools like Hydra expect.
command:- ${env}- ${interpreter}- ${program}- ${args_no_hyphens}
Right now, sweeps do not support nested values, but we plan on supporting this in the near future.