Nope, wandb/local can run in air gapped environments. The only requirement is that the machines that train your models on can connect to this server over the network.
The default docker image runs MySQL and Minio inside of the container and writes all data in sub folders of
/vol . You can configure external MySQL and Object Storage by getting a license. Email [email protected] for more details.
We strive to release upgraded versions of our server at least once a month.
Experiments that are in progress will enter a backoff retry loop and continue attempting to connect for 24 hours.
A single instance of wandb/local without an external MySQL store will scale to up to 10's concurrent experiments being tracked at once. Instances connected to an external MySQL store will scale to 100's of concurrent runs. If you have a need for tracking more concurrent experiments send us a note at [email protected] to inquire about our multi instance high availability installation options.
If you're unable to connect to your instance you can put it in restore mode by setting the LOCAL_RESTORE environment variable when you start local. If you're starting wandb local using our cli you can do so with
wandb local -e LOCAL_RESTORE=true Look at the logs printed on startup for a temporary username / password to access the instance.
To restore a machine to reporting metrics to our cloud hosted solution, run
wandb login --host=https://api.wandb.ai.