Nope, wandb/local can run in air gapped environments. The only requirement is that the machines that train your models on can connect to this server over the network.
The default docker image runs MySQL and Minio inside of the container and writes all data in sub folders of
/vol . You can configure external MySQL and Object Storage by getting a license. Email [email protected] for more details.
We strive to release upgraded versions of our server at least once a month.
Experiments that are in progress will enter a backoff retry loop and continue attempting to connect for 24 hours.
Make sure you configure external metadata and object stores to avoid risking permanent data loss. There are no backups of the database if the disk runs out of space. The instance will stop working.
A single instance of wandb/local without an external MySQL store will scale to up to 10's concurrent experiments being tracked at once. Instances connected to an external MySQL store will scale to 100's of concurrent runs. If you have a need for tracking more concurrent experiments send us a note at [email protected] to inquire about our multi instance high availability installation options.
If you're unable to connect to your instance you can put it in restore mode by setting the LOCAL_RESTORE environment variable when you start local. If you're starting wandb local using our cli you can do so with
wandb local -e LOCAL_RESTORE=true Look at the logs printed on startup for a temporary username / password to access the instance.
To restore a machine to reporting metrics to our cloud hosted solution, run
wandb login --host=https://api.wandb.ai.
Yes to both. The wandb server needs to be able to read from the bucket in order to generate signed URLs for use by clients, and it needs to have write access in order to update file metadata. Because the server generates temporary signed URLs for use by clients, there’s no need to make the s3 bucket public or explicitly grant permissions to any end-users.