WANDB_DIR to a directory on persistent storage to preserve local run data and diagnostic logs in ephemeral environments. Otherwise, files in the local wandb/ directory can be lost when the pod or container is deleted.
See the following related articles:
- How can I define the local location for
wandbfiles? - Why is my run showing as crashed?
- Environment variables