Create a launch job
A job is a blueprint that contains contextual information about a W&B run it is created from; such as the run's source code, software dependencies, hyperparameters, artifact version, and so forth.
Once you have a launch job, you can add them to a pre-configured launch queue. The launch agent that was deployed by you or someone on your team, will poll that queue and send the job (as a Docker image) to the compute resource that was configured on launch queue.
There are three ways to create a launch job:
- With a Python script
- With a Docker image
- With Git repository
The following sections show how to create a job based on each use case.
Before you get started
Before you create a launch job, find out the name of your queue and the entity it belongs to. Then, follow these instructions to find out the status of your queue and to check if an agent is polling that queue:
- Navigate to wandb.ai/launch.
- From the All entities dropdown, select the entity the launch queue belongs to. This will filter queues based on W&B entities.
- From the filtered results, check that the queue exists.
- Hover your mouse to the right of the launch queue and select
View queue
. - Select the Agents tab. Within the Agents tab you sill see a list of Agent IDs and their statuses. Ensure that one of the agent IDs has a polling status.
Create a job with a W&B artifact
- CLI
- Python SDK
Create a launch job with the W&B CLI.
Ensure the path with your Python script has a requirements.txt
file with the Python dependencies required to run your code. A Python runtime is also required. The python runtime can either be specified manually with the runtime parameter or can be auto-detected from a runtime.txt
or .python-version file
.
Copy and paste the following code snippet. Replace the values within "<>"
based on your use case:
wandb job create --project "<project-name>" -e "<your-entity>" \
--name "<name-for-job>" code "<path-to-script/code.py>"
For a full list of flags you can use, see the wandb job create
command documentation.
You do not need to use the run.log_code()
function within your Python script when you create a launch job with the W&B CLI.
Log your code as an artifact to create a launch job. To do so, log your code to your run as an artifact with run.log_code()
.
The following sample Python code shows how to integrate the run.log_code()
function (see highlighted portion) into a Python script.
import random
import wandb
def run_training_run(epochs, lr):
settings = wandb.Settings(job_source="artifact")
run = wandb.init(
project="launch_demo",
job_type="eval",
settings=settings,
entity="<your-entity>",
# Simulate tracking hyperparameters
config={
"learning_rate": lr,
"epochs": epochs,
},
)
offset = random.random() / 5
print(f"lr: {lr}")
for epoch in range(2, epochs):
# simulating a training run
acc = 1 - 2**-epoch - random.random() / epoch - offset
loss = 2**-epoch + random.random() / epoch + offset
wandb.log({"acc": acc, "loss": loss})
run.log_code()
run.finish()
run_training_run(epochs=10, lr=0.01)
You can specify a name for your job with the WANDB_JOB_NAME
environment variable. You can also specify a name by setting the job_name
parameter in wandb.Settings
and passing it to wandb.init
. For example:
settings = wandb.Settings(job_name="my-job-name")
wandb.init(settings=settings)
If you do not specify a name, W&B will automatically generate a launch job name for you. It will create a job name with the following format: job-<code-artifact-name>
.
For more information on the run.log_code()
command, see the API Reference guide.
Create a job with a Docker image
Create a job with a Docker image with the W&B CLI or by creating a Docker container from the image. To create an image-based job, you must first create the Docker image. The Docker image should contain the source code (such as the Dockerfile, requirements.txt file, and so on) required to execute the W&B run.
As an example, suppose we have a directory called fashion_mnist_train
with the following directory structure:
fashion_mnist_train
│ data_loader.py
│ Dockerfile
│ job.py
│ requirements.txt
└───configs
│ │ example.yml
We create a Docker image called fashion-mnist
with the docker build
command:
docker build . -t fashion-mnist
For more information on how to build Docker images, see the Docker build reference documentation.
- W&B CLI
- Docker run
Create a launch job with the W&B CLI. Copy the following code snippet and replace the values within "<>"
based on your use case:
wandb job create --project "<project-name>" --entity "<your-entity>" \
--name "<name-for-job>" image image-name:tag
For a full list of flags you can use, see the wandb job create
command documentation.
Associate your run with a Docker image. W&B will look for an image tag in the WANDB_DOCKER
environment variable, and if WANDB_DOCKER
is set, a job will be created from the specified image tag. Ensure that the WANDB_DOCKER
environment variable is set to the full image tag.
Create a launch job by building a Docker container from a Docker image. Copy the following code snippet and replace the values within "<>"
based on your use case:
docker run -e WANDB_PROJECT="<project-name>" \
-e WANDB_ENTITY="<your-entity>" \
-e WANDB_API_KEY="<your-w&B-api-key>" \
-e WANDB_DOCKER="<docker-image-name>" image:tag
You can specify a name for your job with the WANDB_JOB_NAME
environment variable. If you do not specify a name, W&B will automatically generate a launch job name for you. W&B will create a job name with the following format: job-<image>-<name>
.
Make sure you specify is set to the full image tag. For example, if your agent will run images from an ECR repository, you should set WANDB_DOCKER
to the full image tag, including the ECR repository URL: 123456789012.dkr.ecr.us-east-1.amazonaws.com/my-image:develop
. The docker tag, in this case 'develop'
, is added as an alias to the resulting job.
Create a job with Git
Create a Git-based job with W&B Launch. Code and other assets are cloned from a certain commit, branch, or tag in a git repository.
- CLI
- Autogenerate from git commit
wandb job create --project "<project-name>" --entity "<your-entity>" \
--name "<name-for-job>" git https://github.com/repo-name \
--entry-point "<path-to-script/code.py>"
Ensure the path with your Python script has a requirements.txt
file with the Python dependencies required to run your code. A Python runtime is also required. The python runtime can either be specified manually with the runtime parameter or can be auto-detected from a runtime.txt
or .python-version file
.
You can specify a name for your job with the WANDB_JOB_NAME
environment variable. If you do not specify a name, W&B will automatically generate a launch job name for you. It will create a job name with the following format: job-<git-remote-url>-<path-to-script>
.
Launch job names
By default, W&B automatically generates a job name for you. The name is generated depending on how the job is created (GitHub, code artifact, or Docker image). Alternatively, you can define a launch job's name with environment variables or with the W&B Python SDK.
Default launch job names
The following table describes the job naming convention used by default based on job source:
Source | Naming convention |
---|---|
GitHub | job-<git-remote-url>-<path-to-script> |
Code artifact | job-<code-artifact-name> |
Docker image | job-<image-name> |
Name your launch job
Name your job with a W&B environment variable or with the W&B Python SDK
- Environment variable
- W&B Python SDK
Set the WANDB_JOB_NAME
environment variable to your preferred job name. For example:
WANDB_JOB_NAME=awesome-job-name
Define the name of your job with wandb.Settings
. Then pass this object when you initialize W&B with wandb.init
. For example:
settings = wandb.Settings(job_name="my-job-name")
wandb.init(settings=settings)
For docker image jobs, the version alias is automatically added as an alias to the job.