How do I page through large API results in W&B? - Weights & Biases Documentation

You can page through API result using the standard lazy-iterator pattern and per_page parameter. Additionally, you can use the following checkpointing and bulk-download tips to more effectively page through the results.

Checkpoint processed run IDs

For very large projects, record IDs you have already handled and skip them on restart:

import json
import pathlib
import wandb

checkpoint_file = pathlib.Path("processed_ids.json")
processed = set(json.loads(checkpoint_file.read_text())) if checkpoint_file.exists() else set()

api = wandb.Api()
for run in api.runs("my-entity/my-project"):
    if run.id in processed:
        continue
    process(run)
    processed.add(run.id)
    checkpoint_file.write_text(json.dumps(list(processed)))

Avoid list(api.runs(...)) on huge projects unless you need random access — it forces every page into memory.

Rate limits on bulk downloads

If each run triggers extra API calls (for example run.file("output.log").download()), add a short delay to avoid 429 errors:

import time

for run in api.runs("my-entity/my-project"):
    run.file("output.log").download()
    time.sleep(0.1)

Runs Experiments API Artifacts

⌘I

​Checkpoint processed run IDs

​Rate limits on bulk downloads

Checkpoint processed run IDs

Rate limits on bulk downloads