This guide shows you how to use EvaluationLogger to record predictions and scores from your existing Python or TypeScript code, so you can evaluate model performance in Weave without first defining a full dataset and scorer suite.
This approach is helpful in complex workflows where the entire dataset or all scorers might not be defined upfront.
In contrast to the standard Evaluation object, which requires a predefined Dataset and list of Scorer objects, the EvaluationLogger lets you log individual predictions and their associated scores incrementally as they become available.
Prefer a more structured evaluation?If you prefer a more opinionated evaluation framework with predefined datasets and scorers, see Weave’s standard Evaluation framework.The EvaluationLogger offers flexibility while the standard framework offers structure and guidance. Basic workflow
Following these steps records a complete evaluation in Weave, with per-prediction scores and an aggregated summary that you can review in the Weave UI.
- Initialize the logger: Create an instance of
EvaluationLogger, optionally providing metadata about the model and dataset. Weave uses defaults if omitted.
To capture token usage and cost for LLM calls (for example, OpenAI), initialize EvaluationLogger before any LLM invocations.
If you call your LLM first and then log predictions afterward, Weave doesn’t capture token and cost data.
- Log predictions: Call
log_prediction() for each input and output pair from your system.
- Log scores: Use the returned
ScoreLogger to log_score() for the prediction. Multiple scores per prediction are supported.
- Finish prediction: Always call
finish() after logging scores for a prediction to finalize it.
- Log summary: After all predictions are processed, call
log_summary() to aggregate scores and add optional custom metrics.
After calling finish() on a prediction, no more scores can be logged for it.
log_example().
Basic example
The following example shows how to use EvaluationLogger to log predictions and scores inline with your existing code.
The user_model model function is defined and applied to a list of inputs. For each example:
- The input and output are logged using
log_prediction.
- A correctness score (
correctness_score) is logged through log_score.
finish() finalizes logging for that prediction.
Finally, log_summary records any aggregate metrics and triggers automatic score summarization in Weave.import weave
from openai import OpenAI
from weave import EvaluationLogger
weave.init('your-team/your-project')
# Initialize EvaluationLogger BEFORE calling the model to ensure token tracking
eval_logger = EvaluationLogger(
model="my_model",
dataset="my_dataset"
)
# Example input data (this can be any data structure you want)
eval_samples = [
{'inputs': {'a': 1, 'b': 2}, 'expected': 3},
{'inputs': {'a': 2, 'b': 3}, 'expected': 5},
{'inputs': {'a': 3, 'b': 4}, 'expected': 7},
]
# Example model logic using OpenAI
@weave.op
def user_model(a: int, b: int) -> int:
oai = OpenAI()
response = oai.chat.completions.create(
messages=[{"role": "user", "content": f"What is {a}+{b}?"}],
model="gpt-4o-mini"
)
# Use the response in some way (here we just return a + b for simplicity)
return a + b
# Iterate through examples, predict, and log
for sample in eval_samples:
inputs = sample["inputs"]
model_output = user_model(**inputs) # Pass inputs as kwargs
# Log the prediction input and output
pred_logger = eval_logger.log_prediction(
inputs=inputs,
output=model_output
)
# Calculate and log a score for this prediction
expected = sample["expected"]
correctness_score = model_output == expected
pred_logger.log_score(
scorer="correctness", # Simple string name for the scorer
score=correctness_score
)
# Finish logging for this specific prediction
pred_logger.finish()
# Log a final summary for the entire evaluation.
# Weave auto-aggregates the 'correctness' scores logged above.
summary_stats = {"subjective_overall_score": 0.8}
eval_logger.log_summary(summary_stats)
print("Evaluation logging complete. View results in the Weave UI.")
The TypeScript SDK provides two API patterns:
- Fire-and-forget API (recommended for most cases): Use
logPrediction() without await for synchronous, non-blocking logging.
- Awaitable API: Use
logPredictionAsync() with await when you need to ensure operations complete before proceeding.
Use fire-and-forget for:
- High throughput: Process multiple predictions in parallel without waiting for each logging operation.
- Minimal code disruption: Add evaluation logging without restructuring your existing async/await flow.
- Simplicity: Less boilerplate code and cleaner syntax for most evaluation scenarios.
The fire-and-forget pattern is safe because logSummary() automatically waits for all pending operations to complete before aggregating results.The following example evaluates model predictions with the fire-and-forget pattern. It sets up an evaluation logger, runs a model on three test samples, and then logs the prediction without using await:import weave, {EvaluationLogger} from 'weave';
import OpenAI from 'openai';
await weave.init('your-team/your-project');
// Initialize EvaluationLogger BEFORE calling the model to ensure token tracking
const evalLogger = new EvaluationLogger({
name: 'my-eval',
model: 'my_model',
dataset: 'my_dataset'
});
// Example input data
const evalSamples = [
{inputs: {a: 1, b: 2}, expected: 3},
{inputs: {a: 2, b: 3}, expected: 5},
{inputs: {a: 3, b: 4}, expected: 7},
];
// Example model logic using OpenAI
const userModel = weave.op(async function userModel(a: number, b: number): Promise<number> {
const oai = new OpenAI();
const response = await oai.chat.completions.create({
messages: [{role: 'user', content: `What is ${a}+${b}?`}],
model: 'gpt-4o-mini'
});
return a + b;
});
// Iterate through examples, predict, and log using fire-and-forget pattern
for (const sample of evalSamples) {
const {inputs} = sample;
const modelOutput = await userModel(inputs.a, inputs.b);
// Fire-and-forget: No await needed for logPrediction
const scoreLogger = evalLogger.logPrediction(inputs, modelOutput);
// Calculate and log a score for this prediction
const correctnessScore = modelOutput === sample.expected;
// Fire-and-forget: No await needed for logScore
scoreLogger.logScore('correctness', correctnessScore);
// Fire-and-forget: No await needed for finish
scoreLogger.finish();
}
// logSummary waits for all pending operations to complete internally
const summaryStats = {subjective_overall_score: 0.8};
await evalLogger.logSummary(summaryStats);
console.log('Evaluation logging complete. View results in the Weave UI.');
logPrediction() without await, it uses logPredictionAsync() with await to ensure each operation completes before proceeding to the next one:// Use logPredictionAsync instead of logPrediction
const scoreLogger = await evalLogger.logPredictionAsync(inputs, modelOutput);
// Await each operation
await scoreLogger.logScore('correctness', correctnessScore);
await scoreLogger.finish();
Simplified logging with log_example()
Use log_example() to log inputs, an output, and scores in a single call. This convenience method combines log_prediction(), log_score(), and finish() into one step, and it’s useful when you already have the inputs, model outputs, and scores ready to log, such as during batch or offline evaluations.
import weave
from weave import EvaluationLogger
weave.init('your-team-name/your-project-name')
eval_logger = EvaluationLogger(
model="my_model",
dataset="my_dataset"
)
eval_samples = [
{'inputs': {'a': 1, 'b': 2}, 'expected': 3},
{'inputs': {'a': 2, 'b': 3}, 'expected': 5},
{'inputs': {'a': 3, 'b': 4}, 'expected': 7},
]
for sample in eval_samples:
inputs = sample['inputs']
output = inputs['a'] + inputs['b']
eval_logger.log_example(
inputs=inputs,
output=output,
scores={"correctness": output == sample['expected']}
)
eval_logger.log_summary({"avg_score": 1.0})
log_example() call is equivalent to:
pred = eval_logger.log_prediction(inputs=inputs, output=output)
pred.log_score(scorer="correctness", score=output == sample['expected'])
pred.finish()
log_example() is not available for the Weave TypeScript SDK. TypeScript users should use the logPrediction() and logScore() pattern shown in the basic example.Advanced usage
The EvaluationLogger offers flexible patterns beyond the basic workflow to accommodate more complex evaluation scenarios. The following sections describe advanced techniques, including how to use context managers for automatic resource management, separate model execution from logging, work with rich media data, and compare multiple model evaluations side by side.
Use context managers
The EvaluationLogger supports context managers (with statements) for both predictions and scores. This can provide cleaner code, automatic resource cleanup, and better tracking of nested operations like LLM judge calls.
Using with statements in this context provides:
- Automatic
finish() calls when exiting the context.
- Better token and cost tracking for nested LLM calls.
- Setting output after model execution within the prediction context.
import openai
import weave
weave.init("nested-evaluation-example")
oai = openai.OpenAI()
# Initialize the logger
ev = weave.EvaluationLogger(
model="gpt-4o-mini",
dataset="joke_dataset"
)
user_prompt = "Tell me a joke"
# Use context manager for prediction - no need to call finish()
with ev.log_prediction(inputs={"user_prompt": user_prompt}) as pred:
# Make your model call within the context
result = oai.chat.completions.create(
model="gpt-4o-mini",
messages=[{"role": "user", "content": user_prompt}],
)
# Set the output after the model call
pred.output = result.choices[0].message.content
# Log simple scores
pred.log_score("correctness", 1.0)
pred.log_score("ambiguity", 0.3)
# Use nested context manager for scores that require LLM calls
with pred.log_score("llm_judge") as score:
judge_result = oai.chat.completions.create(
model="gpt-4o-mini",
messages=[
{"role": "system", "content": "Rate how funny the joke is from 1-5"},
{"role": "user", "content": pred.output},
],
)
# Set the score value after computation
score.value = judge_result.choices[0].message.content
# finish() is automatically called when exiting the 'with' block
ev.log_summary({"avg_score": 1.0})
TypeScript doesn’t have Python’s with statement pattern for context managers. Instead, use the fire-and-forget pattern with explicit finish() calls.The following example logs a prediction, adds scores and an LLM judge score, then finalizes the prediction with finish():import weave from 'weave';
import OpenAI from 'openai';
import {EvaluationLogger} from 'weave/evaluationLogger';
await weave.init('your-team/your-project');
const oai = new OpenAI();
// Initialize the logger
const ev = new EvaluationLogger({
name: 'joke-eval',
model: 'gpt-4o-mini',
dataset: 'joke_dataset',
});
const userPrompt = 'Tell me a joke';
// Get model output
const result = await oai.chat.completions.create({
model: 'gpt-4o-mini',
messages: [{role: 'user', content: userPrompt}],
});
const modelOutput = result.choices[0].message.content;
// Log prediction with output
const pred = ev.logPrediction({user_prompt: userPrompt}, modelOutput);
// Log simple scores
pred.logScore('correctness', 1.0);
pred.logScore('ambiguity', 0.3);
// For LLM judge scores, make the call and log the result
const judgeResult = await oai.chat.completions.create({
model: 'gpt-4o-mini',
messages: [
{role: 'system', content: 'Rate how funny the joke is from 1-5'},
{role: 'user', content: modelOutput || ''},
],
});
pred.logScore('llm_judge', judgeResult.choices[0].message.content);
// Explicitly call finish when done scoring
pred.finish();
await ev.logSummary({avg_score: 1.0});
While TypeScript doesn’t have automatic cleanup with context managers, logSummary() automatically finishes any unfinished predictions before aggregating results. You can rely on this behavior if you prefer not to call finish() explicitly.
Link to an existing dataset
When you pass raw datasets as inputs to log_prediction, Weave reimports the data with every evaluation run. This stores duplicate data, which can waste space if the dataset is large or if many evaluations reuse it.
To avoid this duplication, publish your dataset to Weave before running any evaluations, then pass the published dataset’s rows as inputs. Weave resolves references to published rows using internal references instead of reimporting the data. This technique gives you the same linked experience as the standard Evaluation framework, where each prediction links back to a specific dataset row in the Weave UI.
The following example publishes a dataset and links to it in the EvaluationLogger, before retrieving and iterating over it like any other dataset.
import weave
from weave import EvaluationLogger
weave.init("your-team-name/your-project-name")
# Publish the dataset (only needs to happen once)
dataset = weave.Dataset(
name="my_eval_dataset",
rows=[
{"question": "What is the capitol of France?", "expected": "Paris"},
{"question": "What U.S. state is Seattle in?", "expected": "Washington"},
{"question": "In what country is Mount Fuji located in?", "expected": "Japan"},
],
)
weave.publish(dataset)
# Retrieve the published dataset
dataset = weave.ref("my_eval_dataset").get()
import weave, {EvaluationLogger, Dataset} from 'weave';
await weave.init('your-team-name/your-project-name');
// Publish the dataset (only needs to happen once)
const dataset = new Dataset({
name: 'my_eval_dataset',
rows: [
{"question": "What is the capitol of France?", "expected": "Paris"},
{"question": "What U.S. state is Seattle in?", "expected": "Washington"},
{"question": "In what country is Mount Fuji located in?", "expected": "Japan"},
],
});
const datasetRef = await dataset.save();
// Retrieve the published dataset
const published = await datasetRef.get();
Get outputs before logging
You can first compute your model outputs, then separately log predictions and scores. This separates evaluation and logging logic, which can make code easier to test and maintain when different parts of your system handle prediction generation and scoring.
# Initialize EvaluationLogger BEFORE calling the model to ensure token tracking
ev = EvaluationLogger(
model="example_model",
dataset="example_dataset"
)
# Model outputs (e.g. OpenAI calls) must happen after logger init for token tracking
outputs = [your_output_generator(**inputs) for inputs in your_dataset]
preds = [ev.log_prediction(inputs, output) for inputs, output in zip(your_dataset, outputs)]
for pred, output in zip(preds, outputs):
pred.log_score(scorer="greater_than_5_scorer", score=output > 5)
pred.log_score(scorer="greater_than_7_scorer", score=output > 7)
pred.finish()
ev.log_summary()
The fire-and-forget pattern excels when processing multiple predictions in parallel.The following example batch-processes evaluations in parallel by creating multiple concurrent instances of the EvaluationLogger:// Initialize EvaluationLogger BEFORE calling the model to ensure token tracking
const ev = new EvaluationLogger({
name: 'parallel-eval',
model: 'example_model',
dataset: 'example_dataset'
});
// Model outputs, such as OpenAI calls, must happen after logger init for token tracking
const outputs = await Promise.all(
yourDataset.map(inputs => yourOutputGenerator(inputs))
);
// Fire-and-forget: Process all predictions without awaiting
const preds = yourDataset.map((inputs, i) =>
ev.logPrediction(inputs, outputs[i])
);
preds.forEach((pred, i) => {
const output = outputs[i];
// Fire-and-forget: No await needed
pred.logScore('greater_than_5_scorer', output > 5);
pred.logScore('greater_than_7_scorer', output > 7);
pred.finish();
});
// logSummary waits for all pending operations
await ev.logSummary();
log_prediction or log_score methods.
import io
import wave
import struct
from PIL import Image
import random
from typing import Any
import weave
def generate_random_audio_wave_read(duration=2, sample_rate=44100):
n_samples = duration * sample_rate
amplitude = 32767 # 16-bit max amplitude
buffer = io.BytesIO()
# Write wave data to the buffer
with wave.open(buffer, 'wb') as wf:
wf.setnchannels(1)
wf.setsampwidth(2) # 16-bit
wf.setframerate(sample_rate)
for _ in range(n_samples):
sample = random.randint(-amplitude, amplitude)
wf.writeframes(struct.pack('<h', sample))
# Rewind the buffer to the beginning so we can read from it
buffer.seek(0)
# Return a Wave_read object
return wave.open(buffer, 'rb')
rich_media_dataset = [
{
'image': Image.new(
"RGB",
(100, 100),
color=(
random.randint(0, 255),
random.randint(0, 255),
random.randint(0, 255),
),
),
"audio": generate_random_audio_wave_read(),
}
for _ in range(5)
]
@weave.op
def your_output_generator(image: Image.Image, audio) -> dict[str, Any]:
return {
"result": random.randint(0, 10),
"image": image,
"audio": audio,
}
ev = EvaluationLogger(model="example_model", dataset="example_dataset")
for inputs in rich_media_dataset:
output = your_output_generator(**inputs)
pred = ev.log_prediction(inputs, output)
pred.log_score(scorer="greater_than_5_scorer", score=output["result"] > 5)
pred.log_score(scorer="greater_than_7_scorer", score=output["result"] > 7)
ev.log_summary()
The TypeScript SDK supports logging images and audio using the weaveImage and weaveAudio functions. The following example loads image and audio files, processes them through a model, and logs the results with scores.import weave, {EvaluationLogger} from 'weave';
import * as fs from 'fs';
await weave.init('your-team/your-project');
// Load images and audio from files
const richMediaDataset = [
{
image: weave.weaveImage({data: fs.readFileSync('sample1.png')}),
audio: weave.weaveAudio({data: fs.readFileSync('sample1.wav')}),
},
{
image: weave.weaveImage({data: fs.readFileSync('sample2.png')}),
audio: weave.weaveAudio({data: fs.readFileSync('sample2.wav')}),
},
];
// Model that processes media and returns results
const yourOutputGenerator = weave.op(
async (inputs: {image: any; audio: any}) => {
const result = Math.floor(Math.random() * 10);
return {
result,
image: inputs.image,
audio: inputs.audio,
};
},
{name: 'yourOutputGenerator'}
);
const ev = new EvaluationLogger({
name: 'rich-media-eval',
model: 'example_model',
dataset: 'example_dataset',
});
for (const inputs of richMediaDataset) {
const output = await yourOutputGenerator(inputs);
// Log prediction with rich media in both inputs and outputs
const pred = ev.logPrediction(inputs, output);
pred.logScore('greater_than_5_scorer', output.result > 5);
pred.logScore('greater_than_7_scorer', output.result > 7);
pred.finish();
}
await ev.logSummary();
Log and compare multiple evaluations
With EvaluationLogger, you can log and compare multiple evaluations side by side in the Weave UI. This is useful for assessing how different models perform on the same dataset.
- Run the following code sample.
- In the Weave UI, navigate to the
Evals tab.
- Select the evals that you want to compare.
- Click the Compare button. In the Compare view, you can:
- Choose which Evals to add or remove.
- Choose which metrics to show or hide.
- Page through specific examples to see how different models performed for the same input on a given dataset.
For more information about comparisons, see Comparisons.
import weave
models = [
"model1",
"model2",
{"name": "model3", "metadata": {"coolness": 9001}}
]
for model in models:
# EvalLogger must be initialized before model calls to capture tokens
ev = EvaluationLogger(
name="comparison-eval",
model=model,
dataset="example_dataset",
scorers=["greater_than_3_scorer", "greater_than_5_scorer", "greater_than_7_scorer"],
eval_attributes={"experiment_id": "exp_123"}
)
for inputs in your_dataset:
output = your_output_generator(**inputs)
pred = ev.log_prediction(inputs=inputs, output=output)
pred.log_score(scorer="greater_than_3_scorer", score=output > 3)
pred.log_score(scorer="greater_than_5_scorer", score=output > 5)
pred.log_score(scorer="greater_than_7_scorer", score=output > 7)
pred.finish()
ev.log_summary()
import weave from 'weave';
import {EvaluationLogger} from 'weave/evaluationLogger';
import {WeaveObject} from 'weave/weaveObject';
await weave.init('your-team/your-project');
const models = [
'model1',
'model2',
new WeaveObject({name: 'model3', metadata: {coolness: 9001}})
];
for (const model of models) {
// EvalLogger must be initialized before model calls to capture tokens
const ev = new EvaluationLogger({
name: 'comparison-eval',
model: model,
dataset: 'example_dataset',
description: 'Model comparison evaluation',
scorers: ['greater_than_3_scorer', 'greater_than_5_scorer', 'greater_than_7_scorer'],
attributes: {experiment_id: 'exp_123'}
});
for (const inputs of yourDataset) {
const output = await yourOutputGenerator(inputs);
// Fire-and-forget pattern for clean, efficient logging
const pred = ev.logPrediction(inputs, output);
pred.logScore('greater_than_3_scorer', output > 3);
pred.logScore('greater_than_5_scorer', output > 5);
pred.logScore('greater_than_7_scorer', output > 7);
pred.finish();
}
await ev.logSummary();
}
Usage tips
The following tips help you get the most out of EvaluationLogger:
- Call
finish() promptly after each prediction.
- Use
log_summary to capture metrics not tied to single predictions (for example, overall latency).
- Rich media logging is useful for qualitative analysis.
- Auto-finish behavior: While we recommend explicitly calling
finish() on each prediction for clarity, logSummary() automatically finishes any unfinished predictions. However, once the script calls finish(), it can’t log any more scores.
- Configuration options: Use configuration options, including
name, description, dataset, model, scorers, and attributes to organize and filter your evaluations in the Weave UI.
