Monitor using built-in signals

Documentation Index

Fetch the complete documentation index at: https://docs.wandb.ai/llms.txt

Use this file to discover all available pages before exploring further.

In modern agent development, standard system metrics like latency, token count, and cost are insufficient for understanding complex agent behavior. Inspecting individual traces provides deep insight but doesn’t scale to the millions of traces generated in a live environment. Signals provide a high-level monitoring solution to this problem by offering automated, behavioral scoring for agents in production:

• Automated scoring: Every incoming production trace is automatically processed and scored on common quality issues and errors.
• Infrastructure: Processing is powered by CoreWeave compute and CoreWeave GPUs for scalability across millions of traces.

By using signals within production, you can:

• Gain behavioral insight: Move beyond simple system metrics to understand if your agent is hallucinating, failing to follow conversation patterns, or losing grounding in its evidence.
• Accelerate the research loop: Use the scores and failure analyses generated by signals to identify specific weaknesses, which you can use to research model improvement, data annotation, or reinforcement learning.

​

Available signals

W&B Weave offers monitors with built-in signals: preset scorers that evaluate production traces for common quality issues and errors out of the box, with no custom setup. Each built-in signal uses a benchmarked LLM prompt to classify traces and saves the results as comma-delimited tags representing the detected issues. Signals use a Serverless Inference model to score traces, so no external API keys are required. W&B Weave provides 13 preset signals organized into two groups.

​

Quality signals

Quality signals evaluate successful root-level traces for output quality and safety issues.

​

Error signals

Error signals categorize failed traces by root cause to help you identify and resolve infrastructure and application issues.

​

How signals work

Each signal uses an LLM-as-a-judge approach to classify traces:

• Trace selection: Quality signals evaluate successful root-level traces. Error signals evaluate failed traces. Child spans and intermediate Calls are not scored.
• Prompt construction: Weave constructs a prompt that includes the trace metadata, inputs, outputs, exception details (if any), and the operation’s source code. The signal’s classifier prompt is appended with instructions for the specific issue to detect.
• LLM scoring: For each signal, a Serverless Inference model performs a binary classification (whether that issue is present on the trace). Detected issues are returned as comma-delimited string tags (for example, "Low-quality, User-frustration, Forgetful").

When multiple signals from the same group (Quality or Error) are active, Weave batches the signals into a single LLM call for efficiency. The model evaluates all active classifiers in one pass and returns results for each.

​

Enable signals from the Monitors page

To enable signals:

1. Navigate to wandb.ai and then open your Weave project.
2. In the Weave project sidebar, select Monitors.
3. At the top of the Monitors page, a row of suggested signal cards appears. Each card shows the signal name, a description, and an + Add signal button.
4. To enable a single signal, select the + Add signal button on the signal card. The signal begins scoring new traces immediately.
5. To enable multiple signals at once, select the + [X] more signals button. This opens a drawer that lists all available signals grouped by category.
6. Select the signals you want to turn on, then select Add signals.

After enabling signals, Weave automatically scores incoming traces.

​

Manage active signals

To view or remove active signals:

1. From the Monitors page, select the Manage signals () button. This opens a drawer listing all currently active signals grouped by category.
2. Hover over a signal and select the Remove signal () button to disable the signal.

Removing a signal stops scoring new traces. Existing scores from the signal are preserved.

​

Use built-in signals

​

See tagged agent traces on the Agents page

If you are tracing agentic applications, signal results appear in two places on the Agents page:

• The Signals tab shows all scored turns across your agents, with the scorer name, numeric score, and 24-hour trend for each.
• The Conversations tab’s conversation detail panel shows a Scores section in the Events panel. This includes the rubric breakdown and per-criterion confidence for each active scorer.

For full details, see View agent activity.

​

See tagged Call traces on the Traces page

If you are tracing individual functions as Ops with the @weave.op decorator, signal results are stored as feedback on the Call object and are queryable from the Traces page. You can quickly scan your traces for certain behavior in the Traces page using the Signals column. The Signals column is populated with tags when their criteria are met, and you can hover over these tags to see the confidence in the score and the reasoning. Using the trace table toolbar, you can filter the trace table to only show traces that have triggered certain signals. You can view additional signal details in the Traces page by selecting the classifier Call that the signal generates and reviewing the Trace Details view. Under Call Output, review classifier_meta for the reasoning. For example, the following screenshot shows a Quality-classifiers signal with Low-quality match and confidence (0.9) with a reason for this rating.

​

See signals in the project dashboard

You can also review signals at a project level:

1. In the project sidebar, select Project.
2. At the top of the Project dashboard, select the Weave tab.
3. In the Weave dashboard panels, locate Monitor Scores.

In the Monitor Scores project panel, you can see time-based graphs of signals that occurred for the project.

​

Alert on signals

You can set up automated triggers that notify your team through tools like Slack when an agent’s performance drops below a certain threshold. To get notified when a signal is triggered, set up an automation.

---

For specific monitoring beyond what is provided by the built-in signals, see Set up custom monitors.