> ## Documentation Index
> Fetch the complete documentation index at: https://docs.wandb.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Collect feedback and use annotations

> Collect and analyze feedback for LLM applications through UI and SDK

Efficiently evaluating LLM applications requires robust tooling to collect and analyze feedback. W\&B Weave provides an integrated feedback system, allowing users to provide Call feedback directly through the UI or programmatically through the SDK. Various feedback types are supported, including emoji reactions, textual comments, and structured data, enabling teams to:

* Build evaluation datasets for performance monitoring.
* Identify and resolve LLM content issues effectively.
* Gather examples for advanced tasks like fine-tuning.

This guide covers how to use Weave’s feedback functionality in both the UI and SDK, query and manage feedback, and use human annotations for detailed evaluations.

* [Provide feedback in the UI](#provide-feedback-in-the-ui)
* [Provide feedback through the SDK](#provide-feedback-through-the-sdk)
* [Add human annotations](#add-human-annotations)

## Provide feedback in the UI

In the Weave UI, you can add and view feedback [from the Call details panel](#from-the-call-details-panel) or [using the icons](#use-the-icons).

### From the Call details panel

1. In the Weave project sidebar, navigate to **Traces**.
2. Find the row for the Call that you want to add feedback to.
3. Click the linked Trace name to open the trace tree and Call details panel.
4. In the Call details tab bar, select **Feedback**.
5. Add, view, or delete feedback:
   * *Add and view feedback using the icons* located in the upper right corner of the Call details feedback view.
   * *View and delete feedback from the Call details feedback table.* Delete feedback by clicking the trashcan icon in the rightmost column of the appropriate feedback row.

<Frame>
  <img src="https://mintcdn.com/wb-21fd5541/4ANo4MV8FzCjYewG/weave/guides/tracking/imgs/feedback_tab.png?fit=max&auto=format&n=4ANo4MV8FzCjYewG&q=85&s=78d53f5ffac8949500ced6e95590acfe" alt="Screenshot of Feedback tab in Call details" width="1266" height="548" data-path="weave/guides/tracking/imgs/feedback_tab.png" />
</Frame>

### Use the feedback icons

You can add or remove a reaction, and add a note using the icons that are located in both the Traces table and individual Call details panel.

* *Traces table*: Located in **Feedback** column in the appropriate row in the **Traces** table.
* *Call details panel*: Located in the upper right corner of each Call details panel.

To add a reaction:

1. Click the emoji icon.
2. Add a thumbs up, thumbs down, or click the **+** icon for more emojis.

To remove a reaction:

1. Hover over the emoji reaction you want to remove.
2. Click the reaction to remove it.

> You can also delete feedback from the [**Feedback** column on the Call details panel](#from-the-call-details-panel).

To add a comment:

1. Click the comment bubble icon.
2. In the text box, add your note. The maximum number of characters in a feedback note is 1024.
3. To save the note, press the **Enter** key. You can add additional notes.

<Warning>
  The maximum number of characters in a feedback note is 1024. If a note exceeds this limit, it will not be created.
</Warning>

<Frame>
  <img src="https://mintcdn.com/wb-21fd5541/4ANo4MV8FzCjYewG/weave/guides/tracking/imgs/feedback_calls.png?fit=max&auto=format&n=4ANo4MV8FzCjYewG&q=85&s=95558011495dae1295e540718a614bb4" alt="Screenshot of calls grid with feedback column" width="3170" height="1589" data-path="weave/guides/tracking/imgs/feedback_calls.png" />
</Frame>

## Provide feedback through the SDK

You can find SDK usage examples for feedback in the UI under the **Use** tab in the Call details panel.

You can use the Weave Python SDK to programmatically [add](/weave/reference/python-sdk/trace/feedback#method-add), [remove](/weave/reference/python-sdk/trace/feedback#method-purge), and [query feedback](/weave/reference/python-sdk/trace/weave_client#method-get_feedback) on calls. The TypeScript SDK does not currently support feedback functionality.

### Query a project's feedback

You can query the feedback for your Weave project using the SDK. The SDK supports the following feedback query operations:

* `client.get_feedback()`: Returns all feedback in a project.
* `client.get_feedback("<feedback_uuid>")`: Returns a specific feedback object specified by `<feedback_uuid>` as a collection.
* `client.get_feedback(reaction="<reaction_type>")`: Returns all feedback objects for a specific reaction type.

You can also get additional information for each feedback object in `client.get_feedback()`:

* `id`: The feedback object ID.
* `created_at`: The creation time information for the feedback object.
* `feedback_type`: The type of feedback (reaction, note, custom).
* `payload`: The feedback payload.

<Tabs>
  <Tab title="Python">
    ```python lines theme={null}
    import weave
    client = weave.init('intro-example')

    # Get all feedback in a project
    all_feedback = client.get_feedback()

    # Fetch a specific feedback object by id.
    # The API returns a collection, which is expected to contain at most one item.
    one_feedback = client.get_feedback("<feedback_uuid>")[0]

    # Find all feedback objects with a specific reaction. You can specify offset and limit.
    thumbs_up = client.get_feedback(reaction="👍", limit=10)

    # After retrieval, view the details of individual feedback objects.
    for f in client.get_feedback():
        print(f.id)
        print(f.created_at)
        print(f.feedback_type)
        print(f.payload)
    ```
  </Tab>

  <Tab title="TypeScript">
    ```plaintext theme={null}
    This feature is not available in TypeScript yet.
    ```
  </Tab>
</Tabs>

### Add feedback to a Call

You can add feedback to a Call using the Call's UUID. To use the UUID to get a particular Call, [retrieve it during or after Call execution](#retrieve-the-call-uuid). The SDK supports the following operations for adding feedback to a Call:

* `call.feedback.add_reaction("<reaction_type>")`: Add one of the supported `<reaction_types>` (emojis), such as 👍.
* `call.feedback.add_note("<note>")`: Add a note.
* `call.feedback.add("<label>", <object>)`: Add a custom feedback `<object>` specified by `<label>`.

<Warning>
  The maximum number of characters in a feedback note is 1024. If a note exceeds this limit, it will not be created.
</Warning>

<Tabs>
  <Tab title="Python">
    ```python lines theme={null}
    import weave
    client = weave.init('intro-example')

    call = client.get_call("<call_uuid>")

    # Adding an emoji reaction
    call.feedback.add_reaction("👍")

    # Adding a note
    call.feedback.add_note("this is a note")

    # Adding custom key/value pairs.
    # The first argument is a user-defined "type" string.
    # Feedback must be JSON serializable and less than 1 KB when serialized.
    call.feedback.add("correctness", { "value": 5 })
    ```
  </Tab>

  <Tab title="TypeScript">
    ```plaintext theme={null}
    This feature is not available in TypeScript yet.
    ```
  </Tab>
</Tabs>

#### Retrieve the Call UUID

For scenarios where you need to add feedback immediately after a Call, you can retrieve the Call UUID programmatically during or after the Call execution.

##### During Call execution

To retrieve the UUID during Call execution, get the current Call, and return the ID.

<Tabs>
  <Tab title="Python">
    ```python lines theme={null}

    import weave
    weave.init("uuid")

    @weave.op()
    def simple_operation(input_value):
        # Perform some simple operation
        output = f"Processed {input_value}"
        # Get the current call ID
        current_call = weave.require_current_call()
        call_id = current_call.id
        return output, call_id
    ```
  </Tab>

  <Tab title="TypeScript">
    ```plaintext theme={null}
    This feature is not available in TypeScript yet.
    ```
  </Tab>
</Tabs>

##### After Call execution

Alternatively, you can use `call()` method to execute the operation and retrieve the ID after Call execution:

<Tabs>
  <Tab title="Python">
    ```python lines theme={null}
    import weave
    weave.init("uuid")

    @weave.op()
    def simple_operation(input_value):
        return f"Processed {input_value}"

    # Execute the operation and retrieve the result and call ID
    result, call = simple_operation.call("example input")
    call_id = call.id
    ```
  </Tab>

  <Tab title="TypeScript">
    ```plaintext theme={null}
    This feature is not available in TypeScript yet.
    ```
  </Tab>
</Tabs>

### Delete feedback from a Call

You can delete feedback from a particular call by specifying a UUID.

<Tabs>
  <Tab title="Python">
    ```python lines theme={null}
    call.feedback.purge("<feedback_uuid>")
    ```
  </Tab>

  <Tab title="TypeScript">
    ```plaintext theme={null}
    This feature is not available in TypeScript yet.
    ```
  </Tab>
</Tabs>

## Add human annotations

Human annotations are supported in the Weave UI. This functionality allows you to create custom fields to add additional, human-entered data to your Traces as feedback. To make human annotations, you must first create a Human Annotation scorer using either the [UI](#create-a-human-annotation-scorer-in-the-ui) or the [API](#create-a-human-annotation-scorer-using-the-api). Then, you can [use the scorer in the UI to make annotations](#use-the-human-annotation-scorer-in-the-ui), and [modify your annotation scorers using the API](#modify-a-human-annotation-scorer-using-the-api).

### Create a human annotation scorer in the UI

To create a human annotation scorer in the UI, do the following:

1. In the project sidebar, navigate to **Assets**.
2. In the Assets navigation panel, click **Scorers**.
3. In the **Scorers** panel header, click **New scorer**.
4. In the **Create Scorer** modal dialog, set:
   * `Scorer type` to `Human annotation`
   * `Name`
   * `Description`
   * `Type`, which determines the type of feedback that will be collected, such as `boolean` or `integer`.
5. Click **Create scorer**. Now, you can use your scorer to make annotations.

In the following example, a human annotator is asked to select which type of document the LLM ingested. As such, the `Type` selected for the score configuration is an `enum` containing the possible document types.

<Frame>
  <img src="https://mintcdn.com/wb-21fd5541/4ANo4MV8FzCjYewG/weave/guides/tracking/imgs/human-annotation-scorer-form.png?fit=max&auto=format&n=4ANo4MV8FzCjYewG&q=85&s=1b3a4407620d55f1d4b159a19218d424" alt="Create Scorer modal dialog" width="3454" height="1862" data-path="weave/guides/tracking/imgs/human-annotation-scorer-form.png" />
</Frame>

### Use the human annotation scorer in the UI

Once you create a human annotation scorer, it becomes available to use on the Traces page.

To use the scorer, do the following:

1. In the project sidebar, navigate to **Traces**.

2. Find the row for the Call that you want to add a human annotation to.

3. Click the linked Trace name to open the trace tree and Call details panel.

4. In the upper right corner of the Call details tab bar, click the **Show feedback** button.

   <img src="https://mintcdn.com/wb-21fd5541/4ANo4MV8FzCjYewG/weave/guides/tracking/imgs/marker-icon.png?fit=max&auto=format&n=4ANo4MV8FzCjYewG&q=85&s=cae1b79d0eec7f3310d6b1d928cea24d" alt="Marker icon in Call header" width="306" height="84" data-path="weave/guides/tracking/imgs/marker-icon.png" />

   Your available human annotation scorers display in an additional **Annotate** panel.

   <img src="https://mintcdn.com/wb-21fd5541/4ANo4MV8FzCjYewG/weave/guides/tracking/imgs/full-feedback-sidebar.png?fit=max&auto=format&n=4ANo4MV8FzCjYewG&q=85&s=281c47385fc6c4358067d5c7f9b4c97a" alt="Human Annotation scorer feedback panel" width="3454" height="1862" data-path="weave/guides/tracking/imgs/full-feedback-sidebar.png" />

5. Make an annotation.

6. Click **Save**.

7. In the Call details panel tab bar, click the **Feedback** tab to view the Feedback table. The new annotation displays in the table. You can also view the annotations in the **Annotations** column in the main Traces table.

   > Refresh the Traces table to view the most up-to-date information.

<Frame>
  <img src="https://mintcdn.com/wb-21fd5541/4ANo4MV8FzCjYewG/weave/guides/tracking/imgs/feedback-in-the-table.png?fit=max&auto=format&n=4ANo4MV8FzCjYewG&q=85&s=226efa32450b40914f3d32765481f058" alt="Human Annotation scorer feedback in Traces table" width="3456" height="1569" data-path="weave/guides/tracking/imgs/feedback-in-the-table.png" />
</Frame>

### Create a human annotation scorer using the API

You can also create human annotation scorers through the API. Each scorer is its own object, which you create and update independently. To create a human annotation scorer programmatically, do the following:

1. Import the `AnnotationSpec` class from `weave.flow.annotation_spec`.
2. Use the `publish` method from `weave` to create the scorer.

In the following example, two scorers are created. The first scorer, `Temperature`, is used to score the perceived temperature of the LLM call. The second scorer, `Tone`, is used to score the tone of the LLM response. Each scorer is created using `save` with an associated object ID (`temperature-scorer` and `tone-scorer`).

<Tabs>
  <Tab title="Python">
    ```python lines theme={null}
    import weave
    from weave.flow.annotation_spec import AnnotationSpec

    client = weave.init("feedback-example")

    spec1 = AnnotationSpec(
      name="Temperature",
      description="The perceived temperature of the llm call",
      field_schema={
        "type": "number",
        "minimum": -1,
        "maximum": 1,
      }
    )
    spec2 = AnnotationSpec(
      name="Tone",
      description="The tone of the llm response",
      field_schema={
        "type": "string",
        "enum": ["Aggressive", "Neutral", "Polite", "N/A"],
      },
    )
    weave.publish(spec1, "temperature-scorer")
    weave.publish(spec2, "tone-scorer")
    ```
  </Tab>

  <Tab title="TypeScript">
    ```plaintext theme={null}
    This feature is not available in TypeScript yet.
    ```
  </Tab>
</Tabs>

### Modify a human annotation scorer using the API

Expanding on [creating a human annotation scorer using the API](#create-a-human-annotation-scorer-using-the-api), the following example creates an updated version of the `Temperature` scorer, by using the original object ID (`temperature-scorer`) on `publish`. The result is an updated object, with a history of all versions.

> You can view human annotation scorer object history in the **Scorers** tab under **Human annotations**.

<Tabs>
  <Tab title="Python">
    ```python lines theme={null}
    import weave
    from weave.flow.annotation_spec import AnnotationSpec

    client = weave.init("feedback-example")

    # create a new version of the scorer
    spec1 = AnnotationSpec(
      name="Temperature",
      description="The perceived temperature of the llm call",
      field_schema={
        "type": "integer",  # <<- change type to integer
        "minimum": -1,
        "maximum": 1,
      }
    )
    weave.publish(spec1, "temperature-scorer")
    ```
  </Tab>

  <Tab title="TypeScript">
    ```plaintext theme={null}
    This feature is not available in TypeScript yet.
    ```
  </Tab>
</Tabs>

<Frame>
  <img src="https://mintcdn.com/wb-21fd5541/4ANo4MV8FzCjYewG/weave/guides/tracking/imgs/human-annotation-scorer-history.png?fit=max&auto=format&n=4ANo4MV8FzCjYewG&q=85&s=457c3ed3707635dbc12217d895787af1" alt="Human Annotation scorer history" width="2686" height="1434" data-path="weave/guides/tracking/imgs/human-annotation-scorer-history.png" />
</Frame>

### Use a human annotation scorer using the API

The feedback API allows you to use a human annotation scorer by specifying a specially constructed name and an `annotation_ref` field. You can obtain the `annotation_spec_ref` from the UI by selecting the appropriate tab, or during the creation of the `AnnotationSpec`.

<Tabs>
  <Tab title="Python">
    ```python lines theme={null}
    import weave

    client = weave.init("feedback-example")

    call = client.get_call("<call_id>")
    annotation_spec = weave.ref("<annotation_spec_ref_uri>")

    call.feedback.add(
      feedback_type="wandb.annotation." + annotation_spec.name,
      payload={"value": 1},
      annotation_ref=annotation_spec.uri(),
    )
    ```
  </Tab>
</Tabs>